Patterns for LLM Tool Use: Caching, Searching, and Chunking

Designing high‑performance GPT‑5 agents is as much about how they call tools as which tools they call. Three repeatable patterns do the heavy lifting for speed, cost, and reliability:

• Caching previous results to reuse answers.
• Search‑first filtering to narrow large datasets before retrieval.
• Chunking big responses into manageable pieces.

Below, you'll learn when to use each pattern, how to implement it, and how MCPify exposes them as first‑class capabilities so agents can choose the optimal plan in real time.

---

TL;DR

• Cache first: Check for exact or semantic cache hits before any expensive API/model call. Add TTLs and explicit invalidation.
• Search (and filter) before fetch: Use backend filters, sorting, and field selection to pull only what's relevant.
• Chunk intentionally: Page, stream, or JSON‑slice large payloads so they fit context and stay navigable.
• Make it explicit: MCPify turns caching, search/filter parameters, and chunking controls into transparent, richly documented tools an LLM can understand and optimize.

---

Why these patterns matter for GPT‑5‑based agents

• Context is finite: Even with large windows, dumping unfiltered data wastes tokens and risks missing critical details.
• Latency and cost compound: Redundant calls and mega‑payloads slow responses and inflate spend.
• Reliability improves with structure: Deterministic steps (cache check → filter → chunk) reduce failure modes and hallucination.

Goal: Give the model precise levers (not guesswork) to control what to fetch, how much to fetch, and when to avoid fetching at all.

---

Pattern 1: Cache Utilization

What it is

Store and reuse results from previous tool or model calls. Caches can hold full responses (exact matches), semantically similar answers, or expensive intermediates (embeddings, parsed schemas).

When to use it

• Questions repeat across users or sessions.
• Data changes slowly relative to your SLA.
• You want predictable P95 latency and lower cost per answer.

How to implement it (simple Python)

from time import time

CACHE = {}  # {cache_key: {"value": obj, "ts": epoch, "ttl": seconds}}

def cache_get(key):
    item = CACHE.get(key)
    if not item:
        return None
    if time() - item["ts"] > item["ttl"]:
        return None
    return item["value"]

def cache_set(key, value, ttl=900):  # 15-minute default
    CACHE[key] = {"value": value, "ts": time(), "ttl": ttl}

def fetch_user_profile(api, user_id):
    key = f"user_profile:{user_id}"
    cached = cache_get(key)
    if cached:
        return {"source": "cache", "data": cached}

    data = api.get_user_profile(user_id=user_id)  # external call
    cache_set(key, data, ttl=1800)  # 30-minute TTL
    return {"source": "live", "data": data}

Tips

• Use stable cache keys based on request parameters and auth context.
• Make freshness explicit: set TTLs relative to domain volatility (e.g., prices vs. static docs).
• Add manual invalidation for administrative workflows and staleness checks for critical reads.
• Consider semantic caching (embedding similarity) for Q&A reuse when wording varies.

How MCPify helps

• Cache transparency: Tools can expose cache status, TTL, and freshness to the agent.
• Cost and latency annotations: Let the LLM weigh cache hits vs live calls intelligently.
• Explicit invalidation: Provide a cache.invalidate tool so agents can force‑refresh when appropriate.

Links:

• MCPify Cache Control
• MCPify Cost and Latency Metadata

---

Pattern 2: Search‑First Filtering

What it is

Narrow the dataset at the source. Use query parameters, filters, sorting, and field selection to avoid fetching irrelevant data.

When to use it

• Large collections (orders, events, logs, products).
• APIs with rich query/filter capabilities.
• You need small, precise payloads for follow‑up steps.

How to implement it (agent plan + API call)

{
  "thought": "Find running shoes under $100 with >4.2 rating, return id,title,price only.",
  "action": "catalog.search",
  "args": {
    "q": "running shoes",
    "filters": {"price_max": 100, "rating_min": 4.2, "category": "athletic"},
    "sort": {"field": "rating", "direction": "desc"},
    "fields": ["id", "title", "price"],
    "limit": 25
  }
}

Tips

• Prefer filtering server‑side over client‑side post‑processing.
• Always request minimal fields required for the next step.
• Use stable sort and cursor‑based pagination for deterministic iteration.
• For search‑led workflows, combine keyword + structured filters.

How MCPify helps

• Dedicated search/filter tools: Every native API filter is exposed as a first‑class argument.
• Field selection: Fetch only needed fields to reduce tokens.
• Explicit pagination controls: Clear limit, cursor, and page semantics.

Links:

• MCPify Search and Filter Tools
• OpenAPI for expressive filters
• Elasticsearch Query DSL

---

Pattern 3: Chunking (Pagination, Streaming, and JSON Slicing)

What it is

Break large responses (or documents) into smaller, navigable pieces that fit the model's context. Chunking applies to lists, long JSON, and unstructured text.

When to use it

• Responses risk exceeding context window or token budget.
• You need to scan, rank, or summarize large sets in stages.
• You want to stream partial results for faster first tokens.

How to implement it

A. Paginate through large lists

def list_all_orders(api, start_date, end_date, page_size=100):
    cursor = None
    while True:
        resp = api.list_orders(
            date_from=start_date, date_to=end_date, limit=page_size, cursor=cursor
        )
        for order in resp["items"]:
            yield {"id": order["id"], "total": order["total"], "created_at": order["created_at"]}

        cursor = resp.get("next_cursor")
        if not cursor:
            break

B. Stream progressively (SSE)

const es = new EventSource("https://api.example.com/logs/stream?topic=payments");
es.onmessage = (event) => {
  const payload = JSON.parse(event.data);
  // process incremental chunks
};

C. Slice JSON with JSONPath

import json
from jsonpath_ng import parse

def extract_totals(orders_json):
    jsonpath_expr = parse("$..orders[*].total")
    return [match.value for match in jsonpath_expr.find(orders_json)]

Tips

• Make chunks semantically coherent (e.g., section‑level for documents).
• Use field‑level slicing for large objects.
• Combine search‑first with chunking: filter to a relevant subset, then page through it.
• Track iteration depth and stop early once you have enough evidence.

How MCPify helps

• JSONPath tooling for precise field extraction and array slicing.
• SSE support for streaming long‑lived tasks and real‑time feeds.
• Transparent pagination with rate limit awareness and backoff hints.

Links:

• MCPify JSON Navigation Tools
• MCPify Pagination and Streaming
• JSONPath (Goessner)
• MDN EventSource

---

Recognizing the right pattern (decision guide)

Ask these questions:

1. Have we seen this before?

   • Yes → Try cache (exact or semantic).
   • No → Continue.

2. Is the dataset large or broad?

   • Yes → Use search‑first filtering with minimal fields.
   • No → Continue.

3. Will the response exceed context or be slow to parse?

   • Yes → Chunk (paginate, stream, or JSON‑slice).
   • No → Direct fetch may be okay.

Bonus: If the workflow involves multiple steps or cross‑service joins, use stateful external memory to store intermediate lists and IDs instead of re‑fetching.

---

Implementation recipes

Cache keys and freshness

def key_for(endpoint: str, params: dict, auth_actor: str) -> str:
    # Stable, actor-aware
    base = endpoint + "|" + auth_actor
    pairs = [f"{k}={params[k]}" for k in sorted(params.keys())]
    return base + "|" + "&".join(pairs)

# TTL policy: default 15m; override per endpoint
TTL = {"get_user_profile": 1800, "list_orders": 600, "search_products": 300}

• Per‑endpoint TTLs: Match domain volatility.
• Manual invalidation: Admin tool to flush keys by prefix, user, or tag.
• Staleness headers: Return age_seconds so the agent can decide to refresh.

Cost‑aware planning

Expose cost_hint, latency_p50/p95, and rate_limit in tool metadata. Let the agent choose between:

• One big fetch vs. several filtered fetches.
• Fetch now vs. reuse cache and defer refresh.

Stateful operations (scratchpad)

Maintain an external memory store for intermediate sets:

{
  "action": "state.put",
  "args": {"key": "candidates:v1", "value": ["id_123","id_456","id_789"], "ttl": 3600}
}

The agent can later state.get("candidates:v1") instead of recomputing.

---

Anti‑patterns to avoid

• Fetch‑all and pray: Pulling entire tables and filtering in the prompt.
• Hidden caching: Opaque server caches that the LLM can't see or control.
• Oversized chunks: Long documents or arrays stuffed into a single context window.
• Fan‑out thundering herds: Dozens of parallel calls without rate‑limit awareness or batch tools.
• Unbounded retries: No backoff; agents loop on transient errors.

---

How MCPify makes these patterns explicit

MCPify turns each capability into a transparent, AI‑ready tool:

• Caching: Cache status, TTLs, invalidation, and freshness are visible to the agent. See MCPify Cache Control.

• Search‑first filtering: Every native filter, sort, and field selector is exposed as first‑class parameters. See MCPify Search and Filter Tools.

• Chunking and response navigation: JSONPath selectors, field‑level slicing, cursor pagination, and SSE streaming. See MCPify JSON Navigation and Pagination and Streaming.

• Stateful memory and batching: External scratchpad for multi‑step workflows and batch operations for efficient fan‑out. See MCPify Stateful Tools and MCPify Batch.

• Cost and performance transparency: Per‑tool cost hints, rate limits, and latency stats so GPT‑5 agents choose the cheapest, fastest plan. See MCPify Metadata.

---

Checklist: before you ship the agent

• Cache check before every tool call; semantic cache for Q&A reuse.
• Prefer server‑side filters; request only needed fields.
• Chunk big responses with pagination, JSONPath, or SSE.
• Track hit rate, tokens/answer, and P95 latency.
• Expose metadata (cost, latency, rate limits) to the model.
• Provide state.put/get and cache.invalidate tools.
• Add guardrails for retries, backoff, and quotas.

---

Frequently asked questions

How big should a chunk be? Big enough to be meaningful on its own, small enough to fit comfortably in context with the prompt and other evidence. For RAG, many teams target a few hundred tokens per chunk and tune empirically.

When should I skip caching? Highly volatile data (e.g., live bids) or strong consistency reads. For these, keep TTL very short or mark calls as non‑cacheable.

Is search‑first always better than a single fetch? If the dataset is small and stable, a single fetch may be fine. But for anything at scale, search‑first with field selection almost always wins on tokens and latency.

---

Call to action

Ready to give your GPT‑5 agents real levers? MCPify turns caching, searching, and chunking into explicit, AI‑friendly tools.

• Talk to us about your API: Get in touch
• Read the Cache Control guide
• Explore Search and Filter tools and JSON Navigation
• Talk to us about onboarding your APIs: Contact MCPify

---

Sources

• MCPify - Home: mcpify.org
• MCPify - Cache Control: /docs/cache-control
• MCPify - Metadata: /docs/metadata
• MCPify - Search and Filter Tools: /tools/search-filter
• MCPify - JSON Navigation Tools: /tools/jsonpath
• MCPify - Pagination and Streaming: /docs/pagination-and-streaming
• MCPify - Stateful Operations: /docs/stateful-operations
• MCPify - Batch Operations: /docs/batch-operations
• Model Context Protocol (overview): https://modelcontextprotocol.io/
• OpenAPI Specification: https://spec.openapis.org/oas/latest.html
• JSONPath (Goessner): https://goessner.net/articles/JsonPath/
• MDN - EventSource: https://developer.mozilla.org/en-US/docs/Web/API/EventSource
• RFC 9111 - HTTP Caching: https://www.rfc-editor.org/rfc/rfc9111
• Pinecone - Chunking for RAG: https://www.pinecone.io/learn/chunking/