Back to Blog
Technical Tutorials

Token Counting 101: Managing Context Size and API Responses

A practical primer on tokens, context windows, and response sizing. Learn to estimate, trim, chunk, and cache data—and see how MCPify makes token control automatic.

Herman Sjøberg
Herman Sjøberg
AI Integration Expert
August 28, 202511 min read
TokenizationLLM EngineeringMCPContext WindowsGPT‑5Cost Optimization

Key Takeaways

  • Tokens are the billing and capacity unit for LLMs—think in tokens, not characters
  • Context windows are hard caps—treat them as a budget
  • Filter‑first and preview‑chunk patterns for API integrations
  • Practical Python and JavaScript token counting with tiktoken
  • MCPify provides automatic token counting and size previews for every API call

Token Counting 101: Managing Context Size and API Responses

TL;DR

  • LLMs like GPT‑5, Claude, and Gemini operate on tokens (not characters or words).
  • Your model's context window is a hard cap on how many tokens it can consider at once.
  • Counting tokens before you call the model helps you avoid errors, reduce cost, and lower latency.
  • Use practical tactics: filtering, summarization, truncation, and token‑based chunking.
  • MCPify adds token awareness to every API you call: automatic token counts, cache freshness, and response‑size previews—so your agents never overfill context by accident.

Why developers should think in tokens

If your app intermittently fails with "maximum context length" errors, or your bill spikes after a busy day, your workflow is likely over‑tokenized. Tokens are the billing and capacity unit for LLMs; every prompt, tool definition, system message, and piece of retrieved data consumes tokens. Designing token‑smart workflows improves reliability and speed, and it often slashes spend.

Token basics (and what they mean for you)

  • A token is a chunk of text the model processes. In English, a token is on average ~4 characters or ~¾ of a word. That means punctuation, whitespace, and emojis count too. See OpenAI's explainer and tokenizer tool for a concrete feel of how text breaks into tokens.
  • Most vendors expose token counts so you can track input and output usage per request.

Key implications

  • Estimating tokens is more accurate than counting characters or words.
  • Shortening prompts and dropping irrelevant fields are the fastest ways to cut cost and reduce latency.
  • Even if a model can handle long context, irrelevant tokens dilute signal. Feed the model only what it needs, when it needs it.

Context windows: your model's working memory

A context window is the total number of tokens your model can consider at once (prompt + tools + retrieved data + the model's own output). If you go over, you'll need to summarize, filter, or chunk the input.

  • Industry docs describe the context window as the model's working memory and emphasize that exceeding it leads to truncation or errors. Some current models offer hundreds of thousands to millions of tokens of context, but you still need to curate what you send for quality and cost.

Design mindset

  • Treat the context window as a budget.
  • Keep a running estimate of tokens before you call the model.
  • Decide upfront how you'll trim when you're over budget.

How to count tokens in practice

Python (tiktoken)

pip install tiktoken

# Count tokens for budgeting (choose an encoding compatible with your model)
import tiktoken

# Common encodings include "cl100k_base" and "o200k_base"
ENCODING_NAME = "cl100k_base"

enc = tiktoken.get_encoding(ENCODING_NAME)

def count_tokens(text: str) -> int:
    return len(enc.encode(text))

sample = "Hello, token-aware world! 👋"
print("Tokens:", count_tokens(sample))

JavaScript/TypeScript (js‑tiktoken, lite build)

npm install js-tiktoken

// Small and fast: load only the encoder you need
import { Tiktoken } from "js-tiktoken/lite";
import o200k_base from "js-tiktoken/ranks/o200k_base";

const enc = new Tiktoken(o200k_base);

export function countTokens(text: string): number {
  return enc.encode(text).length;
}

console.log("Tokens:", countTokens("Hello, token-aware world! 👋"));

Tip: Count all pieces of your prompt (system messages, tool specs, few‑shot examples, retrieved passages, and user content). Then apply a safety margin for the model's output tokens.

The token diet: practical ways to reduce usage

  1. Don't over‑fetch

    • Ask your upstream API for only the fields you need. Prefer server‑side filters, sorts, and pagination.
    • Use JSONPath or equivalent filters to pick just the slices your logic requires.

  2. Summarize early, summarize often

    • Summarize large documents or long histories before sending to the model.
    • Keep raw source data available out of band (object store, vector DB), but pass compact summaries into the prompt.

  3. Truncate with intent

    • If you must cut, do it deterministically: drop oldest chat turns or least‑relevant fields first.
    • Set hard limits (for example, "never exceed 24k tokens of history") and enforce them in code.

  4. Chunk by tokens, not bytes

    • When you must stream large data into the model, split payloads into token‑sized chunks (for example, 1k–2k tokens).
    • Process, summarize, and then stitch results progressively.

  5. Cache and reuse

    • Cache expensive partials (search results, structured summaries) and store their token counts to avoid re‑incurring usage.
    • Prefer retrieving IDs + targeted fields rather than refetching whole objects.

Designing token‑smart API integrations

When you're pulling from APIs, the risk isn't just "too much data," it's unstructured data. Two practical patterns help:

Pattern A: Filter‑first, then fetch

  • Query the upstream service with strict filters and projected fields.
  • If you need more detail, drill down only on the subset of items the model flagged as relevant.
  • This avoids flooding the model's context with irrelevant JSON.

Pattern B: Preview and chunk

  • Preview size first: inspect the expected token count and only include what fits.
  • If the preview exceeds your budget, chunk and process incrementally (page by page or section by section).
  • Persist partial results and their token counts for re‑use.

How MCPify makes token management automatic

MCPify transforms any REST, GraphQL, or proprietary API into an MCP service (served at https://{service}.mcp.mcpify.org/mcp) that is transparent to LLMs and token‑aware by design:

  • Automatic token counting for response bodies, with consistent encodings.
  • Size previews so agents can decide whether to include, summarize, or chunk data before adding it to the prompt.
  • Fine‑grained JSON navigation tools (JSONPath, array slicing, field extraction) to surgically select only what you need.
  • Explicit pagination and chunking controls so the model can iterate through large datasets safely.
  • Cache transparency (is this result cached, how fresh is it, and what is its token footprint?).
  • Cost and latency metadata so your agent can trade off accuracy, speed, and budget.

Example: Token‑aware API fetch with MCPify

1) Ask MCPify for a preview (no payload yet). The agent calls a preview tool on the MCP service to get a token-and-size estimate before fetching:

{
  "tool": "github.issues.preview",
  "args": { "repo": "acme/payments", "state": "open", "per_page": 100 }
}

Preview response (abridged):

{
  "service": "github",
  "endpoint": "/repos/{owner}/{repo}/issues",
  "filters": { "state": "open", "per_page": 100 },
  "meta": {
    "token_count_estimate": 1437,
    "encoding": "o200k_base",
    "cached": true,
    "latency_ms_estimate": 45,
    "rate_limit": { "remaining": 4875, "reset_in_s": 293 },
    "cost_estimate_usd": 0.0023
  },
  "recommendations": [
    "Apply fields filter: title,number,labels,assignees",
    "Use jsonpath '$[*].title' to extract only titles if summarizing"
  ]
}

2) Fetch with field selection + JSONPath. The agent proceeds with a size-safe call, projecting only needed fields:

{
  "tool": "github.issues.list",
  "args": {
    "repo": "acme/payments",
    "state": "open",
    "fields": "title,number,labels",
    "per_page": 100
  }
}

3) Extract only what you need (MCPify tool call):

{
  "tool": "jsonpath.select",
  "args": {
    "path": "$[*].title",
    "input_reference": "github_issues_741e"
  }
}

MCPify returns both the filtered values and the token_count of the filtered result so your agent can include the minimum necessary text while staying under budget.

Architecturally, MCPify treats LLMs as the intelligence and provides radical transparency about where data lives, how big it is, and what it will cost in tokens to include. That turns "guess‑and‑check" into deterministic token planning.

Worked example: Keep a 24k‑token budget

Let's say you have a 32k‑token model target, and you want to reserve 8k tokens for the model's output. That leaves 24k tokens for everything else.

  1. Standing context (system + tools): 2.5k
  2. Conversation history (summarized): 3.5k
  3. API results (previewed): up to 18k

Workflow

  • MCPify previews your candidate API call at 21k tokens → too large.
  • Use MCPify's JSON tools to extract only title, status, and total fields, cutting it to 6.2k.
  • Keep a link to the full result set via MCPify's reference handle, and fetch additional chunks only if needed.

You finish comfortably under 24k while keeping the 8k output headroom intact.

A quick checklist for token‑safe workflows

  • Count tokens for system prompts, tool specs, and examples.
  • Preview API responses and trim at the source.
  • Keep a token budget and enforce it in code.
  • Summarize long histories; store raw data externally.
  • Chunk by tokens, not bytes.
  • Cache partials and the associated token counts.
  • Use MCPify's metadata to drive decision rules: include, summarize, or chunk.

Call to action

Build token‑smart agents from day one:

Sources

Who This Article Is For

Developers building LLM applications who need to manage token usage and context windows effectively

About the Author

Herman Sjøberg

Herman Sjøberg

AI Integration Expert

Herman excels at assisting businesses in generating value through AI adoption. With expertise in cloud architecture (Azure Solutions Architect Expert), DevOps, and machine learning, he's passionate about making AI integration accessible to everyone through MCPify.

Connect on LinkedIn