Monitoring & Analytics for LLM Tool Usage: Keeping an Eye on Performance

Audience: DevOps, MLOps, and AI engineers searching for practical guidance on monitoring AI agent performance and costs.

TL;DR

If your agents use tools and APIs, you need observability. Track per‑tool latency, success and error rates, usage frequency, token and cost per call, cache hit rates, and rate limiting. Instrument with logs, metrics, and traces, visualize in dashboards, and alert on regressions. MCPify gives you built‑in analytics and structured events across every MCPified API so you can find slow endpoints, prevent cost blowups, and prune unused tools fast. Start with a minimal metrics and tracing setup, then optimize based on data.

---

Why monitoring LLM tool usage matters

AI agents like Claude and GPT‑5 unlock new automation patterns, but their reliability depends on the weakest tool they call. A single slow endpoint can tank your p95. An overly verbose endpoint can double token spend. An unobserved 429 spike can trigger cascading retries and timeouts.

Good observability replaces guesswork with facts:

• Which tools are used most and when
• Which endpoints dominate latency and errors
• How many tokens and dollars each tool consumes
• Where cache is saving time and spend
• Where you are flirting with rate limits

Modern stacks use OpenTelemetry for unified logs, metrics, and traces, then pipe data into Grafana, Datadog, or your preferred platform. See Datadog's LLM Observability and Grafana's LLM observability guide for patterns and dashboards: Datadog LLM Observability, Grafana guide.

---

The metrics that matter

Track these at the tool and endpoint level, with labels for environment, model, user tier, and region:

• Latency: p50, p90, p95, p99 per endpoint. Diagnose tail latency and regressions fast.
• Success and failure rate: 2xx vs 4xx vs 5xx, plus timeouts and client‑canceled requests.
• Throughput and usage frequency: requests per tool and endpoint to find hotspots and dead weight.
• Token usage: prompt, completion, and total tokens per call and per tool. Tie to cost.
• Cost per call and per tool: model and API pricing rollups for real spend visibility.
• Cache metrics: hit ratio, freshness, invalidations, and time saved.
• Pagination behavior: average page size, page count, and over‑fetch patterns.
• Rate limiting: 429s, retry counts, backoff durations.
• Batching and parallelization: batch size distributions and wall‑clock improvements.
• SLO compliance: percent of requests under your SLO, error budget burn.

Pro tip: Correlate metrics and traces so you can jump from a red metric to the exact slow span in one click. OpenTelemetry is designed for this correlation out of the box.

---

Collecting the data: instrumentation patterns

At every tool invocation, emit structured logs, metrics, and traces.

1) Structured event log

Emit one start and one finish event per tool call. Keep it boring and consistent.

{
  "event": "tool.call.finish",
  "tool": "orders.search",
  "endpoint": "GET /orders",
  "request_id": "c91f5f2c",
  "status_code": 200,
  "duration_ms": 312,
  "tokens_prompt": 184,
  "tokens_completion": 92,
  "cost_usd": 0.0017,
  "cache_hit": false,
  "retries": 0,
  "rate_limited": false,
  "timestamp": "2025-08-27T12:34:56Z"
}

2) Metrics with Prometheus client (Python)

Use counters and histograms with tool labels. Expose a /metrics endpoint for scraping.

from prometheus_client import Counter, Histogram, start_http_server
import time
import random

REQUESTS = Counter(
    "tool_requests_total",
    "Total tool calls",
    ["tool", "status"]
)
LATENCY = Histogram(
    "tool_latency_seconds",
    "Tool call latency",
    ["tool"],
    buckets=[0.05, 0.1, 0.25, 0.5, 1, 2, 5]
)

def call_tool(tool):
    start = time.time()
    try:
        # simulate work
        time.sleep(random.uniform(0.05, 0.4))
        status = "success"
        return {"ok": True}
    except Exception:
        status = "error"
        raise
    finally:
        LATENCY.labels(tool).observe(time.time() - start)
        REQUESTS.labels(tool, status).inc()

if __name__ == "__main__":
    start_http_server(8000)  # scrape on :8000/metrics
    while True:
        call_tool("orders.search")

PromQL examples:

# p95 latency by tool (5m window)
histogram_quantile(
  0.95,
  sum(rate(tool_latency_seconds_bucket[5m])) by (le, tool)
)

# error rate by tool
sum(rate(tool_requests_total{status!="success"}[5m])) by (tool)
/
sum(rate(tool_requests_total[5m])) by (tool)

3) Tracing with OpenTelemetry (Node example)

Wrap each tool call in a span. Add attributes for quick filtering.

import { trace } from "@opentelemetry/api";

const tracer = trace.getTracer("agent-tools");

export async function runTool(toolName, fn) {
  const span = tracer.startSpan(`tool:${toolName}`);
  try {
    const start = Date.now();
    const res = await fn();
    span.setAttribute("tool.name", toolName);
    span.setAttribute("http.status_code", 200);
    span.setAttribute("duration_ms", Date.now() - start);
    return res;
  } catch (err) {
    span.setAttribute("tool.name", toolName);
    span.setAttribute("error", true);
    span.setAttribute("error.message", String(err));
    throw err;
  } finally {
    span.end();
  }
}

Ship metrics and traces via your OpenTelemetry Collector, and visualize in Grafana or Datadog.

---

Dashboards and alerts that actually help

Create a top‑level "Agent Tools" dashboard with:

• Top tools by request volume and by token spend
• Latency heatmap per endpoint, plus p50, p95, p99
• Error and timeout rate with drill‑down to traces
• Token and cost over time by tool and by model
• Cache hit ratio and rate‑limit events
• SLO tracking and error budget burn

Essential alerts:

• p95 latency by tool exceeds threshold for 10 minutes
• Error rate by tool exceeds 5 percent over 5 minutes
• Token spend per minute jumps above baseline
• Cache hit ratio drops below 70 percent
• 429 rate limiting spikes beyond baseline

---

Detecting bottlenecks and waste

With proper instrumentation you can quickly spot:

• Slow endpoints: a single path dominating p95. Fix with caching, field selection, or a better query plan.
• Over‑fetching: tools returning massive payloads. Trim response fields, paginate, or use JSON slicing.
• Unused tools: near‑zero usage for weeks. Prune them to simplify the agent's toolset.
• Rate‑limit churn: 429 spikes with inefficient retries. Add backoff and raise pre‑call concurrency gates.
• Token blowups: outliers in token usage. Tighten prompts, enable response chunking, or add server‑side filters.

---

How MCPify helps out of the box

MCPify is the observability‑first gateway for agent tooling:

• Built‑in analytics across all MCPified APIs: per‑tool latency percentiles, success and error rates, token and cost per call, cache hit ratio, and rate‑limit events in one place.
• Structured events emitted for every tool call. Ship to your SIEM, Prometheus, or OTLP collectors without custom glue.
• Token and cost transparency with per‑endpoint rollups, so you know exactly what each tool costs and why.
• Explicit pagination and response slicing tools so LLMs can fetch only what they need and reduce tokens.
• Gateway‑first caching and rate limiting with visibility into savings and protection.

Read the event schema and setup guide: MCPify analytics docs

---

Illustrative optimization example: from red to green

Illustrative worked example — not a measured customer result. The numbers below are invented to show the arithmetic of a typical diagnose‑and‑fix loop.

Symptom: p95 latency for catalog.search climbs from 420 ms to 1.9 s, and token spend jumps 33 percent week over week.

What the dashboards and traces would show:

• 78 percent of requests return a 300‑item payload with many unused fields.
• Cache hit ratio falls from 81 percent to 46 percent after a backend change.
• 13 percent of calls hit 429s with immediate retries.

Fixes:

• Enable response field filtering and page size 50 via MCPify's explicit pagination tools.
• Turn on gateway caching for common queries with a 60 s TTL and cache‑busting on updates.
• Add exponential backoff and a maximum of 2 retries on 429.

Expected result in this scenario:

• p95 latency drops to 560 ms (a 70 percent improvement from the peak in this example).
• Average tokens per call down 38 percent as over‑fetching stops.
• 429s fall to 0.3 percent of traffic, error budget back in the green.

---

Quick‑start checklist

• Emit a tool.call.start and tool.call.finish event for every tool invocation
• Record latency, status, tokens, cost, cache hit, retries, and rate‑limit flags
• Add Prometheus counters and histograms labeled by tool and endpoint
• Wrap calls in OpenTelemetry spans with tool.name, http.status_code, and duration_ms
• Stand up an "Agent Tools" dashboard with volume, latency, errors, tokens, and cost
• Set alerts for p95 latency, error rate, spend spikes, cache drop, and 429 surges
• Use MCPify's built‑in analytics to find slow or unused tools and trim token usage
• Revisit SLOs quarterly and prune or refactor low‑value endpoints

---

Sample MCPify analytics configuration (illustrative)

analytics:
  enabled: true
  sampling: 1.0
  emit:
    - type: "metrics"
      backend: "prometheus"
      endpoint: "http://metrics-pushgateway:9091"
    - type: "traces"
      backend: "otlp"
      endpoint: "http://otel-collector:4317"
    - type: "logs"
      backend: "stdout"

alerts:
  - name: "High tool error rate"
    expr: >
      sum(rate(tool_requests_total{status!="success"}[5m])) by (tool)
      / sum(rate(tool_requests_total[5m])) by (tool) > 0.05
    for: "10m"
    severity: "page"

Note: MCPify emits standard structured events so you can forward them to Datadog, Grafana LGTM, Splunk, or any OTLP‑compatible backend.

---

Call to action

Want tool‑level visibility into what your agents are doing and what it costs?

• Read the setup guide: MCPify analytics docs
• Book a demo to see MCPify's analytics on your own APIs
• Or talk to sales about your stack

---

Sources

• Datadog LLM Observability (product overview): https://www.datadoghq.com/product/llm-observability/
• Datadog LLM Observability (docs): https://docs.datadoghq.com/llm_observability/
• Datadog LLM Observability HTTP API: https://docs.datadoghq.com/llm_observability/instrumentation/api/
• Grafana guide to LLM observability with OpenTelemetry: https://grafana.com/blog/2024/07/18/a-complete-guide-to-llm-observability-with-opentelemetry-and-grafana-cloud/
• Guardrails AI on LLM performance and OpenTelemetry: https://www.guardrailsai.com/blog/opentelemetry-llm-performance
• OpenTelemetry overview: https://opentelemetry.io/docs/specs/otel/overview/
• OpenTelemetry metrics spec: https://opentelemetry.io/docs/specs/otel/metrics/
• OpenTelemetry traces concept: https://opentelemetry.io/docs/concepts/signals/traces/
• Prometheus histograms best practices: https://prometheus.io/docs/practices/histograms/
• Prometheus client_python histogram reference: https://prometheus.github.io/client_python/instrumenting/histogram/
• MCPify homepage: mcpify.org
• MCPify analytics docs: /docs/analytics