LlamaIndex

Track every LlamaIndex agent run with cost, latency, token usage, and tool call data. Call instrument() once and the handlers register on the global LlamaIndex dispatcher. No code changes needed.

Requirements

• Python 3.10 or later
• llama-index-core 0.11 or later

1. Install

pip install agentmetrics-llamaindex

2. Instrument

Call instrument() once at startup, before creating any agents or running queries.

import os
from agentmetrics_llamaindex import instrument

span_handler = instrument(
    api_key=os.environ["AGENTMETRICS_API_KEY"],
    agent_id="my-llamaindex-agent",
    base_url="http://localhost:8099",
)

After this, run your LlamaIndex agents normally. Every invocation is tracked automatically.

response = agent.chat("summarize this document")

3. Flush before exit (optional)

For short-lived scripts, call flush() on the returned span handler to wait for all in-flight requests to complete before the process exits.

span_handler.flush()

What gets tracked

• Duration from agent span start to span end
• Success or failure (exception message is recorded)
• Input and output token counts, including cache read and write tokens
• Cost (estimated from model pricing tables)
• Tool call count, tool names, and tool errors
• Model name

How it works

instrument() registers two handlers on LlamaIndex's root dispatcher:

• AgentMetricsSpanHandler: tracks top-level agent spans (agents, engines, query runners)
• AgentMetricsEventHandler: accumulates LLM token counts and tool calls within each span

Only top-level spans are tracked. Nested sub-spans roll up into the same run summary.

Configuration