Agent Observability
Knowing which agent burned which tokens, in which workspace, on which repository - and whether the work actually succeeded
The Question This Answers
Not "is the platform healthy" and not "which model should we use" - something narrower and harder
Three adjacent disciplines, one boundary
Monitoring owns infrastructure health - CPU, memory, workspace start times, service level objectives. LLMOps owns model selection, prompt management, and version pinning. Agent observability owns the causal chain of a single agent run: which agent, invoked by whom, in which workspace, against which repository and branch, calling which tools and which models, consuming how many tokens, and finishing in success or failure. A Grafana dashboard showing healthy pods tells you nothing about an agent that has spent forty dollars looping on a task it will never complete.
The reason this needs its own practice is that agent work is non-deterministic, multi-step, and expensive per step. A traditional request trace answers "what happened and how long did it take". An agent trace has to answer "what did it decide, why did that cost what it cost, and was the outcome worth it". Those are different questions, and the second one is the one finance will eventually ask.
There is also a diagnostic reason. Teams adopting agents at scale usually have a constraint somewhere - context quality, review capacity, or raw model spend - and they usually guess wrong about which one it is. Instrumentation is how you find out rather than argue about it. If agent throughput is high and merge rates are low, the problem is downstream in review. If token spend per completed task is climbing, the problem is usually upstream in context.
The Span Structure of an Agent Run
OpenTelemetry's generative AI semantic conventions give this a shared vocabulary
OpenTelemetry's generative AI semantic conventions define operations for agent invocation, workflows, tool execution, and model inference, along with token-usage metrics. Using them rather than inventing your own attribute names is worth doing for one practical reason: it is what lets you change vendors without re-instrumenting. The gen_ai.operation.name attribute carries the well-known values chat, invoke_agent, invoke_workflow, execute_tool and embeddings, among others, and span names follow patterns such as invoke_agent {gen_ai.agent.name} and execute_tool {gen_ai.tool.name}.
Two things most published examples get wrong
The conventions moved repositories. The generative AI conventions now live in their own repository at open-telemetry/semantic-conventions-genai. The old paths under the main specification site still return HTTP 200 but now serve only a "this page has moved" stub, which is worse than a broken link because it looks like documentation. There is no tagged release of the generative AI conventions, and every attribute in them is marked Development - OpenTelemetry's term for experimental. Nothing here is stable, so pin your expectations accordingly.
gen_ai.system is deprecated and removed. It was replaced by gen_ai.provider.name. This is the single most common error in currently-published writing on this topic, including articles that otherwise look up to date. Likewise gen_ai.usage.prompt_tokens and gen_ai.usage.completion_tokens are superseded by input_tokens and output_tokens.
Example: attributes on a single agent run
# Root span - the agent invocation. Span kind INTERNAL.
span: invoke_agent refactor-bot
gen_ai.operation.name = "invoke_agent" # Required
gen_ai.provider.name = "<provider>" # Required - NOT gen_ai.system
gen_ai.agent.name = "refactor-bot"
gen_ai.agent.id = "agt_8f21c9"
gen_ai.conversation.id = "run_20a4e7" # a real id, not a hash
# Not in the spec - this is the half you must add yourself,
# and it is the half that makes attribution possible at all.
cde.workspace.id = "ws-4471"
cde.workspace.template = "python-3.12-gpu"
vcs.repository.url = "github.com/acme/payments"
vcs.ref.head = "agent/PAY-1183/idempotency"
vcs.change.id = "PR-4471"
enduser.id = "u_1182" # the invoking human
org.cost_center = "platform-eng"
org.team = "payments"
|- span: chat <model-id> # inference. Span kind CLIENT.
| gen_ai.operation.name = "chat"
| gen_ai.request.model = "<model-id>"
| gen_ai.response.model = "<model-id>"
| gen_ai.usage.input_tokens = 18422
| gen_ai.usage.output_tokens = 1140
| # Cached input is priced differently - record it separately
| gen_ai.usage.cache_read.input_tokens = 16000
|
|- span: execute_tool run_tests # tool call. Span kind INTERNAL.
| gen_ai.operation.name = "execute_tool"
| gen_ai.tool.name = "run_tests" # Required here
| gen_ai.tool.call.id = "call_77b2"
| error.type = "test_failure"
| # gen_ai.tool.call.arguments / .result are Opt-In:
| # on a coding agent they carry source code. See below.
|
|- span: execute_tool apply_patch
gen_ai.tool.name = "apply_patch"
# Terminal outcome - classified, not inferred
agent.outcome = "success" # success | failed | abandoned | budget_halted
agent.outcome.reason = "pr_opened"
Metrics, not just spans
Traces answer "what happened in this run". Metrics answer "what is happening across all runs", and they are much cheaper to retain. The conventions define gen_ai.client.token.usage as a histogram in units of tokens, dimensioned by gen_ai.token.type with values input and output, alongside gen_ai.client.operation.duration.
Less widely known, and more useful for agent fleets specifically, are the agent-level instruments: gen_ai.invoke_agent.duration, gen_ai.invoke_agent.inference_calls, gen_ai.invoke_agent.tool_calls, gen_ai.execute_tool.duration and gen_ai.workflow.duration. Inference calls and tool calls per agent run are the two histograms that expose a looping agent before the bill does.
Backend support, stated precisely
Datadog documents ingesting OpenTelemetry traces that follow the generative AI semantic conventions, using gen_ai.operation.name to classify spans as model, embedding, tool, agent or workflow. Honeycomb documents the same attributes to group spans into conversations and drive its agent timeline view. A claim you will see repeated elsewhere and which we could not substantiate is native support in New Relic: a sweep of their OpenTelemetry and AI monitoring documentation found no reference to gen_ai attributes, and their OpenTelemetry path for language models appears to run through third-party shims rather than the conventions directly. Verify against your own vendor's documentation rather than a summary - including this one.
The standard covers less than you need
The generative AI conventions describe the model interaction well. They say nothing about which repository, branch, workspace, or cost center the interaction belongs to - and those are exactly the dimensions you need to attribute spend. Adopt the standard attributes where they exist and layer your own consistently-named resource attributes over the top. Decide the naming once, centrally, because retrofitting it across dozens of agent integrations is miserable.
Correlate to the human, always
Every agent run should carry the identity of the person or service account that started it. Without it you cannot answer the two questions that matter most in an incident - who launched this, and what else did they launch - and you cannot produce the audit trail that compliance will eventually ask for. This is also the join key between agent telemetry and your existing identity and access tooling.
Tokens as a First-Class Metric
A provider invoice is not an attribution system
Most organizations discover their agent spend through a monthly bill that says one large number and nothing else. That is sufficient to be alarmed and insufficient to act. Token usage needs to be recorded as a metric at the point of consumption, dimensioned by the same attributes as the trace, so that spend can be sliced by team, repository, agent, model, and outcome without asking the provider for a breakdown they cannot give you.
Separate Input From Output
They are priced differently, often by an order of magnitude, and they move for different reasons. Rising input tokens usually means a context problem. Rising output tokens usually means a task-scoping problem. Collapsing them into one number destroys the diagnosis.
Track Cache Behavior
Where a provider offers prompt caching, cached and uncached input are priced differently, and a change in cache hit rate can move the bill substantially without any change in agent behavior. The conventions provide gen_ai.usage.cache_read.input_tokens and gen_ai.usage.cache_creation.input_tokens. Record them or you will misattribute the swing.
Price at Ingest
Convert tokens to currency when the span is recorded, using the rate in force at that moment, and store both. Reconstructing historical cost after a provider changes pricing is otherwise guesswork.
Attribution is an organizational problem before it is a technical one
The technical part - propagating a cost center attribute through a trace - is straightforward. The hard part is agreeing what the dimensions should be, and that argument is worth having early. Attribute to the team that owns the repository, not the individual who pressed the button, unless you want to create a metric people optimize against. Shared platform agents need their own cost center rather than being spread arbitrarily. This is ordinary FinOps practice applied to a new resource type, and the same showback-before-chargeback sequencing applies.
Cost Per Successful Task
The only unit that means anything economically
Cost per agent run is a misleading metric, because a run that fails still costs money and produces nothing. Cost per token is worse - it measures the provider's pricing, not your efficiency. The unit that carries economic meaning is cost per successful task, and computing it requires something most instrumentation skips: an explicit, honest classification of outcome.
Classify the outcome deliberately
"The process exited zero" is not success. Useful terminal states are narrower: a pull request was opened, it passed CI, it was merged, it was reverted within some window. Each is a different bar, and which one you choose determines whether the metric flatters you.
The strictest defensible definition - merged and not reverted - is also the most expensive to measure, because it requires joining agent telemetry to your version control system days after the run finished. It is worth the trouble, because every weaker definition can be satisfied by an agent that produced nothing of value.
Where the metric misleads
Be honest about its limits. Cost per successful task rewards agents that attempt easy work and penalizes those pointed at hard problems, so comparing two agents on it without controlling for task difficulty produces a conclusion you should not act on.
It also ignores the human cost entirely. An agent that produces cheap merged pull requests while consuming hours of senior review time can look excellent on this metric and be a net loss. Pair it with the review-side measures in DevEx metrics before drawing conclusions.
Runaway loops and budget circuit breakers
The characteristic failure of an autonomous agent is not crashing. It is looping - retrying a failing test, re-reading the same files, re-deriving the same wrong conclusion, indefinitely and at full token price. Detection signals that work in practice: token consumption within a single run crossing a threshold, the same tool called repeatedly with near-identical arguments, step count exceeding a ceiling, and elapsed time with no change to the working tree.
Detection is only half of it. A circuit breaker must be able to actually halt the run - per-run token ceilings, per-workspace daily budgets, and a hard organizational cap - and the halt should emit a span with an outcome of budget_halted rather than vanishing silently. An alert that fires after the money is spent is a report, not a control. The trade-off is real: set the ceilings too tight and you will kill legitimate long-running work, so start by observing the distribution before you enforce against it.
Your Traces Contain Source Code
The most under-discussed problem in this area, and the one with real legal exposure
Capturing prompts and completions is enormously useful for debugging agent behavior, and it means your observability backend now holds proprietary source code, plus whatever the agent happened to read - configuration, test fixtures, customer data in a seed file, a credential someone committed years ago. A telemetry pipeline that was scoped for latency histograms is now a secondary copy of your codebase, usually in a third-party system, frequently with a longer retention period than anyone chose deliberately.
On a coding agent the highest-risk attributes are not the ones people expect. gen_ai.tool.call.arguments and gen_ai.tool.call.result are where the patches, file contents and command output actually live.
The specification is unusually direct about this
The content-bearing attributes - gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions and the tool-call pair above - are all classified Opt-In, the strictest tier. The conventions state that model instructions, user messages and model outputs "are considered sensitive and are often large in size", and that instrumentations "SHOULD NOT capture them by default, but SHOULD provide an option for users to opt in".
Three patterns are sanctioned: capture nothing (the default), record content directly on the spans, or store content externally and record only references on the spans. The third is explicitly recommended for production where volume is a concern or sensitive data needs handling securely, precisely because it "enables separate access controls". For a coding agent that is the right default - the trace keeps the shape of what happened and the source code lives somewhere you already govern.
Keep Capture Opt-In
Enable content capture deliberately and per environment. Pre-production is where it earns its keep; production is where it becomes a liability nobody signed off on.
Split Retention
Metrics and span metadata can live for a year cheaply. Prompt and completion bodies should expire in days. One retention policy for both guarantees you picked the wrong one.
Redact Before Export
Run secret scanning in the collector, not at the backend. Once a key has left your network it must be rotated regardless of what the vendor promises about storage.
Inherit Repo Permissions
Trace access is usually granted broadly to engineers. If traces contain code from restricted repositories, your observability tool has quietly become a permissions bypass.
The compensating benefit is genuine: the same trace data that creates this exposure is what makes an audit trail possible. Being able to demonstrate which agent modified a regulated system, under whose authority, with what result, is increasingly something an auditor will ask to see. Design the pipeline so that the durable, low-sensitivity metadata satisfies the audit requirement, and the high-sensitivity content expires quickly. See AI governance for the policy layer and secrets management for keeping credentials out of the traces in the first place.
Related Reading
Agent telemetry is how you find out which of these is actually your constraint
AI Code Review Bottleneck
High agent throughput with low merge rates points downstream, not at the agents
Context Engineering
Rising input tokens per task is a context problem, and this is how you see it
Monitoring
The infrastructure layer underneath - Prometheus, Grafana, and workspace SLOs
LLMOps
Model selection and prompt management, informed by what the telemetry shows
FinOps
Showback, chargeback, and budget policy for agent token spend
AI Governance
Policy for retention, audit trails, and what agents are permitted to touch
DevEx Metrics
The human-side measures that agent cost metrics deliberately ignore
Agentic Engineering
Designing and supervising the agents this telemetry describes