Agent Observability Doesn't Exist Yet — And the Unit of Analysis Is Why

In short: No platform is built natively for agent observability yet. Today's tools trace calls: prompts, tokens, latency, cost. An agent's meaningful unit is the run, and the questions that matter — why this tool, where the loop formed, which memory write flipped the decision, did the run hit its goal — are not derivable from any single span.

The category has a naming problem, and the naming problem is hiding a schema problem. Dashboards designed for one model call have been rebadged for systems that make forty of them, in an order nobody specified in advance. The mismatch surfaces the first time you have to explain a bad run to someone who was not watching it happen.

Why does most “agent observability” turn out to be LLM tracing with a new label?

LLM observability was built for a real job and does it well. It captures the prompt, the completion, the token counts, the latency, the cost, and whatever evaluations you attached. Point it at a chat endpoint and you can answer which prompt version got more expensive last Tuesday, which model slowed down, which output failed a rubric. Honest work — and work that keeps drifting toward the territory of testing, to the point where quality assurance and observability are two names for one job.

Agent debugging asks a different set of questions. In an r/LLMDevs discussion on agent observability from September 2026, the original poster set the two lists side by side: prompts, completions, tokens, latency, cost and evals on one side; on the other, why the agent chose this tool, why a task that should take five steps took fifteen, where the loop formed, which memory change flipped a decision, how multiple agents coordinated and failed, and whether the failure lived in reasoning, retrieval, the tools, or the model. Nothing in the first list produces an answer to anything in the second.

The sentiment reported in that thread is worth taking seriously because it is unusually consistent. Commenters describe adapting existing LLM tracing tools and building the agent-specific parts in-house — duct-taping, in their words. Several describe what is on the market as trace viewers with agent branding on the box. Treat that as reported practitioner opinion about a category, not as a measured verdict on any one product.

Vendors are not hiding the shape of it either. Arize on LLM observability for agents frames agent observability as a specialized form of LLM observability, aimed at systems that run multiple steps, hold memory and route between agents, and says plainly that tools tend to cover parts of the stack, so teams stitch several together. That is the category describing itself accurately. A specialization of a call-level tool is still a call-level tool.

What is the real unit of analysis for an agent — the call, or the run?

Here is the reframe the thread converged on, and it is sharper than any feature comparison. LLM observability is built around a call, or a chain of calls. An agent's meaningful unit is a run: one goal, one trajectory, however many steps it took to reach the goal or miss it. Whether a given run accomplished the thing it was launched to accomplish is not derivable from any single span, and no amount of span enrichment makes it derivable, because it is a claim about the whole path rather than about any point on it.

Once the unit changes, the gap stops looking like missing features. A trace is a tree of things that happened. A run is a decision graph — states, transitions, and the context that existed at the moment each transition was chosen. Those are different data structures. You cannot query the second out of the first, which is why observability keeps getting drawn as its own layer of an agent architecture rather than as an accessory bolted to the model client.

The questions from that thread map cleanly onto what a call-level trace can and cannot hand you.

The third column is the product specification, and none of it is exotic. Every entry is a structure someone has to record deliberately, at the moment it is true. None of it can be reconstructed afterwards from spans that were never asked to carry it. Loops make the point best: two iterations that look byte-identical in a trace mean entirely different things depending on what changed in memory between step three and step seven. No one in the thread had seen a tool treat a run as a decision graph with memory diffs as first-class citizens.

None of this is an argument against evaluation. The JetBrains framing of LLM evaluation versus agent monitoring puts the two as complementary: evaluation tests capability before and during deployment, while live observability watches an agent's reasoning and operational health once it is running. You want both. The narrower claim is that the live half is currently instrumented at the wrong altitude.

What would a platform actually built for runs have to do?

The thread produced a short list of primitives. Each one is a schema decision rather than a dashboard feature, and each one is missing from the category as a shipped default.

Memory diffs as first-class citizens

A run's memory is not a static blob attached to the trace header. It mutates, and the mutations are the story. Recording each write as a diff — key, before, after, the step that caused it — is what turns an identical-looking loop into a readable one, because you can see the state that made step seven behave unlike step three. Memory is the least standardized part of the stack right now, which is exactly why it needs recording rather than assuming; the differences in how Claude, Cursor, Copilot, Codex and LangChain remember are wide enough that no vendor default will fit your system.

Run-to-run divergence diffs

Two runs of the same task, aligned step by step, with the exact point of divergence marked. Not two flat traces open in two browser tabs while you scroll for the difference. Existing tools are built around single-run inspection, so the comparison work lands on the engineer at the worst possible moment.

Counterfactual replay

Rerun the same trajectory with one variable changed — a tool result, a memory write, the model — and show where the paths split. Traces show what happened. Controlled replay is what starts to show what caused it. This one is a proposal from practitioners rather than a shipped feature, and it collides head-on with the deterministic paradox: replay is only informative if you can hold everything else still, and holding a stochastic system still is its own engineering project.

Goal verdicts, side effects and drift

Three things per run that no span carries. A verdict against the goal, which realistically is an assertion you write rather than something inferred from the trace. A diff of what changed outside the agent: files touched, rows written, side-effecting API calls made. And drift across runs, which per-run tracing cannot see by construction — a system where the average trajectory has quietly grown by four steps since March looks perfectly healthy one run at a time.

Parts of this exist. Langfuse on AI agent observability describes capturing every step — LLM calls, tool invocations, retrievals, control-flow decisions — as structured traces, and distinguishes the tools that were available from the tools actually invoked. It documents code evaluators for things like whether a required tool was called and whether the step budget was respected, LLM-as-a-judge evaluators for task completion, and human annotation queues for judging whether a trajectory was reasonable. That is genuinely step-aware tooling. It remains organized around inspecting a run, one run at a time.

The shared vocabulary is further behind than the products. Every gen_ai.* attribute, span, metric and event in the OpenTelemetry semantic-conventions registry carries a stability of Development rather than Stable, per the OpenTelemetry GenAI convention status. In June 2026, with semantic-conventions v1.42.0, the GenAI, provider-specific and MCP conventions moved out to a dedicated open-telemetry/semantic-conventions-genai repository, which as of August 21, 2026 had produced no tagged release.

That does not make the conventions useless — a six-layer walk through how OpenTelemetry traces agent reasoning and MCP tools shows how much surface area has already been sketched. It does mean the shape is still moving, and that a run-level schema is not going to arrive from the standards track this year.

What should you instrument yourself while the category catches up?

Nobody is going to hand you this in the next two quarters. The conventions repository has no release, the products are organized around single traces, and a common vocabulary for AI observability was still unreleased as of August 21, 2026. Meanwhile the engineers in that thread are already writing the missing layer by hand, which is the clearest available signal about where the line falls between what you buy and what you own.

1. Write the per-run goal assertion. One function per task type, returning pass or fail with a reason. It is the only field that makes a run queryable as a success or a failure, and no vendor can write it for you because it encodes your definition of done.
2. Log every memory write as a diff, stamped with the step index that caused it. Key, before, after, step. This is a few lines at the memory-layer boundary and it is the difference between reading a loop and guessing at one.
3. Emit a per-run side-effect manifest: files touched, rows written, external calls that changed state somewhere else. Incident review starts with blast radius, and the trace does not know what a write is.
4. Keep a stable run id and a stable task id across deploys. Without them you cannot compare this week to last month, and drift is invisible until it is an outage.

All four are cheap while your agent surface is small and expensive to retrofit once it is not. Treat observability as a ring in the 2026 agentic architecture — designed in with the rest of the system, not purchased after the first bad week in production.

Key takeaways

• LLM observability instruments the call. Agent debugging is a question about the run. Different unit, different schema.
• Whether a run accomplished its goal is an assertion you write, not a field a trace emits.
• Identical-looking loops differ by what changed in memory between steps, so memory diffs are the primitive most worth adding first.
• Every gen_ai.* convention in OpenTelemetry was still marked Development, with no tagged release from the new repository as of August 21, 2026. Do not wait for the standard.
• Four things to instrument this quarter: the goal assertion, memory diffs, a side-effect manifest, and stable run and task ids.

FAQ

What is the difference between agent observability and LLM observability?

LLM observability instruments the call: prompt, completion, tokens, latency, cost, plus whatever evaluations you attach to it. Agent observability has to instrument the run: which tool was chosen and on what basis, how control flow branched, what changed in memory, what changed outside the system, and whether the goal was met. Arize positions agent observability as a specialized form of LLM observability for multi-step, memory-holding, multi-agent systems, and JetBrains positions evaluation and live observability as complementary rather than interchangeable. Both framings are right; both still inherit the call as the unit of record.

Why can't traces tell me why an agent picked one tool over another?

A span records the call that was made, not the context state that existed when the choice was made. The intermediate reasoning, the retrieved chunks sitting in the window, the memory snapshot at that step — if none of it was captured at decision time, you are inferring cause from outcome. That inference tends to fail on exactly the runs you care about, which are the strange ones.

What is counterfactual replay and why does it matter for agents?

Rerun the same trajectory with exactly one variable changed — a tool result, a memory write, the model — and show where the two paths split. Traces show what happened; controlled replay is what starts to isolate cause. Worth stating plainly: this is a primitive proposed by practitioners in the discussion, not a feature available off the shelf today.

Is there a standard for tracing AI agents yet?

Not a stable one. Every gen_ai.* attribute, span, metric and event in the OpenTelemetry registry is marked Development. The GenAI, provider-specific and MCP conventions moved to a dedicated repository in June 2026 with semantic-conventions v1.42.0, and that repository had produced no tagged release as of August 21, 2026. Adopt the attribute shapes if you like them, but pin your own schema and expect renames.

Should we build our own agent observability layer or buy one?

Split it rather than choosing: buy the tracing substrate, own the run-level assertions, memory diffs and side-effect log, because those encode your definition of a successful run and no vendor can write them for you. The full version of that argument lives in when to stop building your own AI agent eval framework.

If you are drawing that line this quarter, start with the build-versus-buy question for evaluation — it splits along the same seam this post draws for observability, and getting one right usually settles the other.