Information in this post reflects publicly available sources as of October 18, 2026.
A customer writes in, angry. Your support assistant told them their refund had been approved. It had not. They want to know why, and so does your manager.
You open the logs. They show the request arrived at 14:02, returned HTTP 200, and took 6.8 seconds, and nothing more. You cannot see what the model was asked, which documents it retrieved, which tools it called, what those tools returned, or which version of the prompt was live that afternoon. You try the same question yourself and get a correct answer, which proves nothing.
Traditional logs record that a request happened. AI features need a record of why the answer came out the way it did. Building that record is observability for AI applications, and without it every bad answer is a mystery you cannot solve.
Why ordinary monitoring is not enough
Conventional monitoring is built around a clear idea of failure: errors, timeouts, crashes. AI features mostly fail in a different way. The request succeeds, the latency is fine, the status code is 200, and the answer is wrong. Nothing in a standard dashboard turns red.
Three more properties make it harder. Outputs are non-deterministic, so replaying a request later may not reproduce the problem. A single answer often involves many steps, such as several model calls, a retrieval, and a few tool calls, and the mistake can start in any of them. And cost varies per request, so a change in behavior can show up first as a change in spending.
The answer is to capture, for each request, a structured record of every step the system took and what each step saw and produced. That record is a trace.
How it works
Traces and spans
The idea comes from distributed tracing, which most backend engineers already know. A trace represents one request from start to finish. It is made of spans, one for each unit of work, nested to show which step called which. For an AI feature, the spans are model calls, retrievals, tool calls, and agent steps, each carrying its inputs, outputs, token counts, and timing.
With this trace, the refund mystery takes minutes instead of days. The tool correctly reported the refund as pending. But retrieval also returned a chunk from a 2023 policy, back when refunds under $50 were approved automatically, and the model trusted the policy over the tool. The fix belongs in the retrieval index, not the prompt, and you would never have found it from a status code.
The waterfall also answers questions nobody asked yet. Most of the 6.8 seconds went to the final answer, which generated 900 tokens for a reply that needed about 150. That is a latency problem and a cost problem, visible at a glance.
What to record
Each span should capture enough to understand and reproduce that step. In practice, that means:
| Field | Why it matters |
|---|---|
| Model and parameters | Behavior changes when the model version, temperature, or reasoning effort changes |
| Prompt template and version | Lets you connect a change in quality to the prompt change that caused it |
| Input messages and output | The only way to see what the model actually saw and said |
| Finish reason | Separates complete answers from truncated ones and refusals |
| Token counts and cost | Attributes spending to features, users, and steps |
| Tool calls, arguments, and results | Shows whether a tool failed or the model misused it |
| Retrieved document IDs | Shows whether the right context reached the model |
| User feedback, linked to the trace | Connects a complaint to the exact request behind it |
The prompt version is the field teams most often forget, and the one they miss most. Without it, a quality drop on Tuesday cannot be tied to the prompt change deployed on Tuesday morning.
The cleanest fix is to treat prompts as versioned artifacts rather than strings buried in code. Store each template with an identifier and version, render it at request time, and record both on the span. Many observability platforms include prompt management for exactly this reason: the same tool that shows a quality dip can show which prompt version was live and what changed in it.
The OpenTelemetry conventions
You do not need to invent field names. OpenTelemetry, the open standard most backend tracing already uses, defines semantic conventions for generative AI. Spans carry attributes such as gen_ai.operation.name, gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.response.finish_reasons. Agent runs are modeled as a tree, with an invoke_agent span as the parent of chat and execute_tool spans, exactly as in the waterfall above.
A manually instrumented model call looks like this:
from opentelemetry import trace
tracer = trace.get_tracer("support-assistant")
def call_model(messages, model, prompt_version):
with tracer.start_as_current_span(f"chat {model}") as span:
span.set_attribute("gen_ai.operation.name", "chat")
span.set_attribute("gen_ai.provider.name", "openai")
span.set_attribute("gen_ai.request.model", model)
span.set_attribute("app.prompt.version", prompt_version) # your own attribute
response = client.chat.completions.create(model=model, messages=messages)
span.set_attribute("gen_ai.response.model", response.model)
span.set_attribute("gen_ai.usage.input_tokens", response.usage.prompt_tokens)
span.set_attribute("gen_ai.usage.output_tokens", response.usage.completion_tokens)
span.set_attribute("gen_ai.response.finish_reasons", [response.choices[0].finish_reason])
return responseBecause this is ordinary OpenTelemetry, the span joins the same trace as your web request, database queries, and downstream services. You rarely need to write it by hand: instrumentation libraries such as OpenLLMetry and OpenInference patch the common SDKs and frameworks to emit these spans automatically, and agent frameworks increasingly emit traces out of the box.
Two caveats apply. As of mid-2026, every generative AI convention in OpenTelemetry is still marked as in development, so attribute names can change between releases. And capturing the actual prompt and response content is opt-in by design, for reasons covered under what goes wrong.
From traces to signals
Individual traces explain individual failures. To notice failures in the first place, you need aggregate signals built from those traces, and they come in three layers.
Health signals are the familiar ones: latency, error rates, rate limiting, and cost per feature. They catch outages and runaway spending. Most teams already have them.
Behavior signals are specific to AI and cheap to compute from trace data.
A rising share of responses that stop because they hit the token limit means answers are being cut off. A jump in refusals often follows a prompt or model change. More steps per agent task suggests the agent is struggling, and retrieval that suddenly returns nothing points to an indexing problem. None of these show up as errors.
Quality signals are the hardest and the most valuable. Run an LLM judge, built and calibrated as in the evals post, on a sample of production traces. Track thumbs-down rates and how often conversations escalate to a human. When any of these moves, the traces behind it tell you why.
Alert on change, not just on thresholds. A jump in truncated responses right after a deploy, cost per request doubling overnight, a spike in refusals after a model upgrade, or a drop in judge scores for one category are all worth a page. Each of them usually traces back to a specific change, and catching it within hours rather than weeks keeps the damage small.
From a trace to a fix
The payoff comes when observability connects to the rest of the development loop. A complaint leads to a trace, the trace leads to a root cause, and the root cause becomes a permanent test.
Notice where the eval case comes in: before the fix. Writing the test first proves it catches the bug, and then proves the fix works. That is the same flywheel from the evals post, with traces as its main source of new cases.
When to invest, and how much
Start on day one, even in a prototype. During development, capture full traces of everything: they are the fastest way to understand what a model or agent is actually doing, and reading traces is how most teams discover their first real bugs. It costs almost nothing at prototype volume.
Keep reading traces after launch, too. A short weekly review of a random handful of production traces, plus the ones with negative feedback, surfaces problems no metric was designed to catch: an agent taking a strange detour, a tool returning far more data than needed, or users asking for something the feature was never built to do.
In production, scale the detail to the risk. Capture metadata such as tokens, latency, model, prompt version, and finish reason for every request. Capture full content for a sample of traffic, for every error, and for every request that gets negative feedback. Add quality signals once you have enough traffic for the numbers to mean something.
Agents need tracing more than anything else. A single-call feature can sometimes be debugged from its input and output. A ten-step agent, or the multi-agent systems from the earlier post, cannot be debugged at all without a trace that links every step to its parent task.
What goes wrong
Traces leak sensitive data. Prompts contain whatever users type and whatever your tools return: names, addresses, account numbers, sometimes credentials. Storing full content creates a second copy of sensitive data, often with weaker access controls than the primary system.
Redact or mask sensitive fields before export, restrict who can view raw content, and set retention limits. This is why content capture is opt-in in the OpenTelemetry conventions.
Sampling misses the failures. Keeping a random 1% of traces means the one that matters is usually gone. Sample with intent: keep every trace with an error, a refusal, a truncation, negative feedback, or an unusual cost, and sample the rest.
Feedback is not linked to traces. A thumbs-down stored in a product analytics tool, with no trace ID, tells you a user was unhappy and nothing about why. Pass the trace ID to the front end and attach it to every feedback event.
Dashboards watch only health. Latency and error charts stay green while answers get worse. Add at least one behavior signal and one quality signal per feature, and alert on sudden changes, not only on thresholds.
Replays do not reproduce. Re-running a traced request can produce a different answer, because outputs vary and the retrieved documents or tool results may have changed since. Debug from what the trace recorded, not from a fresh run, and store tool results and retrieved content in the trace for exactly this reason.
Cost and complexity
AI traces are large. A single agent request with full prompts, retrieved documents, and tool results can easily reach 20 KB. At a million requests a day, full capture means about 20 GB a day, or 600 GB a month, plus the cost of indexing and searching it.
Sampling changes that sharply. Keeping about 1 KB of metadata for every request and full content for 10% of them comes to about 3 GB a day, roughly 90 GB a month, while still keeping every error and every complaint in full. Quality signals add model costs too: running a judge on 1% of traffic is a small addition to the bill, while judging everything can rival the cost of the feature itself.
The integration work is modest if you build on standards. OpenTelemetry instrumentation libraries cover the major SDKs, and most observability platforms accept OpenTelemetry data, so the instrumentation does not lock you into a vendor.
The current landscape
On the standards side, the OpenTelemetry generative AI semantic conventions define spans, attributes, and metrics for model calls, tool calls, and agents, though they remain in development status as of mid-2026. OpenLLMetry from Traceloop and OpenInference from Arize provide instrumentation for the common SDKs and frameworks, and the OpenAI Agents SDK, among other frameworks, includes built-in tracing.
For storage and analysis, Langfuse is open source under the MIT license for its core and can be self-hosted; ClickHouse acquired it in January 2026 and said it would remain open source. Arize Phoenix is source-available and OpenTelemetry-native, and Helicone, open source under Apache 2.0, works as a gateway that logs traffic when you change your API base URL. LangSmith, Braintrust, and W&B Weave combine tracing with evals, and Datadog LLM Observability adds AI spans to existing Datadog dashboards.
Most of these overlap with the eval platforms from the evals post, and that convergence is the clearest trend. Tracing, evaluation, and datasets are becoming one workflow, because the traces you collect in production are the best source of the test cases you run before the next release.
Summary
AI features fail quietly. The request succeeds, the dashboard stays green, and the answer is wrong. Observability for AI means recording not just that a request happened, but every step it took, what each step saw, and what each step produced, so that any bad answer can be traced back to its cause.
Build it early and on open standards. Capture traces with prompt versions and token counts, link user feedback to them, watch behavior and quality alongside health, and turn every investigated failure into an eval case. Then a complaint stops being a mystery and becomes the start of a fix that stays fixed.
Part of the Applied AI series: AI from the builder's side.