GenAI Telemetry: Instrument the Turn, Not the Model Call

The dashboard said the model was fine. p95 on the chat span was 2.4 seconds, token usage flat week over week, error rate under one percent. The client's support team said the assistant took "about half a minute" to answer anything. Both were true. There were roughly eight thousand spans a day in that trace store and not one of them covered the interval the user was sitting through.

This is the most common instrumentation failure we walk into on LLM products now, and it is never caused by missing telemetry. It is caused by telemetry that describes the SDK instead of the product. Auto-instrumentation hooks the client library, so you get one span per model call — correctly named, correctly attributed, genuinely useful for the question it answers. Meanwhile a single user turn in that product was 4.1 model calls on average, two tool executions, one retry, and about three seconds of our own code between them.

The OpenTelemetry GenAI semantic conventions have made the vocabulary part of this a solved problem. There is now an agreed way to say model, operation, token counts and finish reason across three signal types, and it works whether you are calling a hosted API, a coding agent, or something you run yourself. What the conventions do not do — cannot do — is decide the shape of your trace. That is an architecture decision, and it is the one nobody makes on purpose.

The model call is the wrong unit

A chat span is a faithful record of one request to one provider. It is the right unit for exactly one audience: the people debugging the provider integration. It is the wrong unit for everyone else, because almost every question that gets asked about an LLM feature in production is a question about a turn.

• “What does this feature cost per conversation?”— token attributes live on the call. Summing them needs a parent to sum them under. Without one, cost lands on the model name, which tells you which model is expensive and nothing about which product surface is spending.
• “How long did the user wait?”— the sum of the call durations is not the answer, and neither is the max. On the product above, the four calls added up to 8.4 seconds inside an 11.4-second turn. The missing three seconds were retrieval, schema validation, and one synchronous write we had forgotten was on that path.
• “Is the retry path healthy?”— a call that succeeded on the third attempt and a call that succeeded on the first are two indistinguishable spans unless something above them knows they were the same attempt at the same thing. Retry rate is a turn-level property that people keep trying to measure at call level.

None of that is an argument against per-call spans. Keep them. The argument is that they are leaves, and a tree with only leaves is a pile.

The span we add before anything else

The conventions already have a name for the unit we want. Alongside chat, embeddings and execute_tool there is invoke_agent — a span that covers an agent doing a piece of work rather than a client making a request. Almost nobody emits it, because auto-instrumentation cannot: only your code knows where a turn starts and ends. So that is the first thing we write, before touching a single dashboard.

Once that parent exists, the rest of the tree mostly builds itself. Auto-instrumentation attaches the chat spans to whatever is active, tool wrappers give you execute_tool, and retries stop being anonymous because they are now siblings under one turn. The work is not volume. It is deciding where the boundary goes and then being disciplined about propagating context across every async hop, which in practice means one wrapper that everything on the path is required to go through.

Attributes that earn their storage

Two rules on the custom attributes, both learned the expensive way. Keep them in a namespace you own, so a future version of the conventions can never collide with them. And keep every one of them low cardinality — the tenant id goes on, the user id does not, the conversation id lives in the trace id where it belongs. We once put a request fingerprint on a turn span and turned a metrics backend into a per-request database over a weekend.

Content capture is a second pipeline, not a boolean

The conventions capture no prompt content and no tool arguments by default. Model name, operation, token counts, finish reason, duration — all metadata, none of it a customer's words. Turning content on is a single environment variable, and that switch is the most consequential line in the whole setup, because it silently reclassifies your observability stack.

The moment prompts flow into traces, a system built for operational data starts holding support tickets, contract clauses, medical questions and pasted credentials. It usually has a broader access list than your production database, a retention policy nobody chose deliberately, and a vendor in a region your DPA does not name. None of that is a reason to never capture content — you cannot debug a bad answer from token counts. It is a reason to treat capture as its own pipeline with its own decisions.

• Separate exporter, separate backend. Content events do not go where spans go. Different endpoint, shorter retention, an access list that is reviewed like database access is reviewed. If a single config flag can put customer text in front of the whole engineering org, the flag is the vulnerability.
• Sample, don’t switch.On is a bad default and off is a bad debugging story. We capture a small fixed percentage plus anything on an errored or flagged turn — which is where the interesting content lives anyway. Turn-level sampling matters here: sampling calls independently gives you half a conversation and no way to reconstruct the rest.
• Redact at the SDK, not the backend.Backend redaction means the raw text already crossed the network and already landed on somebody else’s disk. The processor that strips it has to run in your process, before the exporter, and it has to be tested like the rest of the code.
• Write down what the capture is for. Every content pipeline we have inherited was switched on to debug one incident and never switched off. Give it a stated purpose and an expiry date in the same commit.

One more thing that surprises people: the metrics side of the conventions is where most of the operational value actually sits. Token usage and operation duration as histograms, dimensioned by model and operation, cost a fraction of what traces cost and answer the majority of the recurring questions. We keep traces sampled and metrics complete, not the other way round.

What has to be true before we call it instrumented

This is the checklist we run at the end of the work. It is deliberately about questions the telemetry can answer, not about which packages are installed — a project can have every GenAI instrumentation library on npm and fail all six.

That last one is the reason to use the conventions at all rather than inventing your own field names. Portable attributes mean a provider swap is a config change and not a dashboard rewrite, and on the two products where we have since changed models, it was.

If you only do one thing from all of this, add the parent span. Not the metrics, not the redaction processor, not the vendor evaluation — the parent span. It is under a hundred lines, it makes every span you already emit more useful without changing any of them, and it is the piece that no library will ever add for you, because only your code knows where the turn begins.