Observability for AI features: latency, cost and quality

The question that stalls an on-call team in front of an AI feature is simple and uncomfortable: is the system working? In a traditional service, latency and error rate answer it. In an LLM feature, both can look perfect while the answer delivered to the user is wrong.

Three signals instead of one

An AI feature has to be observed along three axes at once, and none of them substitutes for another.

Latency answers whether the user waited too long. With text generation the useful metric is not only total time: it is time to first token, because that is what shapes the perception of responsiveness when you stream.

Cost answers whether the operation is sustainable. It is the axis that does not exist in a traditional service, and the one teams usually discover too late.

Quality answers whether the answer helped. It is the only one of the three that does not come out of a server counter, and it is the one that actually matters.

A dashboard showing only the first two gives a sense of control that does not match the state of the system.

Standardise attributes before inventing your own

OpenTelemetry maintains semantic conventions specific to generative AI systems, defining attribute names for the operation, the requested model, the responding model, input and output token counts, temperature and finish reason.

Adopt them even if you use a single tool today. Renaming an attribute after six months of history is expensive, and standardisation is what lets you swap observability backends without rewriting instrumentation.

The detail that pays off most: record the requested model and the model that actually answered as separate attributes. A vendor pointing an alias at a new version changes your application's behaviour with no deploy on your side, and without that pair of attributes the investigation starts in the wrong place.

Quality needs a proxy, because it has no counter

Since there is no counter for "good answer", quality is observed by approximation. Four proxies work well in practice:

• Explicit user signal, the thumbs up or down. Low volume, high value.
• Implicit signal, such as asking the same question again, abandoning the flow, heavily editing the suggested text or escalating to a human.
• Automated verification of whatever is verifiable: the JSON parses, the citation exists in the source, the value sits inside the possible range.
• Human-reviewed sample, small, continuous and always judged by the same criteria.

None of them is enough alone. Together they catch the silent degradation that no latency alert would ever fire on.

SLOs for an answer that varies

The SLO chapter of Google's SRE book starts from an idea that fits here well: pick a few indicators that represent the user experience and set a realistic target, not a perfect one.

For an AI feature, a reasonable starting set is: time to first token under a threshold for 95% of requests; hard failure rate under a ceiling; complaint or retry rate under a ceiling; average cost per interaction under a value.

The point of an SLO is not the tidy number on the dashboard. It is having an error budget that decides whether the next change can be risky or not. Without one, the discussion about shipping a new prompt version turns into a contest of opinions.

Cost belongs on the main dashboard

Cost per interaction, per route and per user should sit on the same dashboard as latency, not in a separate finance report someone opens at month end.

The reason is operational: a prompt change that doubles the context you send shows up as a step in the cost chart the same day. If that chart is only read thirty days later, the correlation with the deploy is already gone.

Log enough without creating a liability

Prompts and responses contain user data. Logging everything indefinitely creates a privacy liability that is not worth it.

The middle ground that works: always log metadata, meaning versions, tokens, latency, cost and the automated verification result; log content by sampling, with short retention and restricted access; and always log content when verification failed, because that is exactly where the investigation needs to look.

What this means for your team

If your AI feature is already in production without dedicated observability, start with the cheapest step: record prompt version, model version, tokens and cost on every call, using the OpenTelemetry attribute names. It is a few hours of work and it changes the nature of the next investigation.

Then pick one quality proxy and put it on the same dashboard as latency. Just one, whichever is easiest to collect in your product.

The 2025 DORA report reinforces a point worth closing on: AI adoption amplifies the capability that already exists. A team with good observability gets faster with AI. A team without it just finds out about problems later, with more code to review.