LLM Observability: A Buyer's Guide to Watching the AI You Built
The fourth of five guides to the categories called 'AI monitoring'. This one tells you whether your own AI feature is working and what it costs — and sees exactly nothing your developers' coding tools do.
This category answers two questions about software you wrote: is the AI feature behaving, and what is it costing. It is the most mature of the five and the easiest to adopt, which is also why it is the one most often mistaken for covering everything.
Who asks it, and when
The team running an AI feature, usually about a week after the first customer complaint that the assistant said something strange. The cost question arrives on the same day as the first surprising invoice.
What the tools actually do
- Trace every model call your application makes: inputs, outputs, latency, token cost, and the chain of steps that produced it.
- Version prompts as deployable assets, with history and rollback.
- Evaluate quality — datasets, scorers, LLM-as-judge, regression runs between versions.
- Report spend per feature, per user, per model.
Langfuse, LangSmith, Helicone, Braintrust, PromptLayer and Datadog's LLM observability all live here, with different centres of gravity: tracing, evaluation, prompt management, or integration with telemetry you already run. Gateways like LiteLLM sit next door and are often paired with one.
How to evaluate one
- How does data get in — SDK, OpenTelemetry, or a proxy you point your base URL at? This decides your coverage more than any feature does.
- What does it cost at your volume? Per-trace pricing on a chatty agent is a different product from the same tool on a low-volume feature.
- Can you self-host? Several here can, fully. If prompts carry customer data, that answer may be the whole decision.
- Does evaluation fit your workflow, or is it a second system nobody runs? An unused eval suite is worse than none, because it looks like coverage.
The gap, stated precisely
Observability sees what your systems do. A developer running Cursor is not one of your systems: no agent on that path, no span emitted, no instrumentation point to add.
This is not a shortcoming of any product in the category — it is what the category is. Your application's calls are visible because you instrumented them. Cursor talks to Cursor's backend, Claude Code to Anthropic, Copilot to GitHub, all under a vendor's client and the developer's own subscription. No configuration change routes that through your pipeline.
The check takes a minute: compare the users in your observability project against the number of engineers holding a Cursor or Claude Code subscription. If those describe different populations, you have two AI estates and are measuring one.
What it will not cover
A trace ends at the model's output. It does not continue into the repository, so it cannot tell you which lines of which file that answer became. For an AI feature, that boundary is correct and irrelevant. For the question of how your codebase came to exist, it is the entire thing that is missing.
The sequencing question
If you run AI in production, buy something in this category — the arguments against are all bad. Then note what you did not buy: the AI writing the code is a different estate, with different artefacts, and in most engineering organisations it is now the larger share of the tokens and by far the larger share of the consequences.