Observability for AI systems gets framed as a tooling problem. It is not, mostly. It is a problem of agreeing what “working” means before the system is in production. By the time someone is reaching for a dashboard, the wrong things are being asked about the system.
Three signals carry most of the weight. Pick them on day one, name them clearly, and make sure someone owns each one.
1. The model did what the contract said it would
Structured outputs make this easy: validate the output shape, count the failures, and set a budget. Free-form outputs make it hard, but you can still pick a few specific properties that must hold (mentions a known entity, picks an option from a known set, refuses cleanly when asked to do something out of scope). Log the property check, not the raw prompt and response.
2. The plan was followed end-to-end
If the system has a planner, the only thing that matters at runtime is whether the plan executed. Number of completed steps over number of intended steps. When that ratio drops, something upstream of the model changed: a tool, a permission, an input format. The signal points at the right floor.
3. The cost of being right
Cost per resolved task, not cost per call. Tokens per resolved task. Latency for the user-facing turn. When one of these gets worse without an answer-quality reason, you have a regression you can act on. When they stay flat through a model swap, you have evidence the swap was worth it.
Everything else is downstream
Most of the dashboards we end up building are decorative versions of these three signals. There is no shame in starting with three log lines, named clearly, and getting the dashboard later when someone other than the original team needs to read them.