Observability for a Production LLM Feature

Design observability for a production LLM feature. The core problem: quality degradation raises no error. The system returns 200 and a bad answer.

Feature: [describe] Volume: [calls per day] What happens to the output: [displayed / feeds a decision / stored]

Define:

1. What to log per call — request id, prompt version, model, input and output tokens, latency, cost, and a hashed user id. Do not log: privileged document content, keys, identifying information.
2. Quality metrics without ground truth — refusal rate, malformed output rate, empty response rate, anomalous output length, retry rate, and user abandonment after a response.
3. User signals — copy, edit-before-use, rating, and re-asking the same question rephrased. The last is the strongest signal of a bad answer.
4. Alerts — for each metric, the threshold and window. Separate what wakes someone at night from what appears in a daily report. Cost and latency: page immediately. Quality: daily trend.
5. Sampling for manual review — how many, selected by what criterion, reviewed by whom. Prefer sampling biased toward suspicious cases over uniform random sampling.
6. Drift detection — how to distinguish a change in model behaviour from a change in the kinds of questions being asked. These look identical on a graph.
7. What to retain for debugging — for how long and at what granularity, given privacy constraints.

Finally: the first question you will be asked when a user reports a wrong answer, and the data that must be available to answer it.