A model can perform well in a notebook while the product still gives poor answers. The production application may rewrite the question, choose a different provider, retrieve different passages, or reject the response after generation.
I want the main evaluation to enter through the same application path that a real request uses. Otherwise it can certify a component the customer never encounters in isolation.
Keep two layers of evaluation
End-to-end cases tell me whether the product behaves correctly. Diagnostic cases help identify why it does not. I need both.
For a support assistant, the end-to-end path can include tenant policy, model routing, retrieval, response generation, evidence validation, and handoff. A regression at any of those boundaries should be visible in the final outcome.
Diagnostic checks separately inspect retrieved sources, rewritten queries, extracted claims, and provider adapter behavior. They are useful for investigation, but I would not replace the product-level result with a collection of component pass rates.
Write expectations before reading the output
Each evaluation case should identify the customer intent, available evidence, permitted claims, prohibited claims, and expected action. Some cases should end with a clarifying question or a human handoff.
I include ordinary questions, unsupported questions, ambiguous references, outdated material, language changes, and adversarial instructions inside documents. The distribution should reflect the product’s risks without pretending to be a perfect copy of live traffic.
A case description such as “good answer about returns” invites subjective grading. “Must ask which product was ordered; must not promise eligibility” gives a reviewer a decision they can apply consistently.
Use automated judges where they help
Deterministic checks are useful for exact identifiers, allowed links, forbidden claims, and required structured fields. Human review remains important for semantic correctness and whether the response helps the customer.
A model-based judge can reduce review effort, but I treat its score as another measurement that needs calibration. I compare it with human labels across relevant case types and inspect disagreements instead of only measuring average agreement.
The rubric should penalize a confident unsupported answer even when the prose is excellent. It should also recognize unnecessary refusal as a product failure. A judge that rewards caution without usefulness can select a system that never answers.
Protect the test set from repeated tuning
I separate examples used to improve the pipeline from examples reserved for a later check. If every failure immediately becomes part of the tuning loop, performance on the same examples becomes less informative.
Small evaluation sets can still be valuable, but their results should be reported with their denominators and limits. A perfect result on a handful of negative cases is not proof that unsupported answers have been eliminated.
Repeated runs can reveal variability where it matters. I do not need to rerun every deterministic boundary many times, but unstable generation behavior deserves more than one favorable sample.
Compare changes on the same evidence
For a retrieval change, I compare the old and proposed pipelines on the same cases and source snapshot. I record the code version, prompt version, model configuration, and relevant corpus version.
My review table separates supported answers, unsupported answers, unnecessary abstentions, failed handoffs, latency, and cost. One blended score can hide a harmful tradeoff between those dimensions.
The release decision should name which failures block deployment. A cross-tenant disclosure or unauthorized action deserves a different gate from an awkward but accurate sentence.
After deployment, I turn reviewed production failures into new cases and keep a separate holdout for future changes. Evaluation then becomes a maintained description of the product’s obligations, grounded in the path the application actually runs.
Updated 25 September 2026.