LLM Evaluation: A Practical Framework for RAG and AI Applications

Evaluate the system in layers

For RAG, start with retrieval. If the correct evidence is absent from the candidate set, generation cannot recover reliably. Next evaluate whether the answer is supported by that evidence. Then evaluate completeness, relevance, citation correctness, and refusal behavior.

For agents, add trajectory-level metrics: did the system select the right tool, pass valid arguments, stop when the task was complete, and avoid unsafe side effects?

• Retrieval quality.
• Groundedness and answer quality.
• Tool selection and argument validity.
• Safety, latency, and cost.

Build a useful evaluation set

A small representative dataset is more useful than hundreds of synthetic questions that never resemble production. Sample real query patterns, include easy and difficult cases, and deliberately include failures: ambiguous queries, missing evidence, stale documents, conflicting sources, and multi-turn references.

Each example should have enough metadata to explain the expected behavior. For retrieval, record relevant document or chunk IDs. For generation, define what a correct answer must contain and what claims must not be made.

Choose metrics that answer engineering questions

Recall at k answers whether the evidence enters retrieval. Precision-oriented ranking metrics help assess whether the top of the list is useful. Groundedness checks whether claims are supported. Citation validation checks whether cited sources actually exist in the retrieved set. For agents, success rate, unnecessary steps, cost per successful task, and tool error rate are useful operational measures.

Do not optimize a metric in isolation. A system can improve recall by retrieving everything and then become slower and more expensive. Evaluation should expose those trade-offs.

Regression testing for model changes

Model providers update models, embeddings change, prompts evolve, and indexes are rebuilt. Maintain a versioned regression set and run it whenever a production dependency changes. Compare both quality and operational metrics. Pin model versions where possible and gate upgrades on acceptable regression thresholds.

Store the exact prompt configuration, model identifier, retrieval configuration, and tool versions associated with each evaluation run so results remain reproducible.

Production monitoring

Offline evaluation is not enough. Monitor live no-hit queries, unsupported claim rate, refusal rate, latency percentiles, token usage, cost, tool failures, and user feedback. Sample traces for human review and turn recurring failures into new evaluation cases.

A healthy evaluation program is a loop: observe → classify failure → add or update a test → change the system → rerun regression → deploy → observe again.