LLM System Evaluation 101: Make Your Agent Cheaper, Faster, and Better

Most AI agents that fail in production do not fail because the underlying model is bad. They fail because the team building them had no way to know they were failing until a real customer found the problem first. Madhuranjan Kumar has seen this pattern across enough client setups to treat it as close to universal: the demo works cleanly, the first few weeks feel promising, and then a user phrases something in a way nobody anticipated and the whole thing breaks in a way nobody notices for longer than it should. Production breaks in unpredictable ways, and the only real defense is a measurement system built before the surprises arrive.

What follows is a set of seven practices that separate AI agent setups that stay reliable as volume grows from the ones that quietly degrade until someone complains loudly enough for the team to investigate.

1. Measure the thing that breaks most, not the thing that is easiest to measure

The first instinct when building an evaluation system is to measure what is easiest to measure, usually something like response length, keyword presence, or latency. These are easy to compute and they tell you almost nothing about whether the agent is actually doing its job correctly.

The right approach is to spend time watching the system fail before deciding what to measure. Run a batch of real or realistic user inputs through the agent, read every output carefully, and find the failure pattern. For a retrieval-based system, the thing that breaks most is usually one of two things: the retrieval step pulls the wrong information, or the generation step ignores what retrieval found and produces an answer that was not grounded in the retrieved content. Both of these are specific and measurable once you name them. Retrieval relevance measures whether the documents that came back were actually about the question being asked. Faithfulness measures whether the final answer stayed grounded in what was retrieved rather than adding claims from somewhere else.

For a restaurant using an AI agent to take reservations and to-go orders, the thing that breaks most is capture completeness. The agent needs to record party size, date and approximate time, dietary restrictions, and order items. If any of those four fields are missing or recorded incorrectly, the reservation causes a real operational problem downstream. So that is what gets measured. Not whether the agent sounded friendly or kept its reply under two hundred words. Whether it captured the four required fields correctly every time.

2. Build your own benchmark before you trust any public leaderboard

Public leaderboards measure general capability on general tasks. They are useful for a rough sense of which models are capable, but the task they measure is not your task. When teams run their own evaluation on their specific system with their specific inputs, the ranking frequently inverts from what the leaderboard suggested. A cheaper and faster model outperforms an expensive one on a specific business task more often than people expect, because the leaderboard task and the actual business task are rarely measuring the same thing.

The practice is straightforward: define your task clearly, collect a set of representative test cases from your actual domain and your actual user base, run every candidate model through your evaluation on those cases, and compare the results directly. The model that wins on your evaluation is the right one to use, regardless of where it sits on any general leaderboard. This single practice routinely reveals that a team can achieve the same accuracy, or better, at meaningfully lower cost per call, and those savings compound significantly at production volume.

For the restaurant, this means testing the reservation capture task specifically on realistic messy inputs from real customers, not on the idealized examples the development team wrote. The model that correctly captures all four required fields on ninety percent of realistic messy inputs is the right model for the restaurant, whether or not it appears at the top of a reasoning benchmark.

3. Ask for a verdict, not a score

When building an LLM evaluator, the natural instinct is to ask it to score outputs on a numeric scale. A score out of five, or a score out of ten, feels like a meaningful quantitative signal. In practice, it is not. Models assign different numeric grades to identical outputs across different runs, and the difference between a six and a seven carries no consistent meaning when it varies unpredictably from one call to the next.

Classification is stable where numeric scoring is not. Ask the evaluator for a clear categorical judgment: did the agent capture all four required fields correctly, yes or no. Did the retrieved document actually address the question that was asked, relevant or not relevant. Is the final answer grounded in the retrieved content, or does it add claims that were not in the source material, faithful or hallucinated. These binary and multi-class classifications are far more consistent across evaluation runs and far easier to aggregate into an interpretable metric.

The practical payoff is that your evaluation result becomes a percentage: ninety-one percent of reservation captures were complete and correct. That number is interpretable, comparable across model versions, stable across evaluation runs, and actionable. A score out of ten is none of those things. When someone asks whether the last prompt change made things better or worse, a pass rate answers the question clearly. An average score does not.

4. Calibrate your evaluator against the cases that already fooled you

An evaluator that agrees with human judgment on clean, easy cases is not a validated evaluator. The cases that matter are the tricky ones: the edge cases that look almost correct but have a subtle error, the ambiguous inputs that reasonable people might handle differently, the messy phrasing that comes in from real customers rather than from a developer writing test cases in calm conditions.

The calibration process is to build a small golden dataset of cases with known correct judgments, including a meaningful proportion of the hard cases from real production logs or realistic simulations of them, and then run the evaluator against that set. Measure how often the evaluator agrees with your own human judgment on every case in the set. Where it disagrees, read the disagreement carefully, adjust the evaluator's instructions or criteria, and rerun. Repeat this loop until the evaluator matches human judgment on the hard cases as consistently as it does on the easy ones.

For the restaurant, this means the golden dataset should include inputs like: party of four, Friday, two are kids and one is gluten-free, can we do seven-ish. And a to-go order where someone writes the usual plus extra sauce, hold the pickles on mine. Those are the inputs that break things in real operation, and that is exactly where the evaluator needs to hold up. A golden dataset that contains only clean examples will produce an evaluator that looks accurate on the calibration set and fails under real conditions.

5. Trace every production run from day one

Without a trace on every production run, you cannot replay what happened when something goes wrong. You cannot see which step of a multi-step agent caused the failure. You cannot build a growing test dataset from real traffic. And you cannot compare what one model version did versus another on the same input. Tracing is not a debugging feature you bolt on after the first major failure. It is the infrastructure that everything else in the evaluation process depends on.

The mechanics depend on the stack, but the principle is consistent across every setup: every agent step, every tool call, every model call, and the inputs and outputs at each stage should be logged with enough context to reproduce and inspect the run afterward. Good tracing tools reduce the instrumentation cost to adding a decorator or a wrapper around the model client that handles the logging automatically. The cost of doing this from the start is low. The cost of not doing it becomes apparent the moment the first production failure needs to be diagnosed.

For the restaurant, tracing means that when a reservation comes back missing the party size, the team can open the trace immediately, see the exact conversation turn where the field was not captured, see what the retrieval step returned if a knowledge base was involved, and understand specifically where the process broke down. That diagnosis takes minutes instead of hours of manual reconstruction from memory and partial logs.

6. Turn your trickiest production logs into test cases

The most valuable test cases in your evaluation system do not come from brainstorming sessions or from a developer imagining what users might say. They come from production logs. Real users phrase things in ways that nobody on the building team anticipated, and every time that happens and the agent handles it poorly, you have a new test case that would have been nearly impossible to construct in advance.

The practice is to review production logs on a regular cadence, pull out the cases where the agent struggled or produced incorrect output, and add them to the golden dataset with the correct judgment recorded. Over time this turns the benchmark into a living record of the actual distribution of user inputs rather than the theoretical distribution the team imagined during development. An agent that passes a benchmark built from real production failures is materially more reliable at scale than one that passes a benchmark built from idealized examples that real users never actually send.

For the restaurant, this means that every botched reservation and every incomplete order capture becomes a permanent addition to the test set. The agent that handles all of those correctly is the one that has genuinely learned the specific ways real customers phrase their requests in this particular context, not a generic approximation of what restaurant customers might say in the abstract.

7. Compare models on your task before you commit to any tier

Many teams pick the most expensive frontier model by default, assume it performs best, and never verify that assumption against their own specific task. The assumption is often wrong, and the cost of being wrong compounds with every call at production volume.

The model comparison should cover three dimensions in a single view: accuracy on your evaluation metric, latency per call, and cost per call. Accuracy matters for output quality. Latency matters for the user experience. Cost matters for the economics of running the system across thousands or millions of calls per month. A model that is slightly less accurate on your specific task but meaningfully faster and cheaper may be the right choice for a high-volume low-stakes task, while a slower and more expensive model may be justified for a low-volume high-stakes one.

For the restaurant, comparing two or three candidate models on the full golden dataset including the hard cases produces a clear answer. If the less expensive model captures all four required fields correctly on ninety percent of the hard inputs and the premium model does so on ninety-two percent, the team has a real number to weigh against the cost difference per call. That decision should be made from measurement, not from default assumptions about which model is best.

Building this entire system, from the golden dataset through evaluator calibration to model comparison, is a few days of focused work for a first version. The return is the ability to make every future change to the agent with real confidence that it improved things across the board rather than fixing one visible case while quietly breaking three others. That confidence is what separates AI agent setups that stay reliable over months of real use from the ones that accumulate failures nobody notices until the customer makes enough noise to force an investigation.

None of this requires a research team or a significant budget. A working first version, with a focused golden dataset and a calibrated LLM evaluator, can be built in a few concentrated days. The investment is modest. The return compounds indefinitely: reliable agents that improve with every iteration cycle, model comparisons that routinely surface a cheaper alternative that performs equally well on the actual task, and a benchmark that grows more representative with every production case added to it. Teams that build this infrastructure early move faster and spend less. Teams that skip it spend more and still get surprised.