Evaluating AI Agents in Production: A Practical Eval Framework
Most teams ship an agent, watch it work in a demo, and call it done. Here's the eval framework that catches what the demo doesn't.
Shipping an agent is the easy part. Knowing whether it's actually getting better — or quietly getting worse after every prompt tweak — is the part most teams skip, right up until a regression reaches a customer.
Why "it worked in my test" isn't an eval
Running an agent against five hand-picked examples and eyeballing the output tells you almost nothing about production behavior. Real inputs are messier, longer, and weirder than anything you'd think to write by hand. An eval suite needs real (or realistically synthetic) traffic, not a curated highlight reel.
Three layers of evaluation, not one
Unit-level: does a single tool call get the right arguments for a given input? Fast, cheap, runs on every commit. Trajectory-level: given a multi-step task, did the agent take a reasonable path — not necessarily the only path, but one that terminates correctly without looping or calling the wrong tool. Outcome-level: did the task actually get done, verified against ground truth or a human rubric. Most teams only build the third layer, then can't figure out why a regression happened — trajectory-level evals are what tell you where it broke, not just that it broke.
LLM-as-judge works, with real caveats
Using a second model to grade an agent's output scales past what humans can review, but it inherits the judge model's own blind spots and biases. Calibrate it: run the judge against a set of outputs a human has already scored, and don't trust it until its agreement rate with humans is high enough to act on. Re-check that calibration whenever you swap the judge model.
Build the regression suite before you need it
The first time an agent embarrasses you in production is the worst possible time to start building an eval suite. Capture real failure cases as they happen and add them permanently to a regression set — every prompt or tool change runs against the full history of everything that's ever broken before.
What we've actually run into
The failures that actually reach production are almost never the ones anyone predicted. They're edge cases in tool arguments, a retrieved document that's technically relevant but answers the wrong question, a multi-step task that succeeds on step one and quietly fails on step three. An eval suite's job isn't to prove the agent works — it's to catch the ways it doesn't, before a customer does.