08/06/2026
Most teams ship AI agents the same way: try it a few times, it looks good, push to production, hope.
Then a customer asks something you never tested, the agent gives a confident wrong answer, and you find out from an escalation.
The fix isn't a better prompt. It's evals.
An eval is three things: an input, the agent's output, and a scorer that says how good that output was. Run it across a set of real examples and, for the first time, "good" has a number instead of a feeling.
Using a support agent as the example, here's what eval engineering actually looks like:
Score with the right tool. Code for the checkable facts (did the reply include the booking link?), a model to judge tone and accuracy at scale, and a human for the tricky edge cases.
Score the answer and the path. A reply that promises a refund the agent wasn't allowed to give looks perfect on the surface. You catch it by scoring the steps, not just the output.
Make it a loop, not a one-time test. Collect real tickets, score, find the biggest failure bucket, fix it, and keep the set as a regression net so the next change can't quietly break what worked.
And avoid the traps: five happy-path examples, synthetic tickets that don't look real, or a model grading a model with no human anywhere. A green dashboard that checks the wrong thing is worse than no dashboard.
Stop shipping on vibes. Ship on evals.
How are you measuring your agents today?