Evaluation
An agent demo proves that something can work once. Evaluation tells you how often it works, where it breaks, and whether last week's change made it better or worse.
A single model call has one input and one output. You can compare the output to a reference and get a number. An agent plans, calls tools, reads the results, and decides what to do next — often over dozens of steps. Two runs of the same task can take different paths and both be right. Or both look right and one of them quietly booked the wrong flight.
That changes what a test is. Errors compound: a step that succeeds 95 percent of the time, repeated twenty times, gives you a system that finishes cleanly about a third of the time. The agent has side effects, so a wrong action costs more than a wrong sentence. And the environment moves — APIs change, documents get updated, the model behind the agent gets swapped. An evaluation that ran once in March says little about the system in June.
Most teams I talk to know this. What they have instead of an evaluation is a shared channel where people post screenshots of good and bad runs. That's a start, but it doesn't tell you whether you can ship.
The parts are not exotic. The work is in keeping them honest.
Offline evals tell you whether a change is safe to ship. Production sampling tells you whether your offline evals still describe reality. You need both, and the second one feeds the first: every surprising production failure becomes a new test case.
Where an outcome can be checked by code, check it by code. Did the record get created, does the SQL return the expected rows, does the output parse. These checks are cheap, fast, and don't drift.
Much of what agents do can't be checked that way. Was the summary faithful to the source? Was the reply appropriate for an upset customer? Here most teams use a second model as a judge. That works — within limits. A judge model has its own biases, tends to be lenient toward fluent text, and can share blind spots with the model it grades. A judge you haven't compared against human ratings is an opinion, not a measurement.
So the judge needs its own evaluation: a set of cases rated by people, against which you measure how often the judge agrees. When agreement is high, you can let the judge handle volume. When it drops — after a model update, or on a new task type — you need to know that before you trust its scores.
Human judgment enters at three points. First, when you build the reference set: someone has to decide what a good outcome is, and that someone should know the domain. Second, when you calibrate automated judges, as above. Third, in production, for the cases where a wrong action is expensive — an approval step before the agent sends money, deletes data, or writes to a customer.
The mistake I see most often is treating human review as a temporary crutch to be removed as soon as possible. For high-stakes actions it is part of the product. The useful question is which decisions need a person, how you route only those, and how the reviewers' decisions flow back into the test set.
Human ratings are also noisier than people expect. Two experts disagree on a noticeable share of cases. If you don't measure that disagreement, you don't know the ceiling your automated metrics could ever reach.
Expert review doesn't scale to thousands of cases, and for many questions you don't need experts — you need many careful people and a good process. Is this answer helpful? Does this transcript match the audio? Which of these two responses would you prefer? That's what crowdsourcing is for.
This is where I'm not neutral, and you should know it: Crowdee in Berlin is my own company. I'm CTO and Managing Director there, and we run a crowdsourcing platform for studies and human data for AI model training. So I know this topic from the operating side, and that's where my views on it come from.
What I've learned there is that crowd data is only as good as its quality control. That means qualification tasks before someone is allowed to rate, hidden control questions with known answers, several independent ratings per item, and measuring agreement between raters instead of assuming it. Without those, you get a large number of labels and no idea what they're worth.
Crowd ratings and expert ratings answer different questions. Use experts to define what correct means in your domain. Use the crowd to measure it at volume and to catch the cases where real users see things differently than your team does.
Who doesn't need it yet: a team still looking for the use case. If you don't know what the agent is for, there's nothing to measure. Build the thing first, by hand if necessary, and write down what good looks like along the way.
Architecture and evaluation reviews for LLM and agentic systems are part of my advisory work. That means looking at what you've built, how you currently decide whether it works, and what a test setup would have to look like for you to ship with confidence. How I work with companies is described on Work with me.
Advisory is paid and selective. I take it on where there's a real decision to make, and I'll tell you if I think you don't need me. The FAQ covers the questions I get asked most. For the wider picture of what I do, start at the homepage.