How to evaluate an AI agent in production

Evaluating an AI agent in production means scoring every run against versioned test cases on four axes: task success, tool-call correctness, cost per run, and failure mode. Model benchmarks do not transfer to your workflow. Gartner attributes a large share of the 40%-plus of agentic projects it expects to be cancelled by 2027 to inadequate risk controls.

Key takeaways

• Score four axes per run: task success, tool-call correctness, cost per successful task, failure mode.
• Build the golden-case set from real tickets, not synthetic prompts.
• Run the harness before and after every change to separate regression from variance.
• Gartner ties much of its 40%-plus agentic cancellation forecast to inadequate risk controls.

What should you actually measure on an agent?

Four things, because those are the four ways an agent lets you down once it is doing real work. A leaderboard score on a public benchmark tells you none of them.

• Task success: did the run achieve the goal of your workflow, judged by your definition of done? This is the number the sponsor cares about and the one benchmarks cannot supply, because your task is not on any benchmark.
• Tool-call correctness: did the agent call the right tool, with the right arguments, in the right order? Agents rarely fail at language. They fail at integration: calling the wrong endpoint, passing a malformed argument, retrying a call that already succeeded.
• Cost per successful task: what does one good outcome cost, including the retries and dead-ends behind it? Cost per token flatters you; cost per successful task is the honest denominator.
• Failure mode: when a run failed, how did it fail? You cannot install a control for a failure you have not named, so an unclassified failure is an uncontrolled one.

These four are the definition of an evaluation harness, and Gartner ties much of its forecast that over 40% of agentic AI projects will be cancelled by the end of 2027 to inadequate risk controls: the organisational form of not measuring these four (Gartner, 25 June 2025).

Building a golden-case set from real tickets

The test set decides the value of everything downstream, and the single most common mistake is to build it from synthetic prompts that read well and never break. Build it from real tickets instead.

Pull a sample of genuine work items the agent will face. Include the clean ones, and then deliberately over-sample the ugly ones: the malformed inputs, the ambiguous requests, the exceptions that experienced staff handle without thinking. Attach the known-good output where one exists. Put the whole set under version control, because a golden set that drifts silently gives you scores you cannot compare across time.

Coverage beats count. A hundred cases that include your real exception surface are worth more than a thousand representative-but-clean ones, because production does not break on the clean cases. It breaks on the exceptions, and a test set with no exceptions is a test set that certifies the easy 80% and hides the 20% that will page you.

How to catch regressions between model versions

Run the harness before and after every change, and treat the upstream provider swapping a model under you as a change even though you did not make it. The comparison is the point: a single score in isolation tells you nothing, because you cannot tell a bad number from a normal one without a baseline to compare against.

| Situation | Without a harness | With a versioned harness |
|---|---|---|
| Provider updates the model | Behaviour shifts silently | Score delta flags the shift the same day |
| You edit the prompt | Hope it improved | Measured delta, per axis |
| A run looks wrong | Is it a bug or variance? | Re-run the case set; variance is bounded and known |

The reason this matters more than intuition is that human intuition about AI performance is unreliable in a measurable, consistent direction. METR's randomised controlled trial found experienced developers were 19% slower using early-2025 AI tools on real tasks in mature repositories, while estimating afterwards that the tools had sped them up by about 20%: a gap of roughly 39 percentage points between measured and believed (METR, July 2025). METR has since stated the result is historical and does not necessarily reflect current tools or workflows (METR, February 2026). The tools improved; the lesson did not. People cannot feel a regression. A harness can. We go deeper on that gap in the METR result.

Cost per successful task, not cost per token

Cost per token is the number the invoice shows and the wrong number to manage. An agent that answers cheaply but fails half the time, forcing a human to redo the work, is expensive. An agent that costs more per run but succeeds first time is cheap.

Divide total cost (including retries, failed runs and the fallback human effort) by the count of successful outcomes. That denominator is brutal and correct. It is also the number that connects the harness to the business case, because it is the cost the workflow actually bears, and it is what makes an agent's economics legible to a CFO rather than a mystery on a cloud bill.

Human-in-the-loop checkpoints that survive scale

A checkpoint that asks a human to review every output does not survive scale; it just moves the bottleneck. The design rule is to place a human-in-the-loop checkpoint only where an error is both expensive and hard to reverse, and to let the harness carry everything else.

Use the failure-mode axis to place the checkpoints. If a named failure mode is cheap and self-correcting, automate through it. If a failure mode is expensive and silent, that is where the human belongs, not everywhere, there. Checkpoints spread evenly across a workflow are a tax; checkpoints concentrated at the expensive-and-irreversible steps are a control.

What a governance retainer maintains

A harness is not a launch artefact you build once and file. It is a living asset that decays the moment the world moves: a new model, a new tool, a shifted data source, a new class of ticket. Someone has to keep the golden set current, re-baseline after upstream changes, and watch the cost-per-success line. That maintenance is exactly what a governance and CoE retainer exists to carry, and it is the difference between an agent that is trustworthy at launch and one that stays trustworthy.

What this means for a GCC transformation owner

You are being asked to let a probabilistic system touch real work. The only responsible way to say yes is to be able to say, at any moment, exactly how well it is doing and how you know. That sentence is what an evaluation harness buys you, and it is what "inadequate risk controls" (Gartner's third named cancellation cause) means in practice: an agent nobody can currently characterise.

Insist on the harness before the agent goes anywhere near production, insist that it scores the four axes, and insist that it runs on every change. It is the least glamorous part of the build and the part that decides whether you are managing a system or hoping about one.

Sources

1. Gartner, Over 40% of Agentic AI Projects Will Be Canceled by End of 2027, 25 June 2025. https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
2. METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, 10 July 2025. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
3. METR, We are Changing our Developer Productivity Experiment Design, 24 February 2026. https://metr.org/blog/2026-02-24-uplift-update/

Related reading: what an evaluation harness is and why you need one · the METR result on developer productivity · the governance and CoE retainer