AI Agent Evaluation Metrics: How to Measure an Agent Before and After Launch

Why agent evaluation is harder than model evaluation

When you evaluate a plain language model, you usually compare one input to one output. Easy to score, easy to repeat. An agent breaks that assumption. A single user request can fan out into a plan, four tool calls, a retry after a failed call, and a final summary. Any of those steps can be the thing that goes wrong, and the final answer can look fine even when the path to it was a mess.

This is why agent evaluation looks at the trajectory, not only the endpoint. Tool-selection accuracy asks whether the agent picked the correct tool for each sub-task. An agent that hits a web search API when it should have queried the internal CRM has a planning bug, even if it eventually stumbles into a passable answer. The reverse also happens: the right tools in the right order, but a hallucinated number slips into the summary.

There is also non-determinism. Run the same prompt twice and you can get two different tool orderings and two different phrasings. So a useful eval set runs each case several times and reports a rate, not a single pass or fail. And because grading open-ended work by hand does not scale, teams lean on LLM-as-judge scoring for the fuzzy qualities like usefulness and reasoning quality, calibrated against a smaller set of human-labeled examples.

The seven core AI agent evaluation metrics

Start with these seven. They are the smallest set that covers correctness, cost, and safety for a real agent. Each one catches a failure the others miss, which is the whole point of measuring more than one thing.

• Task success rate: the share of requests where the agent fully completed what the user asked. This is your headline number, and it needs either a ground-truth expected outcome or a judge prompt that scores against the original ask, not against a polished-sounding reply.
• Tool-call accuracy: did the agent choose the right tool and pass the right arguments? Split it into selection (correct tool) and argument correctness (correct fields and values), because a right tool with wrong arguments still corrupts a record.
• Groundedness or faithfulness: are the claims in the output actually supported by the data the agent retrieved or the tool results it got back? This is your hallucination guardrail, and it is the metric most likely to look fine in a demo and fail on edge cases.
• Latency: how long from request to finished result, measured at the tail (p95 and p99), not just the average, because the slow 5 percent is what users remember and what blocks a scheduled job from finishing on time.
• Cost per task: total model and tool spend to complete one request, so you can compare a cheaper model that retries twice against a pricier model that gets it in one pass. The cheap model is sometimes the expensive one.
• Escalation rate and safety: how often the agent correctly hands off to a human or refuses an out-of-scope or risky request. Too low means it is bulldozing through things it should not touch; too high means it is useless and dumping everything on your team.

A worked example: scoring a support agent on one ticket

A support rep forwards a ticket to the agent: "Customer Maria Alvarez says her March invoice was double-charged, can you confirm and start a refund?" Walk through how each metric scores this single run, because the abstract definitions click once you see them applied.

The agent queries the billing system for Maria's account, finds two identical charges on March 14 for 240 dollars, and drafts a refund. Task success: it confirmed the double charge and prepared the refund, so this is a pass on the goal. Tool-call accuracy: it called the billing lookup with the correct customer ID and date range, then called the refund tool, so both selection and arguments are correct. Groundedness: the summary it posts says "two charges of 240 dollars on March 14," which matches the tool results exactly, so faithfulness is clean. No invented numbers.

Now the interesting part. A 240 dollar refund is a money-moving action, so the agent should not silently execute it. A well-governed agent pauses and posts an approval card to the rep before the refund goes through, which counts as a correct escalation, not a failure. Latency for the lookup and draft was eight seconds, well within budget. Cost was a few cents. So this run scores well across the board, and the one human touch is exactly where you want it: on the irreversible action, with an audit trail of who approved it.

Offline evaluation: build a test set before you launch

Offline evaluation runs your agent against a fixed dataset of inputs with known good outcomes, before any real user sees it. This is the repeatable, version-over-version foundation. Change the prompt, swap the model, add a tool, then re-run the same set and see whether your numbers went up or down. It is the cheapest place to catch a regression, because nobody is harmed by a bad answer in a test harness.

Build the set from real or realistic requests, not toy examples. Include the boring happy paths, but spend most of your effort on the slices that scare you: ambiguous asks, requests that should be refused, inputs designed to trick the agent into ignoring its instructions, and cases where a tool returns an error and the agent has to recover. The value of an offline set is concentrated in its edge cases. A set of 40 hard cases will teach you more than 400 easy ones.

The catch is realism. An offline set is always an approximation of production. It misses new user behaviors, drifts as your data changes, and cannot reproduce load-dependent latency. That is fine. It is not supposed to be reality, it is supposed to be a stable yardstick. Treat it as regression protection and a gate before launch, then let online evaluation cover the long tail you could not anticipate.

Online evaluation: keep watching after launch

Online evaluation scores the agent on live traffic, after launch, because real users surface inputs your test set never had. This is where you catch distribution shift, the slow creep of new question types, the latency spikes under load, and the genuinely novel ways people misuse a tool. You cannot pre-write every edge case, so you sample real runs and grade them continuously.

You will not hand-grade every production run, so two patterns do the heavy lifting. First, sample a percentage of live traffic and run automated judges over it for groundedness, tool-call correctness, and task success. Second, set alerts on the cheap-to-measure signals: latency above threshold, cost per task creeping up, escalation rate spiking (a sign something upstream broke), and a rising rate of judge-flagged ungrounded answers. When an alert fires, you pull the trace, see the exact tool calls and arguments, and feed the bad case back into your offline set so it can never regress again silently.

Online evaluation also picks up the signals only real users produce: thumbs-up and thumbs-down, the rate at which a human rejects an agent's proposed action at an approval step, and how often a conversation gets reopened because the first resolution did not stick. Those human signals are gold for calibrating your automated judges, because they tell you when the judge and the user disagree.

An AI agent evaluation scorecard you can copy

Here is a scorecard that maps each metric to what it tells you, a sensible way to measure it, and a starting target. Treat the targets as a default to argue with, not gospel. A refund agent touching money should hold a far higher groundedness and approval bar than an internal agent that drafts meeting notes. Tune to the blast radius of being wrong.

Where agent failures actually cluster

When teams trace their failed runs, the failures rarely spread evenly. They pile up in a few predictable spots, and knowing the shape of the distribution tells you where to spend your evaluation effort. The chart below shows a directional breakdown of where production agent failures tend to land. Your mix will differ, but the ranking is a useful prior.

Two patterns show up again and again. Tool problems, picking the wrong tool or passing wrong arguments, are usually the single largest bucket, because that is where the agent translates intent into action and small mistakes compound. Groundedness failures (confident, plausible, wrong) are the second danger zone, and they are the scariest because they pass a casual read. The rest, planning loops, unrecovered tool errors, and over-eager actions that should have escalated, fill out the long tail.

The practical takeaway: if you only have time to build evals for two things, build them for tool-call accuracy and groundedness. They cover well over half of real failures and they are the two most likely to look fine right up until they cost you a corrupted record or a wrong number in front of a customer.

Common mistakes when evaluating AI agents

Most evaluation programs fail in the same handful of ways. None of these are exotic. They are the defaults you fall into when you are in a hurry to ship.

• Scoring only the final answer. The summary can read perfectly while the agent called the wrong tool to produce it. Grade the trajectory, not just the last paragraph, or you will ship planning bugs you never saw.
• Running each case once. Agents are non-deterministic, so a single pass tells you almost nothing. Run each case several times and report a rate, otherwise a flaky failure hides behind a lucky run.
• An offline set made of easy questions. If your test set is all happy paths, a passing score means very little. Weight it toward the ambiguous, adversarial, and error-recovery cases that actually break agents.
• No online monitoring after launch. Offline numbers go stale the moment real traffic shifts. Without sampling and alerts on live runs, your first signal that something broke will be an angry customer, not a dashboard.
• An uncalibrated LLM judge. A judge model is convenient but it has its own opinions and blind spots. Check it against a set of human-labeled examples regularly, or you are scaling up a measurement you do not trust.
• Ignoring cost and latency until the bill arrives. A model that retries three times can quietly double your cost per task and blow your latency budget. Track both from day one, not after finance asks.

A decision framework: which metrics to prioritize

You cannot tune everything at once, and not every agent needs the same emphasis. Match your priority metrics to what the agent does and what it costs to be wrong.

Use task success rate and groundedness as your top two when the agent is informational: it answers questions, summarizes data, or drafts content from your systems. Here the worst outcome is a confident wrong answer, so faithfulness is the wall you defend. A read-only data agent lives or dies on these two.

Prioritize tool-call accuracy and escalation or safety metrics when the agent takes actions that change state: updating a CRM record, issuing a refund, closing a ticket, sending a message to a customer. The cost of a wrong action is far higher than a wrong sentence, so the bar for argument correctness and for correctly pausing on risky steps should be near-perfect. This is exactly where human-in-the-loop approvals, least-privilege access, and audit logs do double duty: they cap the damage of a failure and they give you the trace data your evals need.

Lean on latency and cost per task when the agent runs at high volume or on a schedule: a support triage agent handling thousands of tickets, or a reporting agent that has to finish before the 8 a.m. Slack delivery. At that scale, a small per-task cost or a tail-latency spike multiplies into a real number. Onpilot agents that deliver scheduled work to Slack, Teams, or email with approvals and audit logs make these signals easy to watch, because every run leaves a record you can sample and grade.

Turning evaluation into a habit, not a launch checklist

Evaluation is not a gate you pass once and forget. The model under your agent changes, your data changes, and your users change, so an agent that scored 92 percent at launch can drift to 80 percent in a quarter without a single line of your code changing. The teams that keep agents reliable treat evaluation as a standing process, not a pre-launch ritual.

The loop is simple to state and easy to skip. Keep a versioned offline set as your regression gate. Sample live traffic and auto-grade it. When something fails in production, pull the trace, fix the cause, and add that exact case to your offline set so it is covered forever. Over a few months your offline set stops being a guess and becomes a fingerprint of every way your agent has ever broken, which is the most valuable artifact your evaluation program produces.

Start small. You do not need a research-grade harness on day one. A few dozen real cases, the seven core metrics, multiple runs per case, and one alert on groundedness in production will already put you ahead of most teams shipping agents. Add depth where the failures cluster, and let your trace data tell you where to look next.