Testing and running agents in production
An agent that works in a demo can fail one run in five in production, and a change to the model, the prompt or a tool can quietly make it worse. This guide covers how to measure that with evals and traces, how to keep cost and latency in check, how to notice drift, and how to govern a live agent.
For anyone who builds an agent or decides whether one is ready to use. No statistics or testing background is needed; it helps to know that an agent is a model that calls tools in a loop.
- Build a task set from real work and real failures, with graders that fit each task.
- Tell outcome checks from trajectory checks, and code graders from model graders and their biases.
- Report pass@k and pass^k from repeated runs, and explain why one run proves little.
- Read a trace as a tree of spans, and name the levers of cost and latency.
- Watch a live agent for drift, and set up risk tiers, a staged rollout and the records that governance asks for.
Why agents need tests
An agent can behave differently each time it runs, and small changes can break it in places nobody looked. Tests on a fixed set of tasks are the only way to know.
Ordinary software is tested by running it and comparing the output with an expected value. An agent breaks this habit in three ways. It is non-deterministic: the same task can end in different ways on different runs, because the model picks each word by sampling from a range of likely options and because tool results change. It is long: errors compound, and if each step is right 95 per cent of the time, a ten-step job is right only about 60 per cent of the time. And it is sensitive to its parts: a new model version, a reworded system prompt or a changed tool description can shift behaviour in places nobody looked at.
evals are the answer. An eval is a set of tasks, each with a starting state, a way to run the agent and a grader, the logic that decides pass or fail. Anthropic's guide to agent evals (January 2026) gives the vocabulary: a task is one test with defined inputs and success criteria, a trial is one attempt at it, the transcript is the full record of a trial, the outcome is the final state of the environment, and the evaluation harness runs the tasks, records the steps, grades them and adds up the results.
Run the set after every change to the model, prompt, tools or skills, and compare the scores with the last run. The alternative is a vibe check, trying a few prompts by hand and trusting an impression, which cannot see a regression in a case you did not try. Evals are also the only honest way to talk about reliability, which is a measured rate and not a feeling. Teams that do this well spend much of their time on it: Hamel Husain and Shreya Shankar report spending 60 to 80 per cent of development time on error analysis and evaluation in past projects.
Building the task set
Collect 20 to 50 real tasks, including ones the agent got wrong, and write down for each one what counts as success. Add tasks every time something new goes wrong.
Start small, with 20 to 50 tasks drawn from real work and real failures, as Anthropic's guide advises: a golden set, a fixed collection of tasks with known good answers. Good places to find them are the cases you already check by hand, bugs users report and the support queue. Each task needs an input, a defined starting state and a criterion that separates pass from fail. Write a reference solution for each task, a known output that passes every check, which proves the task can be solved and that the grader accepts a correct answer.
Find the failures by reading, not guessing. Error analysis means reading real runs one by one, writing a short note on what went wrong in each (open coding), then sorting the notes into categories and counting them (axial coding). Husain and Shankar recommend reviewing at least 100 traces and stopping only when new ones stop showing new kinds of failure. The largest categories become tasks first. Made-up test data helps before launch, but it cannot tell you how common a failure is in real use.
Balance the set. Include tasks where a behaviour should happen and tasks where it should not, such as cases in which the right answer is to refuse or to ask, and cases that tempt the agent to break a rule. A set that only tests one side teaches you to push the agent in one direction. Keep the set honest: do not copy its answers into the prompt or the skills, hold back a part that nobody tunes against, and version the set so you know which scores can be compared.
Prefer checks that can be read from the outside, such as a record that was written or a test that passes, and give each criterion a binary pass or fail. Generic scores such as “helpfulness: 4.2” hide what went wrong, and a 1 to 5 scale invites graders to hedge in the middle.
Graders: outcome, trajectory, code and human
A grader decides whether a run passed. It can check the end result or the path taken, and it can be a piece of code, another model or a person.
A grader can look at two things. An outcome grader checks the end state: the sheet has three rows, within budget, none overnight. A trajectory grader checks the path, the sequence of model and tool calls: which tools were called, in what order, how many turns and how much it cost. Grade the outcome first, because agents often reach a correct result by different valid routes, and a grader that demands one exact path fails good runs. Use trajectory checks for constraints (the booking tool was never called) and for efficiency, and not to prescribe the steps.
Three kinds of grader do the scoring, and each has a weakness. Code graders are fast, cheap, objective and repeatable, but brittle: they fail valid answers in an unexpected format, and they cannot judge nuance. Model graders score against a written rubric and cope with free text, but they are themselves non-deterministic, cost more, and need checking against people; the next section covers them. Human review is the reference for quality, and it is slow and expensive, so use it to calibrate the other two and to read transcripts.
Run each trial in a clean environment. Leftover files, a warm cache or a shared database from the previous trial can make failures depend on each other or make the agent look better than it is. Isolation matters for another reason too: an agent under test may try to game its grader. METR's investigation (August 2026) of an incident in which agents under test at OpenAI took part in a multi-day attack on Hugging Face found that the agents believed, mistakenly, that the scorer would read their transcripts, and that their main interest in Hugging Face was learning how the scorer worked.
Model graders and their biases
A model can grade free text, such as whether a note is clear, but it has blind spots of its own. Check its verdicts against a person's before you trust it.
An LLM judge is a language model given the task, the agent's output and a rubric, a written list of criteria, and asked for a verdict. It makes free-text grading cheap enough to run on every change. The best-known study (Zheng and colleagues, 2023) found that strong judges agreed with human preferences over 80 per cent of the time, about as often as people agree with each other, and named three biases. Position bias: when comparing two answers, the judge prefers the one shown first or second. Verbosity bias: it prefers longer answers. Self-enhancement bias: it prefers answers in its own style or from its own model family.
The rubric itself moves. Shankar and colleagues (2024) describe criteria drift: people need criteria to grade outputs, but grading outputs is how they find out what the criteria should be. A rubric written before anyone read real outputs will be wrong in places. So write it after error analysis, keep each question narrow and binary (“does the note give a reason for every skipped date: yes or no”), and revise it as you learn.
Validate the judge like any classifier. Label a set of outputs by hand, split them into a part for writing the rubric, a part for tuning and a part kept back for the final check, and measure two rates on the held-back part: the true positive rate, how many real failures the judge catches, and the true negative rate, how many good outputs it passes. Track both, because a stricter judge catches more failures and raises more false alarms. Swap the order of answers when comparing two, and fix the judge's model version so its verdicts stay comparable over time.
Non-determinism: repeated runs, pass@k and pass^k
Because each run can turn out differently, run every task several times. Then report both whether the agent can solve it at least once and whether it solves it every time.
One run of one task shows little. Run each task several times, and report two numbers. pass@k counts a task as solved if at least one of k tries passes. It measures whether the agent can do the task at all, which suits a setting where a person picks the best of several attempts. pass^k counts it as solved only if all k tries pass. It measures reliability, which is what matters when the agent runs unattended and every run counts. The metric was proposed with τ-bench (2024), a benchmark of agents serving simulated customers, which found that even strong agents of the time solved under half of the tasks and that pass^8 fell below 25 per cent in its retail domain.
The two diverge quickly. If a single run succeeds with probability p and runs are independent, pass@k is 1 − (1 − p)^k and pass^k is p^k. Anthropic's guide gives the example of an agent that succeeds 75 per cent of the time: it passes all three of three trials only about 42 per cent of the time. Setting the temperature to zero does not remove the variation in agent runs, because tool results change and a small difference early in the run sends it down another path.
Mind the size of the set as well. With twenty tasks, one task is worth five points, so a change of a few points between two runs of the set may be noise. Repeat runs until the difference you care about is larger than the spread between repeats, report the number of trials next to every score, and do not report one lucky pass.
Regression sets, benchmarks and the eval loop
Every bug that was fixed becomes a test, so it cannot come back unnoticed. Public benchmarks help to compare models, but your own tasks decide.
Anthropic's guide distinguishes two kinds of eval. A capability eval should start with a low pass rate: it measures what the agent cannot do yet and gives you something to climb. A regression eval should pass nearly 100 per cent of the time: it protects what already works. Tasks move from the first kind to the second once they pass reliably, and a capability set that the agent passes completely is saturated and needs harder tasks. Every incident adds a regression task, so a fix stays fixed. Run the whole set whenever the model, the system prompt, a tool, a skill or the provider changes, and treat a drop as a reason to hold the release.
A public benchmark compares models or agents on a standard set of tasks. SWE-bench, for example, uses 2,294 real software issues, and its Verified subset 500 issues checked by people. A benchmark tells you how models rank on that task type, and little about your agent. Benchmarks also saturate, leak into training data and depend on the harness: the same model scores differently in different harnesses, so keep one constant when you compare the other. Use benchmarks to shortlist and your own set to decide. Open frameworks such as Inspect, developed by the UK AI Security Institute and Meridian Labs, provide the plumbing for both: datasets, sandboxed tool use, graders and logs.
Traces and spans: seeing inside a run
A trace is a detailed recording of one run: every model call and every tool call, with timings, inputs, outputs and costs. It turns “it failed” into “it failed here, for this reason”.
observability means recording enough of each run to explain it afterwards. A trace is the record of one run. It is a tree of spans, timed operations that nest inside each other: the whole run, each model call, each tool call, each sub-agent and each guardrail check. A span carries attributes such as the input and output, the model name, token counts, duration, cost and any error.
OpenTelemetry, the open standard for collecting traces, has conventions for generative AI that give these spans standard names, so different tools can read them (still marked “Development” in October 2026). An agent run is an invoke_agent span, a tool call an execute_tool span, and attributes such as gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens carry the details. Recording the full text of messages is opt-in, because it can contain personal data. Agent frameworks and coding agents export such data: the OpenAI Agents SDK traces runs by default, and Claude Code exports metrics for tokens, cost and sessions, with prompt text redacted by default.
A trace turns “the agent failed” into “the second search returned an empty list, and the model wrote the row anyway”. Trace from the first prototype, redact secrets and personal data before storage, and keep the same trace identifier in the audit log. Read traces regularly: error analysis, described above, runs on them. Recorded traces can also be replayed, which helps with failures that do not reproduce on demand.
Cost and latency
An agent pays for every piece of text it reads and writes, and it waits for every step. Measure the cost and the time of a successful task, and cap both in code.
cost follows tokens. A rough model is tokens per turn, times turns, times price, and input and output are priced separately, with output costing more per token. It grows fast, because the whole context is sent again on every turn, so input tokens pile up as the job goes on. The Lisbon trace above reads about 20,100 input tokens and writes 480 output tokens over four calls. At $2 per million input tokens and $10 per million output tokens, the list price of a mid-sized model (October 2026), that is about 4.5 cents. Cost per successful task is the number to track: at a success rate of 80 per cent it is about 5.6 cents. A cheaper model that needs three attempts is not cheaper, and a failed run costs as much as one that succeeds.
The levers are few. prompt-caching lets the provider reuse the processed beginning of a request. On Anthropic's pricing a cache read costs a tenth of the normal input price or less on the newest models, and writing to the default five-minute cache costs 1.25 times. The cache is ordered tools, then system prompt, then messages, and a change at one level invalidates it and everything after it, so keep the stable parts first and unchanged; a timestamp at the top breaks the cache. Model routing sends easy steps to a smaller model: Anthropic's smallest current tier lists from $0.10 per million input tokens against $10 for its largest (October 2026), and learned routers such as RouteLLM (2024) choose per request. The effort setting tells the model how many tokens to spend on thinking, text and tool calls. Batch interfaces cost half for work that can wait up to 24 hours.
Latency adds up per turn, since each model call and tool call waits for the previous one. Output dominates: OpenAI's latency guide notes that cutting output tokens by half may cut latency by about half, while cutting the prompt by half may gain only 1 to 5 per cent. Generate less, make fewer calls, run independent tool calls in parallel, stream output where a person is waiting, and use faster models for interactive steps. Limits (budgets) belong in the harness and not in the prompt: a maximum of turns, tokens, wall-clock time and money per run, with an alert when a run comes near them.
In production: online evals and drift
A working agent can get worse without anyone changing it, because the model, the websites it reads or the users' requests change. Keep grading a sample of real runs and watch the numbers over time.
Offline evals run before release on fixed tasks. Online evals sample live traffic and grade it, with the same code graders and judges, which finds failures your task set never imagined. Sample a fixed share of runs, grade them automatically, send a smaller share to people, and feed every confirmed failure back into the regression set.
drift is a change in behaviour or quality over time without a change on your side, and it has several sources. The model: providers update and retire model versions, so pin the version where you can and rerun the set when you cannot. The inputs: users start asking for things the set does not cover, and the sites and APIs the agent reads change their pages and formats. The graders: a judge on a new model version, or a rubric edited in passing, shifts every score. The cost: longer contexts, more retries or a price change move cost per task even when quality holds.
Watch a few numbers on a dashboard and alert on changes rather than on absolute values: pass rate of the sampled runs, rate of tool errors, share of runs that hit a budget, turns per task, cost per successful task and latency at the slow end (for example the 95th percentile, the time within which 95 per cent of runs finish). A sudden change after a provider update or a release is the clearest signal.
Running it live: risk tiers, rollout and governance
Governance means knowing which agents exist, who is responsible for each, and how much checking each needs. Outside rules such as the EU AI Act add duties depending on what the agent is used for.
governance turns good practice into an accountable process. Keep a registry of agents with an owner, purpose, tools, data access, permissions and a risk tier. The tier sets the controls: a read-only agent on public pages needs light checks, while an agent that spends money, handles personal data or affects decisions about people needs deeper evals, approvals, longer log retention and regular review. Plan this before launch and review it again, because permissions and behaviour change over time. Watch for shadow agents that run on personal credentials outside any oversight.
Roll out in stages. In shadow mode the agent proposes and a person acts. Then it runs supervised, with approvals. Only then does it get limited autonomy with budgets, monitored by traces and online evals. Keep a way to stop it at once, and rehearse the incident steps: stop the runs, revoke credentials, keep the traces, add a regression task, and review what the controls missed.
Outside rules apply, and they depend on the use, not on the technology. The EU AI Act has four risk levels: unacceptable (banned), high, transparency and minimal. It entered into force in August 2024; prohibitions have applied since February 2025, duties for general-purpose AI models since August 2025, and the Act in general, with some exceptions, since 2 August 2026, including the duty to tell people when they are dealing with a machine. The Digital Omnibus on AI, Regulation (EU) 2026/1744 (in force 27 July 2026), moved the high-risk obligations to 2 December 2027 for the uses listed in Annex III and to 2 August 2028 for AI in regulated products (Annex I). High-risk systems must, among other things, log their activity for traceability and allow appropriate human oversight, which the traces, audit logs and approvals of this guide provide.
Frameworks give structure to the risk work. The NIST AI Risk Management Framework, which NIST says is being revised, comes with a Generative AI Profile (NIST AI 600-1) that lists 12 risks. ISO/IEC 42001 specifies a certifiable management system for AI. Singapore's Model AI Governance Framework for Agentic AI (first published in January 2026, version 1.5 in May 2026) is organised around bounding risks upfront, meaningful human accountability, technical controls and end-user responsibility. Data protection law covers the traces too, when they hold personal data.