Chapter 6: Evaluating Agentic AI Systems — Benchmarks, Methods, and the Hard Questions

Introduction: Why Evaluation Is the Hardest Part of Building Agents

There is a moment that every team building an agentic AI system eventually reaches. The agent has been trained, prompted, and tuned. It works impressively in demos. Stakeholders are excited. And then someone asks the question that nobody has a clean answer to: How do we know if it’s actually good?

This question is deceptively difficult. With a traditional classifier, you split your data, compute accuracy on a held-out test set, and report a number. With a generative language model, you run human preference evaluations and report win rates. These approaches are imperfect, but they are tractable. Agentic AI systems break both approaches in fundamental ways.

An agent does not produce a single output — it executes a sequence of decisions across time, each of which constrains and shapes what comes next. It uses tools. It reads from external systems. It writes to databases, sends emails, executes code. Evaluating whether the final state of the world is correct tells you almost nothing about why it is correct or whether the agent would be correct on a slightly different task tomorrow. And evaluating every intermediate step is prohibitively expensive with human raters at any meaningful scale.

This chapter is about how researchers and practitioners have approached this problem, what they have learned, and where the honest gaps remain. We will cover the major academic benchmarks that have shaped the field, the methodological critiques that exposed their limitations, emerging frameworks for richer evaluation, and a practical guide for teams building agents in production today. By the end, you should have both a clear map of the intellectual landscape and a set of concrete tools you can deploy in your own work.

1. The Benchmarks That Shaped the Field

1.1 What Makes a Good Benchmark for Agents?

Before diving into specific benchmarks, it is worth establishing what we are actually trying to measure. A benchmark for agentic AI should satisfy several properties that, in practice, turn out to be in tension with one another.

First, it should measure real-world relevant capability. A benchmark that tests whether an agent can play tic-tac-toe perfectly tells us very little about whether it can help a software engineer resolve a GitHub issue. The tasks should reflect the kinds of work we actually want agents to do.

Second, it should be reproducible. If the same agent run on the same task produces wildly different scores depending on who is evaluating it, the benchmark is measuring noise as much as capability. Reproducibility pushes toward programmatic success criteria, but the most interesting real-world tasks are precisely the ones that resist easy formalization.

Third, it should be resistant to overfitting and gaming. A benchmark that is known in advance can be optimized for — not by building a better agent, but by memorizing the right answers. This concern is not hypothetical. As we will see, multiple major benchmarks have been criticized for enabling exactly this kind of shortcut.

Fourth, it should measure what matters. Success on a benchmark is only valuable insofar as it predicts real-world performance. If agents that score highly on a benchmark fail in deployment, the benchmark is misleading. If agents that score poorly turn out to be excellent in practice, the benchmark is leaving signal on the table.

No benchmark fully satisfies all four properties. What follows is a survey of the most influential attempts, with honest assessments of where each succeeds and falls short.

1.2 WebArena: Building a Web for Agents to Navigate

WebArena (ICLR 2024) is one of the most carefully designed attempts at a realistic, reproducible benchmark for web-based tasks. It consists of 812 tasks drawn from 241 templates across four self-hosted web applications — an e-commerce platform, a discussion forum, a software development environment, and a content management system. Because these applications run locally rather than against live services, any research group can reproduce the results.

Its defining feature is programmatic evaluation: deterministic reward functions inspect the application’s state after the agent acts (Did the right item land in the cart? Was the merge request opened against the correct branch?), supplemented for information-retrieval tasks by exact-match, must_include, and GPT-4-judged fuzzy_match checks. This makes the benchmark cheap, fast, reproducible, and endlessly extensible — new tasks just need new reward functions. At publication it exposed a stark gap: humans cleared well above 80% on the same tasks while contemporary agents struggled to reach 20%, failing not on raw reasoning but on navigation, error recovery, and holding coherent state across long multi-step sequences.

1.3 SWE-bench: Real Engineering Problems, Real Code

SWE-bench evaluates agents where correctness is perfectly objective: software engineering. Presented as an oral paper at ICLR 2024, it contains 2,294 tasks drawn from real GitHub issues across 12 popular Python repositories (Django, Flask, Requests, NumPy, and others). Each task pairs a natural-language bug or feature description with unit tests that a correct patch must pass — the agent gets the issue and the codebase, produces a patch, and either the tests pass or they don’t. No human judgment, no phrasing games. Crucially, the tasks demand multi-file coordination rather than the isolated function-writing of HumanEval or MBPP: tracing bugs across files, respecting existing conventions, and reasoning about system-wide implications.

Its performance trajectory shows how fast the field moved. At release in late 2023 the best agent (Claude 2) solved just 1.96% of issues; by late 2024 Claude 3.5 Sonnet reached roughly 49% on SWE-bench Verified, a curated high-quality subset — a 25x improvement in about a year. The lesson: benchmarks have a shelf life. A task that stresses today’s frontier can be trivial for tomorrow’s, so the field must keep building harder, more realistic ones.

1.4 GAIA: Designing Tasks That Humans Find Easy and Machines Find Hard

Most benchmarks are hard for humans and easy for machines. GAIA inverts this: it tests “fundamental real-world reasoning abilities” that people find straightforward but that require web browsing, tool use, and multi-modal reasoning that contemporary systems handle poorly. Tasks need no specialized expertise — anyone with general intelligence and internet access could solve them — yet they require multi-step reasoning chains, integrating information across sources, interpreting images or spreadsheets, and fluid tool use.

The initial gap was stark: 92% human accuracy versus roughly 15% for GPT-4 with plugins, the most capable agent at the time — a 77-point gap. The bottleneck wasn’t raw language understanding but the integration of reasoning with reliable tool use, multi-step planning, and error recovery; agents hallucinated tool calls, misread outputs, lost track of progress, and gave up past a few steps. By 2025–2026, leading agents scored 60–70%+, shrinking the gap to 20–30 points — a real improvement, but also a sign that GAIA is no longer the definitive stress test it once was.

1.5 AgentBench: Eight Environments, One Score

WebArena, SWE-bench, and GAIA are each, in an important sense, single-domain benchmarks — and a model that excels at web navigation may be mediocre at software engineering. AgentBench addresses this by evaluating agents across eight distinct environments: operating system interaction, database querying, knowledge graph traversal, a digital card game, lateral thinking puzzles, household simulation, web shopping, and web browsing — breadth designed to surface capability gaps that only appear when an agent must generalize across qualitatively different tasks.

At its August 2023 publication, GPT-4 scored 4.01 overall (and 42.4% on OS-level tasks) while the best open-source model, codellama-34b, managed just 0.96 and the average open-source model 0.51 — a snapshot of a dramatic proprietary/open-source gap that has since narrowed with Llama 3, Mistral, and their successors. More telling than the numbers: performance was uneven across environments — strength on knowledge-graph tasks didn’t predict strength on household simulation — confirming that the underlying capabilities are genuinely distinct and no single environment captures the whole picture.

1.6 Where the Benchmarks Are Going

Every benchmark above shares a fate: as the frontier catches up, scores saturate, and a saturated benchmark stops discriminating between systems. Worse, published benchmarks leak — their tasks drift into the training corpora of the next generation of models, so a high score may reflect memorization rather than capability (a distinct problem from the overfitting discussed in §2, and one no amount of holdout discipline on your side can fix). Three design responses have emerged, and they point at where agent evaluation is heading.

The first is the live, contamination-resistant benchmark. LiveCodeBench (arXiv 2403.07974) continuously collects new problems from programming contests on LeetCode, AtCoder, and Codeforces, tagging each with a release date. A model with training cutoff D is then scored only on problems published after D — measuring genuine generalization to problems it cannot have seen. The same idea has migrated to software engineering: SWE-bench-Live refreshes its task pool monthly, and SWE-Bench Pro (Scale, September 2025) sources tasks from strong-copyleft (GPL) and private commercial codebases specifically because their licensing makes them unlikely to appear in training data. SWE-Bench Pro is also dramatically harder: at launch, top models such as GPT-5 and Claude Opus 4.1 scored roughly 23% on Pro versus over 70% on SWE-bench Verified — about a third of the performance. (That gap is a launch snapshot; it had already narrowed by 2026 as models improved.)

The second is the tool-use benchmark with a simulated user. τ-bench (tau-bench, Sierra Research, arXiv 2406.12045) drops the assumption that a task arrives as a clean, complete specification. Instead it stages a dynamic conversation between the agent — equipped with domain-specific API tools and a policy document it must follow — and a user simulated by another language model, scoring success by comparing the database’s final state against an annotated goal. This tests things single-turn benchmarks cannot: whether the agent asks clarifying questions, follows domain rules, and holds a coherent conversation. It has since grown into a family (τ²-bench adds a dual-control setup where the user also has tools; τ³ extends to more domains and voice). Crucially, τ-bench was also the first of these to report reliability rather than single-run success — the subject of the next Part.

The third response is not a new benchmark but a new leaderboard discipline — reporting cost alongside accuracy — which we take up under the cost dimension below.

2. Methodological Critiques and Hard Lessons

2.1 The Overfitting Problem: When Benchmarks Become Leaderboards

In July 2024, a paper titled “AI Agents That Matter” — authored by Kapoor et al. and accumulating nearly 300 citations within months of publication — delivered a pointed critique of the agent evaluation ecosystem. Its central claim was uncomfortable: most of the benchmarks the field had been using to track progress were fundamentally broken as tools for measuring real-world capability.

The problem was not that the benchmarks were poorly designed. It was that they lacked proper holdout test sets.

In classical machine learning, a holdout test set is a sacrosanct partition of the data that is never used during training, never inspected during development, and only evaluated once — at the moment of final reporting. The discipline required to maintain this partition is precisely what makes test set performance a meaningful proxy for generalization. When that discipline breaks down, the test set becomes, effectively, another part of the training data.

In the agentic benchmark ecosystem, this discipline had largely not been maintained. Task lists were published in full. Solutions were publicly available. The same benchmark tasks that were used to evaluate progress were also, in many cases, used to guide prompt engineering, architecture choices, and fine-tuning decisions. An agent that achieved high performance on such a benchmark might be doing exactly what a lookup table would do — matching inputs to known outputs — rather than demonstrating any genuine generalization.

Kapoor et al. demonstrated this concern concretely by showing that on HumanEval, a widely used code generation benchmark, a simple “warming” strategy — generating multiple solutions and selecting the most common one — achieved approximately 91% accuracy. The state-of-the-art agent architecture at the time (LATS, using GPT-4) also achieved approximately 91% accuracy. The crucial difference: LATS was roughly fifty times more expensive to run. An architecture that cost fifty times as much achieved zero additional accuracy over a simple baseline.

This is not just an abstract methodological critique. It has practical implications for how teams should interpret published benchmark results, how they should design their own evaluations, and how they should think about the relationship between benchmark performance and production performance. A model that tops the SWE-bench leaderboard may have been extensively optimized for SWE-bench specifically. Whether its performance generalizes to your codebase, your issues, your conventions — that is a separate question that the benchmark score does not answer.

2.2 The Cost Dimension: The Number Everyone Forgets to Report

The fifty-times cost finding from Kapoor et al. leads to a broader point that the field has been slow to absorb: cost is not an optional dimension of agent evaluation; it is a required one.

When researchers report that their new agent architecture achieves a new state-of-the-art on a given benchmark, they are almost never reporting the cost to achieve that result. Yet for anyone deploying an agent in production, cost is often the binding constraint. An agent that solves 90% of customer support tickets but costs $10 per ticket may be strictly worse than one that solves 85% but costs $0.50 per ticket, depending on the ticket value and the organization’s cost structure.

The academic incentive structure pushes against reporting cost. Papers are rewarded for beating benchmarks. Costs are implementation details. If reporting costs would make a result look less impressive — which it often would — there is natural selection pressure against including them. The result is a literature that systematically overstates the practical value of more complex architectures.

The practical implication is that any serious agent evaluation should include at minimum: the success rate, the average cost per task (in tokens or dollars), and a comparison against simpler baselines at the same cost level. If a complex architecture does not outperform a simpler one at the same cost, the complex architecture is not adding value regardless of its raw benchmark score.

This principle — compare at equivalent cost points — is one of the most actionable findings in the agent evaluation literature, and one of the most commonly ignored.

By 2025 this critique had grown into infrastructure. Princeton’s Holistic Agent Leaderboard (HAL) (Kapoor, Stroebl, Narayanan et al., arXiv 2510.11977) is a standardized, cost-aware, third-party leaderboard built on exactly the premise above: a one-dimensional accuracy ranking is uninformative to a developer who has to pay the bill, because “agents can be 100× more expensive while only being 1% better,” and a raw leaderboard cannot tell the two apart. HAL reports dollar cost per task alongside accuracy and plots the Pareto frontier — the set of agents for which no other agent is both cheaper and more accurate. To do this at scale it runs a parallel evaluation harness across hundreds of cloud VMs (cutting evaluation time from weeks to hours) and analyzes three dimensions at once: models, agent scaffolds, and benchmarks. Validated across 21,730 agent rollouts spanning 9 models and 9 benchmarks for about $40,000 in compute, its headline finding is a direct rebuke to “bigger is better”: higher reasoning effort reduced accuracy in the majority of runs, and the most costly models are rarely on the Pareto frontier. It also surfaced that the scaffold often matters as much as the model — the same model can swing dramatically in both accuracy and cost depending on the harness wrapped around it.

2.3 Binary Metrics and the Information They Destroy

A third structural limitation of most agent benchmarks is their reliance on binary success metrics. Either the agent completed the task or it did not. This binary framing is convenient — it makes results easy to report and compare — but it destroys an enormous amount of information about what actually went wrong.

Consider two agents, both of which fail a ten-step task:

• Agent A fails on step 1, immediately taking a wrong action that makes the task impossible to complete.
• Agent B completes steps 1 through 9 correctly but makes a critical error on the final step.

Under a binary success metric, both agents score identically: 0. But they are in very different places. Agent B has demonstrated mastery of most of the task; a small targeted improvement might be enough to push it to success. Agent A has demonstrated a fundamental problem with its initial reasoning; no amount of improvement to later steps will help. Binary metrics treat these agents as identical when they are, in any practically meaningful sense, quite different.

This limitation is the target of the “Agent-as-a-Judge” framework proposed by Zhuge et al. in October 2024. The core idea is to represent the requirements of a complex task as a directed acyclic graph (DAG) — a data structure where nodes represent intermediate milestones and edges represent dependencies between them. Evaluation then proceeds by checking each node in the DAG, producing a granular picture of where the agent succeeded, where it failed, and what the failure’s downstream effects were.

A DAG-structured evaluation can answer questions that binary metrics cannot. Which step did the agent fail at most often? Are failures concentrated in the early phases of tasks (planning, tool selection) or the late phases (execution, error recovery)? Are there specific types of subtasks that the agent consistently fails? This kind of diagnostic information is enormously valuable for understanding where to direct improvement efforts.

The Agent-as-a-Judge framework is, as of 2025, a research proposal rather than an established standard. Implementing it in practice requires significant upfront investment — you need to decompose each task into a DAG of intermediate requirements, and doing this well for complex real-world tasks is non-trivial. But the conceptual point is sound, and teams building serious production agents would do well to instrument their evaluations with step-level metrics even if they fall short of full DAG implementation.

2.4 The Reliability Problem: Why One Run Isn’t Enough

There is a fourth limitation that cuts across all the others, and it is the one most likely to mislead you when you read a leaderboard: a benchmark score is almost always the result of a single run, but agents are not deterministic. Run the same agent on the same task twice and you can get different outcomes — because of sampling temperature, because of environmental randomness, because of subtle floating-point non-associativity in how batched inference is computed on a GPU, and because the evaluation procedure itself has moving parts. Each task has its own success rate — perhaps 90% on one, 50% on another — and a single accuracy number in a fixed environment does not measure that variability at all. It measures one draw from a distribution and reports it as if it were the mean.

This produces two distinct kinds of error. The first is false precision in comparisons. When two agents score 71% and 73% on the same benchmark, the natural reading is that the second is better. But if the run-to-run standard deviation is several points, that gap is noise. Anthropic’s Adding Error Bars to Evals (Evan Miller, arXiv 2411.00640) makes the case that the object of interest is not the observed average but the theoretical average across all possible questions, and that eval scores should therefore be reported with the standard error of the mean and a 95% confidence interval (mean ± 1.96 × SEM). It also points out a subtler trap: benchmark questions are often not independent — they cluster into related groups — and clustered standard errors on popular evals can be more than three times larger than the naive calculation suggests. Ignore the clustering and you will “detect” capability differences that do not exist.

The second error is mistaking capability for reliability. An agent that can solve a task is not the same as an agent that reliably solves it, and averaging hides the difference. The metrics that expose it are pass@k and pass^k, and the distinction between them is worth internalizing:

• pass@k measures the probability that at least one of k attempts succeeds. This is a best-of-k capability measure — it rewards an agent that can find the answer eventually. (A 50% pass@1 means the agent succeeds on half the tasks on its first try.)
• pass^k measures the probability that all k attempts succeed. This is a consistency measure, and it degrades multiplicatively: an agent with a 75% per-trial success rate passes all three of three trials only about 42% of the time (0.75³).

pass^k was introduced by τ-bench precisely to expose this decay, and the result was sobering: state-of-the-art function-calling agents like GPT-4o that scored above 60% on a single trial (pass^1) in the retail domain collapsed to below 25% at pass^8. The agent had the capability to complete the task; what it lacked was the reliability to do so eight times in a row. For a production system that must handle thousands of tasks, pass^k is far closer to what “does this work?” actually means.

The encouraging news, as Miller and others argue, is that the statistical tools to fix this — error bars, confidence intervals, reliability metrics, variance audits — already exist and are borrowed wholesale from a century of experimental science. Nothing needs to be invented. What is required is the discipline to run agents more than once and report the spread.

3. The Evaluation Dimensions That Matter

Having surveyed the major benchmarks and their methodological limitations, we are now in a position to think systematically about what dimensions a complete agent evaluation framework should cover.

The table below represents a synthesis of what the research literature considers important, annotated with an honest assessment of how well current benchmarks cover each dimension.

Of these dimensions, the most underserved remain cost efficiency, tool use accuracy, and safety — though the first and last are less barren than they were even a year ago. Cost efficiency is absent from most published evaluations for the reasons discussed above, but leaderboards like HAL now make it a first-class axis. Tool use accuracy — whether the agent is calling the right tools with the right arguments at the right times, rather than just arriving at correct end states through whatever path — is an important signal that end-state metrics erase.

Safety deserves special mention because it is the dimension where “did the agent complete the task?” is most dangerously incomplete. An agent can complete its assigned task and take a harmful action along the way — leaking data, following a malicious instruction embedded in a web page, calling a destructive tool. Measuring this requires scoring the harm separately from the utility. AgentDojo (Debenedetti et al., NeurIPS 2024) is the clearest example of how to do it: 97 realistic tasks across four environments (workspace, Slack, travel, e-banking) with 629 security test cases in which the agent processes untrusted data that may contain prompt injections. It reports three numbers — Benign Utility (task success with no attack present), Utility Under Attack (task success when injections are present, with no harmful side effect), and Targeted Attack Success Rate (the fraction of cases where the attacker’s goal was achieved, where lower is better). The design insight is that security and utility must be scored together, so a defense cannot “win” simply by breaking the agent into uselessness. This treats security as a quantifiable evaluation outcome sitting right alongside task completion — which is the angle relevant here. The broader attack taxonomy and the defenses against prompt injection are the subject of Chapter 14.

4. Automated Evaluation vs. Human Judgment

One of the central tensions in agent evaluation is between the scalability of automated evaluation methods and the reliability of human judgment. Understanding this tradeoff is essential for designing evaluation systems that are both practical and trustworthy.

4.1 The Case for Programmatic Evaluation

WebArena’s approach — examining the state of the world after the agent has acted, using deterministic code — is the gold standard for evaluation where it is applicable. Its advantages are substantial. Programmatic evaluation is fast: a test suite that would take human raters weeks to evaluate can be run in hours. It is cheap: no need to recruit, train, and pay evaluators. It is reproducible: given the same agent and the same task, it will always produce the same score. And it is not subject to the inconsistencies, biases, and fatigue effects that human raters introduce.

The limitation is applicability. Programmatic evaluation only works for tasks where success can be defined precisely enough to check computationally. “Did the correct database row get created?” is programmatically checkable. “Was the agent’s response to the customer’s complaint empathetic and helpful?” is not. The tasks where we most need reliable evaluation — open-ended generation, judgment-intensive decisions, safety-sensitive interactions — are precisely the tasks where programmatic evaluation fails.

4.2 LLM-as-Judge: Promises and Pitfalls

The natural response to this limitation is to use large language models as automated evaluators — a paradigm commonly called “LLM-as-judge.” The idea is appealing: if the tasks require judgment that only a human-level reasoner can provide, use a human-level reasoner. Modern LLMs are capable of sophisticated assessments of text quality, factual accuracy, tone, and relevance. Why not exploit this capability for evaluation?

WebArena already incorporates a version of this with its fuzzy_match criterion, which uses GPT-4 to assess whether an information-seeking response contains the correct answer even when the exact phrasing varies. Various 2025 evaluation frameworks have extended this approach more broadly.

However, claims of high alignment between LLM judges and human judges — figures like “90%+ agreement” that appear in some research papers — have not held up well to adversarial scrutiny. The actual reliability of LLM-as-judge for agentic evaluation remains, as of this writing, a genuinely open question. LLMs used as judges can exhibit the same biases that human judges exhibit (preference for longer responses, sensitivity to presentation order, sycophantic agreement with the agent being evaluated). They can be inconsistent across runs. And they can confidently endorse wrong answers when the correct assessment requires external knowledge or tool use that the judge model lacks.

This does not mean LLM-as-judge is useless — it is often better than the alternatives for open-ended evaluation at scale. But it means it should be deployed with care: calibrated against human judgments before use, monitored for systematic biases, and treated as a useful approximation rather than ground truth.

4.3 Human Evaluation: Expensive, Slow, Still Necessary

Human evaluation remains the gold standard for tasks that require genuine judgment, nuanced assessment, or safety-sensitive decisions. It is the appropriate choice when:

• The tasks are genuinely open-ended and the space of correct responses is difficult to characterize
• The evaluation requires detecting subtle harms or failures that automated systems are likely to miss
• You are calibrating an automated evaluation system and need ground truth labels to validate against
• The stakes of evaluation error are high (a safety evaluation that incorrectly passes a dangerous agent has severe consequences)

The practical challenge is cost and speed. Human evaluation is expensive, slow, hard to scale, and introduces its own forms of inconsistency. The solution is not to eliminate human evaluation but to use it strategically: focus human raters on the cases where automated evaluation is least reliable, and use the human evaluation data to calibrate and validate automated systems that can then be deployed at scale.

5. A Practical Guide for Teams Building Agents Today

The academic benchmarks and methodological frameworks described above provide essential intellectual context. But for a team building an agent in production — whether on Agentforce, Google Gemini, AWS Bedrock, or a custom architecture — the challenge is translating these principles into a concrete evaluation practice.

5.1 Step Zero: Define Success Before You Build

Before writing a single evaluation, you need to answer three questions:

1. What is the agent trying to accomplish? Be specific. Not “resolve customer support tickets” but “resolve tier-1 billing dispute tickets by verifying the customer’s account status, identifying the disputed charge, determining whether the charge is valid under the company’s refund policy, and either issuing a refund or explaining the denial.”

2. What does a correct outcome look like, and how will you know? Can this be verified programmatically — did the right database record change? — or does it require judgment — was the explanation clear and empathetic? Both kinds of tasks require different evaluation infrastructure.

3. What does a harmful or costly failure look like? An agent that refunds a legitimate charge is a different kind of failure from an agent that incorrectly denies a legitimate refund request. Understanding the asymmetry of failures helps you prioritize your evaluation investments.

Your answers to these questions should be written down, agreed upon by the team, and treated as the specification against which you build your evaluation suite. If you cannot answer them, you are not ready to build the agent yet.

5.2 The Four-Layer Evaluation Stack

Think of agent evaluation as a stack of four layers, each answering a different question about the agent’s behavior. Moving up the stack, evaluations become more expensive and more realistic. Moving down, they become cheaper and faster. A mature evaluation practice uses all four layers together.

Layer 1: Unit Testing — Does Each Tool Work?

Before evaluating the agent end-to-end, test each tool or action in isolation. This is the cheapest and fastest way to catch the majority of bugs and misconfigurations. Does the CRM lookup tool return data in the expected schema? Does the email draft tool handle missing required fields without crashing? Does the file-write action write to the correct location with the correct permissions?

Unit testing of tools is often neglected because it feels like infrastructure work rather than AI work. This is a mistake. Many of the failures that show up in expensive end-to-end evaluations could have been caught at this layer in minutes.

Layer 2: Trajectory Evaluation — Does the Agent Take the Right Steps?

The agent may arrive at a correct final state through an incorrect or inefficient path. Trajectory evaluation examines the sequence of tool calls and reasoning steps the agent produces, assessing whether required steps were taken, whether unnecessary steps were skipped, and whether the ordering was appropriate.

For each representative task in your domain, define the expected trajectory: which tools should be called, in what order, with what approximate inputs. Then run the agent and compare its actual trajectory against the expected one. Flag cases where required tools were skipped, unexpected tools were invoked, or the ordering was wrong.

This layer is where you catch the failure mode that end-state metrics miss: the agent that arrives at the right answer by luck or through an unreliable path that will fail on slightly different inputs.

Layer 3: End-to-End Evaluation — Does the Task Complete Correctly?

This is what most people think of when they think of agent evaluation: given a representative sample of tasks, what fraction does the agent complete correctly? This layer is the closest analog to academic benchmarks like SWE-bench and WebArena.

Use programmatic checks wherever possible. Following WebArena’s lead, examine the state of the world after the agent acts and verify it against a known correct state. For open-ended outputs, use LLM-as-judge calibrated against human ratings for a sample of cases. Track both success rate and cost per successful task.

Always compare against simple baselines. Before declaring that your sophisticated agentic architecture is delivering value, verify that a simpler approach — a prompted model without tools, a retrieval-augmented system without agency, a human-in-the-loop workflow — would not achieve comparable results at lower cost.

Layer 4: Regression Testing — Does It Stay Working Over Time?

Agents deployed in production are subject to silent degradation from multiple sources: model updates by the platform, changes to the schema of external APIs, modifications to system prompts or tool definitions, and shifts in the distribution of incoming tasks. A change that seems unrelated to the agent’s core functionality can unexpectedly break a task category the team had considered stable.

Regression testing requires maintaining a golden set of test cases with known-correct outcomes and running this set on every deployment. The golden set should be small enough to run quickly — twenty to fifty cases for most applications — but representative of the critical paths through your agent’s functionality. Any deployment that regresses a passing test should be blocked until the regression is understood and resolved.

5.3 Open-Source Evaluation Tooling

You do not have to build the four-layer stack from scratch. A cluster of open-source libraries has emerged to cover different layers, and knowing which does what — and, importantly, which is actually open-source — saves a great deal of evaluation and vendor time. Hosted commercial platforms (LangSmith, Braintrust, Weights & Biases Weave) are a capable option when you want tracing and evaluation datasets built from production traffic without running the infrastructure yourself; the tools below are pip-installable libraries you run yourself. One caveat worth stating plainly: “open-source” is sometimes a marketing label. Arize Phoenix, for instance, ships under the Elastic License 2.0 — source-available but not OSI-approved — which carries restrictions the genuinely permissive licenses (MIT, Apache 2.0) do not. If licensing matters for your deployment, verify it against the repository rather than the landing page.

A few practical notes. This space is young and moving fast — the agentic-trajectory metrics in DeepEval and TruLens and the red-team/scan modules in Giskard are recent additions, several still marked beta, so pin versions in your own pipeline. The layer assignments are guidance, not doctrine: DeepEval and Giskard both deliberately span multiple layers, and most tools use their own vocabulary (“multi-turn” for what we’ve called trajectory, “response/retrieval evals” for RAG metrics). And none of these replaces judgment — an LLM-as-judge metric shipped by a library is still an LLM-as-judge, subject to every caveat in §4. Calibrate it against human labels before you trust it.

5.4 The One Metric to Rule Them All

If you can track only one metric for your production agent, track cost per successful task completion.

Not raw success rate. Not median latency. Not token count. Cost per successful task completion.

Here is why. Raw success rate, optimized in isolation, leads teams to add complexity — more tools, longer prompts, more sophisticated orchestration — that raises costs without raising value. A success rate increase from 80% to 82% sounds modest in percentage terms but may be the difference between a profitable and unprofitable product if it comes with a 5x cost increase. Conversely, a simplification that reduces success rate from 80% to 78% may be enormously valuable if it also reduces cost by 10x.

Cost per successful task completion forces you to think about the numerator and denominator together. It naturally rewards simple architectures that work reliably over complex ones that work impressively. It connects agent performance to business value in a way that benchmark scores alone cannot.

This principle directly extends Kapoor et al.’s finding that cost can vary by nearly two orders of magnitude for equivalent accuracy. You will rediscover this in production. The question is whether you will have the metrics in place to see it clearly.

Conclusion: Honest Uncertainty and the Road Ahead

The honest state of the field, as of 2025 and 2026, is that agent evaluation is a solved problem in some narrow senses and deeply unsolved in others.

We know how to evaluate agents on well-specified tasks with programmatic success criteria. SWE-bench and WebArena have shown us this. We know that cost is a required evaluation dimension and that complex architectures must be compared against simple baselines at equivalent cost points — and leaderboards like HAL are finally operationalizing it. Kapoor et al. have shown us this. We know that binary end-state metrics destroy important diagnostic information and that richer intermediate feedback is valuable. The Agent-as-a-Judge framework has shown us this. And we now know that a single benchmark run is a draw from a distribution, not a measurement of one — that reliability (pass^k) and capability (pass@k) are different things, and that evals deserve error bars. τ-bench and the error-bar literature have shown us this.

What we do not know — or do not know well — is how to evaluate safety comprehensively (AgentDojo is a start, not a solution), how to measure generalization to genuinely novel tasks as benchmarks keep leaking into training data, and whether any of the automated evaluation methods we currently use are reliable enough to be trusted without human calibration. These are active research problems without consensus solutions.

For practitioners, the implication is straightforward: apply the methods that are well-understood with rigor and care, maintain appropriate skepticism about benchmark claims (yours and others’), and invest in the evaluation infrastructure that will let you catch failures before they reach production. The field is moving fast. Your evaluation practice needs to move with it.