Measuring tool use

Start with one tool from CMU’s lecture: create_grocery_cart(items). The user says “Make a cart with bananas, milk, and coffee.” The model calls the tool with those three items, the tool returns “Cart C17 · 3 items · $31.42”, and the model tells the user it created a three-item cart for $31.42¹.

That process can go wrong in four places.

1. Selection. The model might reply “Done!” without calling any tool, or it might call search_web instead. The question here is “Right tool?”, and the typical failure is a “Missing or extra call”¹.
2. Arguments. The model might call the right tool but with ["bananas", "milk"]. The question is “Right values?”, and the typical failure is a “Wrong field or value”¹.
3. Trajectory. Consider a bigger task: check stock, then build the cart. The model might build the cart first. Each call might be correct on its own, but the order is wrong. CMU’s question is “Right order?”, and the failure is a “Bad dependency”¹.
4. Task. All calls might be correct, but then the model tells the user the cart costs $13.42. The question is “Goal achieved?”, and the failure CMU names is a “Plausible wrong answer”¹.

CMU calls this table “Evaluate the whole stack” and gives each level its own example metric: precision and recall for selection, an AST (abstract syntax tree) or schema match for arguments, sequence success for the trajectory, end-to-end success for the task¹. An AST match parses the call the model wrote and compares its structure with the call you expected. Look at the column that asks whether you need to execute anything. For selection and arguments the answer is no; for the trajectory it is “Usually”; for the task it is yes¹. You can grade the first two by comparing the call the model wrote with the call you expected, without touching a real system. Berkeley’s function-calling leaderboard does exactly this when it cannot run a call: “Ideally one should use execution evaluation, but when we evaluate the answers, not all the results are easily executable (e.g., Java functions). So we use the AST as a complement to the execution evaluation.”²

Two things follow from this. First, the levels hide each other. If the model never calls the tool, you have no arguments to grade and no trajectory to follow. A selection failure then looks like a task failure unless you record it separately. Second, one end-to-end score cannot tell you which level broke. Chapter 3’s four gates (valid, correct, safe, done) described what a single call can be; these four levels tell you where in a run to look.

Chapter 3 quoted MCP-Atlas’s three largest failure categories to show that deciding to act, not getting the format right, is the hard part. Here is the whole picture, because it tells you where to put your logging.

MCP-Atlas has 1,000 tasks on 36 real MCP servers with 220 tools, and its prompts “avoid naming specific tools or servers, requiring agents to identify and orchestrate 3-6 tool calls across multiple servers”³. It scores each task with “a claims-based rubric that awards partial credit based on the factual claims satisfied in the model’s final answer”³. Then it sorts the failures. Averaged across the models it tested, “No tools called” is 36.0% of failures and “Partial task completion” is 25.8%; incorrect tool parameters are 14.2%, response quality 8.5%, missing tool calls 6.5%, flawed reasoning 3.5%, skipped steps 3.0%, misunderstood task 1.5%, and other 1.0%³. The authors’ own summary: “primary failures arising from inadequate tool usage and task understanding”³.

Now put these categories on the four levels. The mapping is ours, not the paper’s. Failures like ‘no tools called’ and ‘missing tool calls’ are selection failures. ‘Incorrect tool parameters’ is an arguments failure. ‘Skipped steps’ is a trajectory failure. ‘Partial task completion,’ ‘misunderstood task,’ and ‘response quality’ are task-level failures. When we look at it this way, about four out of ten failures involve a tool that should have been called but wasn’t. Only about one in seven failures is due to a wrong value. If your logs only record whether each tool call parsed, you will miss most of what goes wrong.

Hold this split loosely. It comes from one benchmark, with its specific task design and its own judge, averaged across models. Its public leaderboard was later re-scored: in April 2026, Scale upgraded “the scoring judge and adding retry handling for transient tool errors” and replaced the 20-turn limit with “a tool call budget of 100 max tool calls per task”⁴. Use this split as an example of where failures can cluster, not as a definitive law.

CMU adds a second way to sort failures: by who failed. The model can pick the wrong tool, pass bad arguments or ignore the result; the harness can fail to parse, authenticate, validate or match an ID; the provider or tool can time out, hit a rate limit or throw an execution error¹. The slide’s conclusion: “Production evaluation needs benchmark accuracy and operational failure rates.”¹ Benchmarks mostly score the model’s share. Your own server will add harness and tool failures, so log them too. CMU’s slide also charts tool-call error rates by provider¹. OpenRouter’s public rankings page, as fetched in October 2026, lists “Tool usage across models on OpenRouter”⁵, but we could not confirm the per-provider error figures there, so this book does not repeat them.

You will see these names in papers and model announcements. This section maps what each family gives an agent and how it grades agents. It is not a leaderboard.

• BFCL, the Berkeley Function Calling Leaderboard. Its versions build on each other: “BFCL-v1 introducing AST as an evaluation metric, BFCL-v2 introducing enterprise and OSS-contributed functions, BFCL-v3 introducing multi-turn interactions, and BFCL-v4 introducing holistic agentic evaluation.”⁶ V1 grades single-turn calls of four kinds (simple, multiple, parallel, parallel multiple) by AST match or by execution². V2 adds live, user-contributed functions to fight data contamination; an average entry offers 3 functions to choose from and the largest offers 37⁷. V4’s web-search part has 100 multihop questions answered with a search API and a page fetcher⁸. The early versions sit mostly at selection and arguments; the later ones reach the trajectory and the task.
• MCP benchmarks. These connect agents to real MCP servers. MCP-Bench uses 28 live servers with 250 tools and grades “tool-level schema understanding and usage, trajectory-level planning, and task completion”⁹. MCP-Atlas is the one from the previous section³. MCP-Universe spans 6 domains and 11 servers with execution-based evaluators, including “dynamic evaluators that automatically retrieve real-time ground truth for temporally sensitive tasks”; GPT-5, the best of the models its abstract names, scored 43.72%¹⁰. LiveMCPBench has 95 daily tasks over 70 servers and 527 tools, and found “retrieval errors account for nearly half of all failures”¹¹. MCPVerse puts more than 550 tools in front of the model, an action space over 140k tokens¹². MCPMark has 127 tasks that involve “a broad range of create, read, update, and delete (CRUD) operations”, each checked by “a programmatic script for automatic verification”¹³.
• The ToolBench line. ToolLLM collected 16,464 real RESTful APIs in 49 categories from RapidAPI Hub and had ChatGPT generate instructions and solution paths (2023)¹⁴. In 2024, StableToolBench reported “a notable decline in the performance of all methods over time, which raises concerns about the stability of ToolBench as a benchmark”, and replaced live APIs with a virtual API server¹⁵. The lesson: when a benchmark calls live services, the services change underneath it.
• Coding benchmarks. SWE-bench has 2,294 problems from real GitHub issues in 12 Python repositories; in 2023 the best model, Claude 2, solved “a mere 1.96% of the issues”¹⁶. Anthropic notes that LLMs “have progressed from 40% to >80% on this eval” (the Verified subset) “in just one year”¹⁷. Terminal-Bench reports a “Resolution rate of Terminal-Bench 4.0 tasks ± 95% confidence interval”, and every row names both a model and an agent; in October 2026 the top row was GPT-6 Astra (max) with Codex at 58.2% ± 2.8%¹⁸.

When you read any of these numbers, ask four questions. Which level does it grade? Who wrote the tasks, and is the grader a script or a model? What setup produced it? When was it measured? The next two sections explain why the third and fourth questions are so important.

Run the same task twice with the same tools and the same database. Why would the result differ? Because the model samples its words. τ-bench says so directly: “for the same task, the user prompt and database transitions are the same, with just the LM sampling of the user and agent messages generating sufficient stochasticity”¹⁹. Stanford’s lecture asks the plain version of the question: “Do agents produce the same outcome when given the same task twice?”²⁰

So, you run each task k times. We can summarize these k trials in two ways.

• pass@k asks if at least one of the k trials succeeded. This measures whether the agent can do the task.
• pass^k (read “pass hat k”) asks if all k trials succeeded. τ-bench introduced this metric, “defined as the chance that all k i.i.d. task trials are successful, averaged across tasks”¹⁹.

At k = 1 the two are the same number, the plain average success rate: τ-bench reports “pass^1=pass@1=E[r]=E[c/n]” as its main metric¹⁹. As k grows, they split. Here is a worked example of our own. Suppose an agent succeeds on a task 80% of the time, and each try is independent of the others. The chance that at least one of k tries succeeds is 1 − 0.2^k; the chance that all succeed is 0.8^k.

The same agent can appear almost perfect by one measure and unreliable by the other. Which measure you need depends on the job. If a person will review several attempts and choose a good one, pass@k is appropriate. If the agent runs unattended every day, the job requires pass^k.

The published numbers follow the same shape. On MCPMark the best model, gpt-5-medium, “reaches only 52.56% pass@1 and 33.86% pass^4”, while claude-sonnet-4 and o3 “fall below 30% pass@1 and 15% pass^4”¹³. The HAL reliability dashboard evaluated 15 agents on two benchmarks and found “recent capability gains have yielded only small improvements in reliability”; in its words, “The gap between capability (pass@k) and reliability (pass^k) is substantial across all models.”²¹ It also found a “what but not when” pattern: agents “reliably select similar action types across runs but vary in execution order”²¹. In the four-level language, the trajectory is where repeated runs drift.

Repetition is one kind of stress; small changes to the task are another. ReliabilityBench tested Gemini 2.0 Flash and GPT-4o with ReAct agents and Reflexion agents (which add a reflect-and-retry step) over 1,280 episodes. Agents at “96.9% pass@1 at ε=0 drop to 88.1% at ε=0.2”, where ε sets how strongly the task is perturbed, and among injected faults “rate limiting faults cause the largest impact”²². Chapter 6 discusses what a timeout or a retry does to a write.

The Berkeley leaderboard snapshot last updated on 2026-04-12 lists Claude-Opus-4-5-20251101 twice⁶. In the row marked “(FC)”, the model calls tools through its native function-calling API, and its overall accuracy is 77.47, rank 1⁶. In the row marked “(Prompt)”, the tools are described in the prompt instead, and the same model scores 33.47, rank 57⁶. Same model, same snapshot, more than 40 points apart.

The page does not explain this gap⁶. Chapter 3 showed that each model family writes tool calls in its own wire format. One plausible interpretation is that native mode lets the model use the format it was designed for. This is our interpretation, not one stated by the leaderboard. The lesson is clear without inference: a score belongs to a model in a specific setup, not just to the model itself.

Other parts of the setup also change scores.

• The format of the prompt. Berkeley’s format-sensitivity study tested 39 models on 26 prompt variations²³. When models had to return calls in XML instead of Python or JSON, “even large LLMs (watt-tool-70B, claude-3.7-sonnet) show significant performance drops”²³. For the format of the function documentation, “performance is highest with functions in JSON format, lower with XML, and lowest with Python”²³.
• The wording of the instruction. HAL found that “sensitivity to superficial instruction paraphrasing varies substantially across models”²¹.
• The harness. Terminal-Bench names an agent next to every model in its table¹⁸. MCP-Universe found that “enterprise-level agents like Cursor cannot achieve better performance than standard ReAct frameworks”¹⁰.
• The grader and the budget. MCP-Atlas re-scored every model after changing its judge, adding retries for transient tool errors, and switching from a turn limit to a tool-call budget⁴.
• The date. A live leaderboard “will be updated periodically”⁶, and services behind a benchmark can drift¹⁵.

So, when you record a number, record its setup with it: the model and version, calling mode, harness, prompt and tool formats, grader, budget, k, and date. For your own tool server, this list is also a list of knobs you can intentionally adjust.

A grader that checks only the final answer or the database’s final state tells you if the task was done. It does not tell you how. HarnessAudit, a benchmark that audits the harness instead of the answer, states this clearly: “a run can return a correct, benign answer along a trajectory that accessed unauthorized resources, leaked context to the wrong agent, or triggered irreversible side effects.”²⁴

Its findings are direct. On its overall harness-safety score, “Even the best system reaches only 0.32 overall. Strong task completion does not imply safe execution.”²⁴ The project page does not explain the score’s scale²⁴. “More than 50% of agents commit at least one violation per task”²⁴. Across harnesses, “task completion shows a consistent negative association with safety adherence”²⁴. The violations are not obvious errors: “Agents rarely call obviously wrong tools, but routinely apply seemingly reasonable tools to unauthorized resources.”²⁴ Stanford’s first lecture dedicates a slide to this point, titled “Task Completion ≠ Safety Compliance”²⁰.

Notice where this sits relative to the four levels. Every call in Run B is a reasonable tool with well-formed arguments, and the task passes. A selection grader that compares the calls with an expected list would see one extra read, but nothing in that comparison says the read was forbidden. The problem is which resources the calls touched, and you see it only when you check the trajectory against a policy. That is why CMU lists safety as its own quality dimension, next to the four levels rather than inside them: “Policy · permissions · side effects”, with the instruction to “Test adversarial outputs and consequential actions.”¹

For your own experiment, this means two things. Log what each call touched, not just what it returned. Also, grade at least one property of the trajectory, such as never reading another customer’s record, in addition to the outcome. Chapter 6 discusses the gates that should prevent such calls; this section focuses on noticing them during measurement.

CMU’s first assignment includes a small experiment. Students give a chess-playing agent one of two observations: the board alone, or the board plus the list of legal moves. They run each observation with two models²⁶. For each of the four runs, they record “total play_move calls, calls rejected as illegal, invalid-move rate, and whether game_over: true was reached”²⁶. Then comes the crucial line: “You are graded on the experiment and evidence, not on a particular result or winning the game.”²⁶

This is the shift this chapter asks you to make. A score answers “how good is it?” An experiment answers a narrower question: what changed when I changed this one thing, and why?

Start with a few words from Anthropic’s guide to agent evaluations. “A task (a.k.a problem or test case) is a single test with defined inputs and success criteria.”¹⁷ “Each attempt at a task is a trial. Because model outputs vary between runs, we run multiple trials to produce more consistent results.”¹⁷ The outcome is what happened in the world, not what the agent said. A flight-booking agent might announce at the end of the transcript that the flight is booked, “but the outcome is whether a reservation exists in the environment’s SQL database.”¹⁷

OpenAI’s evaluation guide names a habit to avoid: “Vibe-based evals”, which treats “it seems like it’s working” as your evaluation strategy²⁵. It then lays out five steps: define the objective, collect a dataset, define metrics, run and compare, and keep evaluating²⁵. (The page also says OpenAI’s hosted Evals platform is “scheduled to shut down on November 30, 2026”; the method does not depend on it²⁵.)

One more warning before we get to the recipe. Graders can be wrong. Anthropic reports that Opus 4.5 “solved a τ2-bench problem about booking a flight by discovering a loophole in the policy”¹⁷. The run failed the evaluation as written, yet it “actually came up with a better solution for the user”¹⁷. Read the transcripts of failed trials. Sometimes the failure is in your grader.

For your Studio tool server, the recipe looks like this.

1. Ask one question. For example: does adding an example call to the tool description reduce argument errors?
2. Fix a task set. Use the same tasks for every condition. Each task needs an outcome you can check in the system’s state.
3. Change one thing per condition. You will have a baseline and two variants. If you change the description and the model together, you cannot say which one mattered.
4. Run several trials of every task in every condition. Report pass^1 and pass^k side by side.
5. Record every failed trial by level and by source. These levels are selection, arguments, trajectory, or task; the sources are model, harness, or tool.
6. Read the failures, then compare. Did the change move failures from one level to another, or remove them? With ten tasks and a few trials, a difference of one or two tasks can be noise; say so when you report it.

1. Design a three-condition experiment for your Studio tool server. Name the single change you make between conditions. Describe the task set and how many trials each task gets. For every failed trial, state what you record. Which number would make you keep the change?
2. Explain pass@k and pass^k to a classmate without looking them up. Then, calculate pass^3 for an agent that succeeds 90% of the time on each independent try.
3. Find a tool-use benchmark number in a recent model announcement. Write down the level it grades. List every part of the setup you would need to know before comparing it with another number.