Many tools

Every tool you add has two costs.

The first cost is context and money. Chapter 3 showed that tool definitions travel as ordinary text in the prompt. OpenAI explains the consequence: “callable function definitions count against the model’s context limit and are billed as input tokens”¹. You pay for every definition on every turn, whether the model uses that tool or not.

The second cost is the choice itself. A 2026 study of shortlist length describes this dilemma in three sentences: “How long should that shortlist be? Show too many tools and the model struggles to choose. Show too few and the correct tool may not appear”².

Several research groups have measured the first part of that sentence. Each group used different models, tasks, and measures. So, read every number with its specific conditions:

• RAG-MCP (2025) retrieved the few most relevant tool descriptions before calling the model, instead of putting all of them in the prompt. It reports that this “more than triples tool selection accuracy (43.13% vs 13.62% baseline)” and cuts prompt tokens by over 50%⁴. The abstract does not name the model.
• LongFuncEval (IBM, 2025) grew the catalog step by step across several long-context models and saw “a performance drop of 7% to 85% as the number of tools increases”⁵.
• ToolMenuBench (2026) compared six ways of filtering the visible menu across seven model backends, three menu sizes and seven evaluation settings. Its own method, causal minimal tool filtering, raised average task success “from 32.1% under all-tools exposure to 85.7%”⁶.
• Less-is-More (2024) moved the question to edge hardware: offering fewer tools cut execution time by up to 70% and power use by up to 40%⁷.

The effect is not the same for every model. MCPVerse (2025) found that “most models suffer performance degradation when confronted with larger tool sets”, but that agentic models such as Claude-4-Sonnet “can effectively leverage expanded exploration spaces to improve accuracy”⁸. LiveMCPBench (2025), with 70 servers and 527 tools, found that retrieval errors “account for nearly half of all failures”⁹. A bigger catalog can help an agentic model such as Claude-4-Sonnet and still hurt most others. Much of the damage happens when the model tries to find the right tool.

So, where is the line? Vendors give rules of thumb. OpenAI says: “Aim for fewer than 20 functions available at the start of a turn at any one time, though this is just a soft suggestion”¹. Anthropic says: “Claude’s ability to pick the right tool degrades once you exceed 30–50 available tools”³. OpenAI’s namespace advice adds a third number: “fewer than 10 functions” per namespace¹⁰. These are three different kinds of statements about three different setups. Read them as vendor guidance for their own models, not as measured laws. ToolMenuBench varied “tool-menu size, distractor type, state-dependent task structure, and risk exposure”⁶; each of those changes the result. There is no universal threshold, only yours.

The cheapest tool is the one you never define. Before you consider search indexes or sandboxes, first remove tools.

Start by looking at arguments. OpenAI’s guide offers a small example worth remembering: “Don’t make the model fill arguments you already know. For example, if you already have an order_id based on a previous menu, don’t include an order_id parameter. Instead, define submit_refund() with no parameters and pass the order_id in your code”¹. The next sentence states the second rule: “Combine functions that are always called in sequence”¹. If your agent always calls get_user, then get_user_orders, then get_user_address, these three actions are effectively one tool, even if you wrote them as three separate ones.

Next, remove entire tools. The Claude Code team writes: “Claude Code currently has ~20 tools, and our team frequently revisits if we need all of them for Claude to be most effective. The bar to add a new tool is high, because this gives the model one more option to think about”¹¹. They also remove tools that newer models no longer need: “As model capabilities increase, the tools that your models once needed might now be constraining them”¹¹. As we saw in chapter 4, Sierra went further. They replaced the GitHub MCP server’s hundreds of tools with the gh command line and a read-only token for specific repositories¹³. CMU’s slides state the principle behind all of this: “Call tools only when their benefit outweighs latency, cost, failure, and risk”¹².

Consider a made-up support agent with 30 tools. Eight are admin tools the support agent never uses; delete them. Three order lookups always run together; merge them into one. The refund tool asks for an order ID the application already has; remove that parameter. You are now down to about 18 tools before you write any infrastructure.

The second step is grouping. When tools belong to different systems, OpenAI suggests using namespaces: “Use namespaces to group related tools by domain, such as crm, billing, or shipping“¹. They recommend keeping each namespace “to fewer than 10 functions for better token efficiency and model performance”¹⁰. Anthropic’s approach uses a naming convention: “prefix by service or resource (for example, github_, slack_) so one search matches the whole group”³. Grouping alone does not shorten the prompt. It helps when the model must first choose between systems¹. It also prepares for the next step, where an entire group can be loaded at once.

What do these moves cost? Design time, and a tool set that fits your task instead of mirroring someone’s API. That is usually a price worth paying.

Sometimes you cannot subtract enough. An engineering agent may need GitHub, Slack, and three monitoring systems. The third move is to stop showing everything up front.

Here is how Anthropic does this. “You provide all your tool definitions to the API, but mark tools with defer_loading: true to make them discoverable on-demand. Deferred tools aren’t loaded into Claude’s context initially”¹⁴. The model gets a search tool instead. When it needs a capability, it searches, and only the matching definitions enter the context. Anthropic’s documentation describes the effect on its five-server setup: “Tool search typically reduces this by over 85 percent, loading only the 3–5 tools Claude needs for a given request”³. Note the two hedges: “a typical multiserver setup” and “typically”³. The launch post’s before-and-after comparison counts total context consumption, not definitions alone: about 77K tokens before any work began without tool search, and about 8.7K with it¹⁴.

Accuracy also moved, according to Anthropic’s internal evaluations: “Opus 4 improved from 49% to 74%, and Opus 4.5 improved from 79.5% to 88.1% with Tool Search Tool enabled”¹⁴. The post names no task set or sample size, so we should treat these as the vendor’s own report¹⁴.

Anthropic also says when not to bother with this method. “Use tool search when any of the following apply: You have 10 or more tools available. Your tool definitions consume more than 10k tokens”³. And it explains the other side: “Standard tool calling, without tool search, is a better fit when you have fewer than 10 tools, every tool is used in every request, or your tool definitions are small (less than 100 tokens total)”³. It also suggests keeping your “3–5 most frequently used tools non-deferred”³, so the common path needs no search.

OpenAI’s tool search works the same way, in the Responses API from gpt-5.4 on¹⁰. One detail changes the arithmetic: “For an individual deferred function, the model still sees the function name and description, so in practice tool search is mostly deferring the parameter schema”¹⁰. That is why OpenAI recommends deferring namespaces or MCP servers rather than single functions: “token savings are usually more material there”¹⁰.

Harnesses do this for MCP servers too. In Claude Code, “Only tool names and server instructions load at session start, so adding more MCP servers has minimal impact on your context window”¹⁵. It sets no fixed per-server tool cap: “the practical limit is your context window budget”¹⁵. Cursor shows MCP tools by name only and lets the agent look them up when the task calls for it¹⁶. In an internal A/B test, “in runs that called an MCP tool, this strategy reduced total agent tokens by 46.9%”, which Cursor calls statistically significant “with high variance based on the number of MCPs installed”¹⁶.

Watch out for one trap in the documentation. OpenAI’s own example of deferred loading for an MCP server sets "defer_loading": true next to "require_approval": "never"¹⁷. Deferring definitions and approving calls are separate decisions. If you copy the example, you turn off approval along with saving tokens.

What does this move cost? A lookup that can miss. The model now has to notice that it needs a tool and then find it. LiveMCPBench’s finding that retrieval errors cause nearly half of its failures⁹ says that step is not free. We will come back to this in the section on big catalogs.

Tools have a third cost, one that is not obvious in the bill. To see it, you need to understand what a prompt cache is.

In a long conversation, most of each request repeats the previous one. It includes the same tool definitions, the same system prompt, and the same earlier messages, with only one new message added at the end. Providers save work by storing the model’s internal state for this repeated beginning. OpenAI calls this a reusable prefix: “the unchanged tokens at the beginning of a prompt”²⁰. Reused tokens are cheaper. OpenAI discounts them “up to 95%” and turns caching on by default for supported models²⁰. For GPT-5.6 and later, writing the cache costs 1.25 times the normal input rate, and reading it costs 0.1 times on most models²⁰.

The cache only works if the prefix is exactly the same. Anthropic states this: “Cache hits require 100% identical prompt segments”¹⁸. The order of elements also matters: “the cache follows the hierarchy: tools → system → messages. Changes at each level invalidate that level and all subsequent levels”¹⁸. Tool definitions come first. Anthropic’s table explains it in one line: “Modifying tool definitions (names, descriptions, parameters) invalidates the entire cache”¹⁸.

Consider what this means. Your agent is 30 turns into a task, with a long, cached history. You decide to remove a tool the agent no longer needs. This edit goes at the very front of the prefix. As a result, every token after it — the system prompt and all 30 turns — is processed again at the full rate. Saving a few hundred tokens in a definition can cost you the entire conversation’s cache.

The two major vendors address this issue the same way: put tool changes at the end of the prompt, not the front.

• Anthropic lets you “declare the full tool set in tools up front, then use tool_addition and tool_removal blocks to offer a tool to the model, or withdraw it, from a specific point in the conversation onward. The tools array itself never changes, so the cached prefix stays intact”¹⁹. This works on a listed set of recent Claude models and “is not available on Claude Sonnet 5”¹⁹.
• OpenAI builds the same idea into tool search: “When new tools are discovered by the model, they are injected at the end of the context window”¹⁰. The page gives the reason: tool search “is designed to preserve the model’s cache”¹⁰.

The idea is older than either product. Prompt Cache, a 2023 paper from Yale, stored the attention states of reusable “prompt modules” and reported time-to-first-token improvements “from 8× for GPU-based inference to 60× for CPU-based inference” on document question answering and recommendation workloads²¹. Tool lists were not among its tests²¹.

Definitions are only half of what tools put into context. Every call also returns a result. Results can be much longer than definitions.

Length hurts. LongFuncEval found “a 7% to 91% degradation in answer retrieval as the tool responses length increases”⁵. Anthropic’s context-engineering post names the general effect context rot: “as the number of tokens in the context window increases, the model’s ability to accurately recall information from that context decreases”²³. Chroma’s 2025 report tested 18 models, including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3, and found that “their performance grows increasingly unreliable as input length grows”²⁴.

Design results the way you design definitions: return only what the next step needs. Anthropic’s tool guide suggests letting the agent choose: “You can enable both by exposing a simple response_format enum parameter in your tool, allowing your agent to control whether tools return ‘concise’ or ‘detailed’ responses”²². In its Slack example, the concise form used about a third of the tokens²². This is one example, not a general ratio. For anything that can grow, the guide recommends “some combination of pagination, range selection, filtering, and/or truncation with sensible default parameter values”, and notes: “For Claude Code, we restrict tool responses to 25,000 tokens by default”²². The same guide does not expect bigger windows to remove the problem: “We expect the effective context length of agents to grow over time, but the need for context-efficient tools to remain”²².

Truncation is the blunt tool on that list. Cursor’s engineers describe the problem: “The common approach coding agents take is to truncate long shell commands or MCP results. This can lead to data loss”¹⁶. Their fix keeps the whole result outside the context: “In Cursor, we instead write the output to a file and give the agent the ability to read it. The agent calls tail to check the end, and then read more if it needs to”¹⁶. Cursor gives no number for this change¹⁶; the 46.9% in the tool-search section belongs to a different change in the same post.

Old results also linger. Anthropic’s context-engineering post calls clearing them low-hanging fruit: “once a tool has been called deep in the message history, why would the agent need to see the raw result again?”²³ Some agents make the cleanup a tool. ReSum (2025) has a web agent periodically call “an external tool to condense interaction histories into compact summaries” and reports “a 4.5% improvement over ReAct in training-free settings”, on search tasks the abstract does not name²⁵.

Chapter 4’s section on code execution showed the mechanism: the model writes one program, the program calls the tools, and intermediate results stay in a sandbox. Here is why that matters more as things grow.

When you use direct calls, every result goes through the model’s context. The context size increases with the number of calls multiplied by the size of each result. With a program, the context grows only with the size of the final answer. Anthropic describes this scaling problem clearly: “as the number of connected tools grows, loading all tool definitions upfront and passing intermediate results through the context window slows down agents and increases costs”²⁶. Cloudflare’s post argues for this approach from the selection perspective: “If you present an LLM with too many tools, or overly complex tools, it may struggle”²⁸.

This rung brings two costs. The first is the sandbox, with the limits and monitoring that chapter 4 described²⁶. The second is how it interacts with the earlier rungs. OpenAI’s documentation states: “Tool search runs as a top-level Responses API tool, not from inside generated JavaScript. Function, custom, and MCP tools with defer_loading: true are not initially available to a program”²⁷. For “Writes or approval-sensitive actions”, OpenAI’s table shows: “Use direct tool calling by default to preserve a clear authorization boundary”²⁷. So code carries the bulk reads well, while writes stay as direct calls the harness can see and approve.

Loading tools on demand shifts the problem. Instead of choosing from all tools in the prompt, a system must find the right few in an index. That search is a problem in itself, and research shows it is harder than it appears.

ToolRet (2025) created a benchmark of 7.6k retrieval tasks using a corpus of 43k tools³⁰. Its finding was: “Surprisingly, even the models with strong performance in conventional IR benchmarks, exhibit poor performance on ToolRet. This low retrieval quality degrades the task pass rate of tool-use LLMs”³⁰. It also highlighted a blind spot in many evaluations: “Most tool-use benchmarks simplify this step by manually pre-annotating a small set of relevant tools for each task, which is far from the real-world scenarios”³⁰.

Fixed shortlists also miss hard cases. On ToolBench’s 3,251 tools (the API benchmark described at the end of this section), the 2026 shortlist study found that a fixed shortlist of 5 tools had slightly higher coverage overall than its adaptive agent, 64.7% against 61.9%, where coverage is how often the correct tool makes the list². But the fixed list “finds nothing on hard queries (correct tool ranked 6th–20th)”, while the adaptive agent found 16.7% of them “by searching deeper”².

Two designs give the agent more control over the search:

• ScaleMCP (2025) gives the agent a retriever it can call again mid-conversation and keeps the tool index in sync with live MCP servers, which act as “the single source of truth”²⁹. It was evaluated on a self-built set of 5,000 financial-metric MCP servers²⁹.
• MCP-Zero (2025) has the model write a structured request for the capability it lacks, then routes it in two stages, to servers and then to tools³¹. The authors built a set of 308 MCP servers and 2,797 tools from the official repository³¹. Their headline token figure comes from a different benchmark: “98% reduction in token consumption on APIBank while maintaining high accuracy”³¹.

Large catalogs also change over time. ToolLLM (2023) collected 16,464 real APIs from RapidAPI and paired its model with a neural API retriever³². When StableToolBench (2024) later re-ran that benchmark, it found “a notable decline in the performance of all methods over time”³³. Live tools change, so a setup that scored well last month needs to be measured again.

Every number in this chapter comes from someone else’s model, tasks, and harness. Your threshold depends on all three. The setup alone can significantly change results.

Take one model on the Berkeley Function Calling Leaderboard snapshot of 2026-04-12. Claude-Opus-4-5-20251101 scores 77.47 overall with native function calling and 33.47 when the same model is prompted to call functions in text³⁴. Same model, same leaderboard, a different way of asking. Format matters too: BFCL’s format-sensitivity study found that “even large LLMs (watt-tool-70B, claude-3.7-sonnet) show significant performance drops when return format is in XML format”³⁵. Chapter 8 returns to how setup shapes results. For this chapter, the lesson is narrow: measure in the setup you will ship.

A threshold experiment requires four elements.

1. Fixed tasks. Use the same set of tasks for every condition, with known correct tool calls.
2. One variable. Change only the tool set. For example, run your 3 core tools, then a set of 8, then a set of 8 that includes a few look-alikes. ToolMenuBench varied “distractor type” alongside menu size⁶; include look-alikes as one condition.
3. Repeats. Run each task multiple times per condition. A single run can be due to luck.
4. Failure levels. Record where each failure occurred. CMU’s slides divide evaluation into four levels: selection (“Right tool?”), arguments (“Right values?”), trajectory (“Right order?”), and task (“Goal achieved?”)¹². Also record tokens per run.

One more correction helps you make fair comparisons between shortlists of different lengths. The shortlist study notes that “Most systems apply a fixed shortlist size to every query, but no standard metric exists to evaluate whether that size was appropriate”². Their method corrects for chance, because a longer list will find the right tool more often just by luck². On BFCL’s 370 tools, their learned policy “nearly matches the coverage of showing 50 tools (90.3% vs 90.8%) while presenting only 7 on average”².

Your threshold is the tool count where selection or task success begins to decline, or where tokens increase faster than success does. Document this threshold with your model, tasks, and the date. It will change when any of these factors change.

1. List the four moves in order without looking back: subtract and merge, group, load on demand, call from code. For each move, state its cost.
2. Explain to a classmate why removing a tool in the middle of a long conversation might cost more than keeping it. What are the two vendor answers to this?
3. Look at your Studio tool server. Which of its tools could you merge or drop? Which result could be shorter? What experiment would tell you how many tools it can carry?