How a call happens

We start with a small object you write. Anthropic’s API requires three fields: a name, a description, and an input_schema³. The name “Must match the regex ^[a-zA-Z0-9_-]{1,128}$”, so 1 to 128 letters, digits, underscores, or hyphens³. The description is “A detailed plaintext description of what the tool does, when it should be used, and how it behaves”³. The input schema is “A JSON Schema object defining the expected parameters for the tool”³. OpenAI launched the same idea in June 2023: developers “describe functions to the model via JSON Schema”⁴.

JSON Schema is a standard language. It defines the required shape of a piece of JSON. It “uses keywords to assert constraints on JSON instances”⁵. The validation vocabulary provides the keywords you will use most often: type (one of seven types, such as “object”, “string”, or “integer”), enum (the value must equal one item from a list), and required (these property names must be present)⁶. So the schema in the figure says: give me an object with a property city that is a string, and city is not optional.

A schema checks the shape of data. It cannot determine if a city name is correct. The model chooses between tools by reading their definitions. Claude “determines when to call a tool based on the user’s request and the tool’s description”². OpenAI states that the definitions are “injected into the system message in a syntax the model has been trained on”, so they “count against the model’s context limit and are billed as input tokens”¹. Every word in the definition is text the model reads and pays for on every turn.

Two design habits follow, both from OpenAI’s guide. Make bad calls impossible to write: “Use enums and object structure to prevent invalid states. For example, toggle_light(on: bool, off: bool) allows for invalid calls”¹. And do not ask the model for what your code already knows: “Don’t make the model fill arguments you already know”¹. Anthropic adds an optional input_examples field, and each example “must be valid according to the tool’s input_schema”, or the request fails with a 400 error³.

An API receives a list of messages. Each message has a role, such as system, user, or assistant⁸. The model itself does not see this list directly. A language model performs one core function: it continues a sequence of tokens. “The list of role and content dictionaries that you pass to a chat model get converted to a token sequence, often with control tokens like <|user|> or <|assistant|>“⁸. The program that converts the message list into a token sequence is called the chat template. Models use different templates, sometimes even when they were fine-tuned from the same base model⁸.

It matters which one you use. With the wrong control tokens, Hugging Face warns, “these models would have drastically worse performance”⁸. The model learned its own markers in training.

Tools ride along the same way. CMU’s slide on tool definitions says it plainly: “The chat template renders it as text”⁹. Qwen2.5’s template, for example, tells the model that its function signatures are provided inside XML tools tags, and pastes every tool’s JSON between those tags in the system prompt⁷. It asks the model to answer with a JSON object inside tool_call tags, and it wraps each result that comes back in tool_response tags⁷.

So a tool call is not a special channel. It is more tokens. CMU puts the two possible continuations side by side: the model can continue with text (“The weather is sunny.”) or with a tool call, and “The model requests an action. The harness executes it.”¹⁰

Take the call get_weather with city Pittsburgh and ask four open models to write it. You get four different strings. Qwen2.5 writes a flat JSON object inside tool_call tags⁷. Qwen3.8 nests a function= tag and one parameter= tag per argument inside tool_call, and its template warns that “Function calls MUST follow the specified format”¹¹. Mistral’s tokenizer has dedicated control tokens such as TOOL_CALLS and BEGIN_TOOL_RESULTS¹²; CMU’s example of a Mistral Small 4 call even carries its call ID inside the output¹⁰. DeepSeek V3.2 uses a block it calls DSML, where each parameter has a string attribute “set to ‘true’ for string type parameters and ‘false’ for other types”¹³.

CMU summarizes it in one line: “One schema; model-specific serialization and parsing.”¹⁰

Notice that two versions from the same family, Qwen2.5 and Qwen3.8, disagree^7,11. Someone has to convert the model’s string back into a function name and arguments. If you call a hosted API, the vendor does this and gives you the tool name and its arguments in JSON format. If you serve an open model yourself, that task is yours. Mistral’s model card shows the serving command with --tool-call-parser mistral¹⁴, and DeepSeek’s parsing code says it “will not attempt to correct malformed output that may be generated by the model”¹³.

Suppose the user asks for the weather, today’s calendar and a flight status. CMU works the arithmetic. One call per turn means four model passes of 0.8 s plus three tools of 1.2, 0.9 and 1.1 s, about 6.4 seconds¹⁰. If the model asks for all three in one turn, the harness runs them together: “Two generation phases plus the slowest tool: 2 × 0.8 + max(1.2, 0.9, 1.1) ≈ 2.8 seconds”¹⁰.

Running three calls at once creates a bookkeeping problem. The results can come back in any order. How does the model know which “72°F” belongs to which request? A call ID. In OpenAI’s Chat Completions API, the assistant message holds an array of tool_calls, each with an id, and each result is a message with role tool and a tool_call_id, the “Tool call that this message is responding to”¹⁵. Anthropic’s tool_use block carries an id the same way³. In CMU’s words, “tool_call_id matches results to calls, including parallel calls.”¹⁰

Parallel is not always allowed. CMU’s rule is short: “Parallelize only independent, safely concurrent calls.”¹⁰ Its counterexample is a refund: find the customer ID, use it to fetch orders, then refund the selected order¹⁰. Each step needs the output of the one before, so they must run in order. Independence is about data, and also about side effects. Two calls that each write the same database row do not depend on each other’s output, but running them at once means the final value depends on which finished last.

Stanford’s slide names four stages of a call: Select and Arguments, where the model picks a tool and fills its fields, then Validate and Execute¹⁸. After the model returns its string, the harness owns the rest. OpenHands, an open-source agent platform¹⁹, shows the harness’s part in its code. Its dispatcher parses the argument string using parse_tool_call_arguments, normalizes the tool name, and finds it in a tools_map¹⁶. CMU’s walkthrough of this same code explains that it then validates with action_from_arguments(args), runs executor(action), and returns an ObservationEvent stamped with the call ID¹⁰. The OpenHands documentation describes the middle step as an “Execution Lifecycle - Validate inputs, execute logic, wrap outputs”²⁰.

Why validate at all, if the API already returned JSON? Because nothing guarantees it is the JSON you asked for. OpenAI’s own API reference says the model “does not always generate valid JSON, and may hallucinate parameters not defined by your function schema. Validate the arguments in your code before calling your function.”¹⁵

Validation is not the same as permission. A call to delete a branch can be perfectly valid, but you might never want it to run. In OpenHands, permission is a separate step. A hook can block an action, and the model then receives a UserRejectObservation with rejection_source="hook"¹⁶. The codebase is careful about what it does not know. Its ToolAnnotations defaults to destructiveHint true, and idempotentHint defaults to false²¹. The defaults are the cautious ones: unless a tool’s author says otherwise, its hints say it may destroy data and is not safe to repeat.

One more job sits inside dispatch: the error message. Every failure in the figure goes back to the model as text, so write that text for its reader. Anthropic advises: “prompt-engineer your error responses to clearly communicate specific and actionable improvements, rather than opaque error codes or tracebacks”¹⁷. Compare two replies to a call that omitted the city. KeyError: 'city' tells the model that something broke. “Missing required argument city: give a city name such as Pittsburgh” tells it what to do next. OpenHands follows this pattern when a name is unknown; it responds with “Tool ‘{tool_name}’ not found. Available: {available}”¹⁶.

Errors are also a signal to you. Anthropic again: “lots of tool errors for invalid parameters might suggest tools could use clearer descriptions or better examples”¹⁷. When it launched Claude’s web search tool, Claude kept appending “2025” to its queries, and the fix was a better description¹⁷.

Here is a failure the harness cannot fix: the model writes {"name":"get_weather", "arguments":{"city":"Pittsburgh"} and stops. One closing brace is missing. CMU’s slide: “The harness cannot parse it.”¹⁰ Stanford frames the goal behind every fix for this: “Require a shared contract of how these system components will communicate.”²² A program downstream needs output it can read.

You can ask the model in the prompt to format its output correctly. Or you can make it impossible for the model to produce incorrect output. This second method is constrained decoding. At each step, the model assigns a score, called a logit, to every token in its vocabulary. Before the model samples a token, the decoder checks each candidate token against a grammar. It sets the score of any token that would break the grammar to minus infinity¹⁰. After softmax, the step that turns scores into probabilities, those tokens have a probability of zero. In CMU’s example, the model considers eight candidate tokens. Four of them survive the grammar check. The sampler then picks token 3, which is the most likely survivor¹⁰. The model cannot write output with a missing brace because, at the end, only tokens that correctly close the object remain.

Why is this hard to do fast? JSON nests. To know whether ] is allowed, you must know whether you are inside an array, inside an object inside an array, and so on, to any depth. A finite list of states is not enough; you need a stack. CMU: “The stack remembers arbitrarily deep nesting.”¹⁰ A grammar engine with a stack is a pushdown automaton. And a single token can span several grammar symbols, or end in the middle of one, so the check runs on the token’s bytes, below tokenization¹⁰.

The engines differ in how they avoid checking the entire vocabulary at every step:

• XGrammar divides “the vocabulary into context-independent tokens that can be prechecked and context-dependent tokens that need to be interpreted during runtime”²³. For Llama 3.1 with a JSON grammar, only 1,134 of about 128,000 tokens were context-dependent, and a further trick cut them to 120²³.
• SGLang builds “a compressed finite-state machine”, so that when only one path is possible it can decode several tokens at once²⁴. Guidance does the same in spirit: tokens the grammar already determines are inserted without asking the model²⁵.
• LMQL compiles constraints into token masks²⁶. Guidance notes that “A JSON schema is actually a context free grammar”²⁵.
• vLLM exposes five modes, choice, regex, json, grammar and structural_tag, on top of XGrammar or Guidance²⁷.

Treat the speed numbers as best-case scenarios, because each paper states this. XGrammar reports “up to 100x speedup over existing solutions”²³. SGLang reports “up to 6.4× higher throughput” for its whole runtime, across tasks from JSON decoding to multi-turn chat, not for constrained decoding alone²⁴. LMQL reports “an up to 80% reduction of inference cost”²⁶.

A related idea operates at a higher level. In DSPy, a signature is a “natural-language typed declaration of a function”: you declare the types for inputs and outputs, and the framework manages “structured formatting and parsing logic”²⁸.

Constrained decoding solves a real problem, but it is easy to think it solves more. CMU lists what constraints can enforce: syntax (valid JSON), shape (the expected fields), types (city is a string), and values (units is C or F)¹⁰. The slide then adds a sentence to remember: “Constraints ensure form—not truth, permissions, tool choice, or task success.”¹⁰

Walk the four gates in the figure with one example. A call with city Pgh and units K fails the first gate because K is not in the enum¹⁰. If you change the units to F, it becomes valid. But if the user asked about Pittsburgh and the model wrote another city, the call is valid but wrong. A valid, correct call to delete production data can still be one the harness should refuse. And a run where every call was valid and correct can still end before the job is done.

The benchmarks show this gap clearly. In MCP-Bench, five strong models (o3, gpt-5, gpt-oss-120b, qwen3-235b-a22b-2507, and gpt-4o) were “all surpassing 98% in schema compliance and valid tool naming”²⁹. The authors concluded: “while basic execution has largely converged, planning and reasoning capabilities remain the key differentiators among models”²⁹. MCP-Atlas, a different benchmark of 1,000 tasks on 36 real MCP servers, broke down why runs failed³⁰. Averaged across the models it tested, the largest share of failures was “No tools called” at 36.0%, then “Partial task completion” at 25.8%; “Incorrect tool parameters” was 14.2%³⁰. Its public leaderboard’s top pass rate is 83.6%, after an April 2026 rescoring³¹. These are separate benchmarks with separate methods, so read them side by side, not as one combined number. Together, they say the same thing: getting the format right is mostly solved; deciding to act and finishing the task is not.

This section provides reference material. Vendors often change these switches, so check the current documentation before you rely on any detail here.

Strict mode is the hosted switch for the same goal as constrained decoding: arguments that reliably fit your schema. OpenAI: “Setting strict to true will ensure function calls reliably adhere to the function schema, instead of being best effort.”¹ Two details catch people. A strict request whose schema does not meet the requirements “will be rejected with details about the missing constraints”¹. And the default differs by API: Responses requests try to normalize a schema into strict mode, while “Chat Completions requests remain non-strict by default”¹. Gemini’s equivalent is a tool_choice mode called validated: “Model ensures function schema adherence”³³.

tool_choice determines whether the model can, must, or must not call a tool. Gemini documents four values: auto (the model decides), any (it must call a function), none (it may not), and validated³³. Forcing a tool call has side effects. On Anthropic’s API, setting any or tool makes the API prefill the assistant message, so the model “will not emit a natural language response or explanation before tool_use content blocks”³. Anthropic’s newest models go further. On Claude Fable 5.1 and Mythos 5.1, “tool_choice types any and tool aren’t supported and return a 400 error”³². Opus 5.5 and Sonnet 5.5 return the same 400, and the Opus 5.5 entry advises: “use auto with strict tool use”³².

Forcing can also create a loop. OpenAI’s Agents SDK release of March 2025 listed “Reset tool use behavior when tool_choice is set” among its key changes, with a fix titled “Fix potential infinite tool call loop by resetting tool_choice after …”³⁴. The likely mechanism is simple: if every turn must call a tool, the model can never stop and answer.

So far, every call we have made has used a JSON object. This is one of three ways to format calls. Early agents used plain text. LangChain’s 2023 post describes this: a model “would generate two strings: 1. A tool name 2. An input string for the chosen tool”, which “confined the agent to one tool per turn”³⁵. JSON then added typed arguments. The third option is to let the model write code.

CodeAct, a 2024 paper, proposed “executable Python code to consolidate LLM agents’ actions into a unified action space”³⁶. It compared text, JSON, and code across 17 models. On API-Bank, code was the best format for 8 of the 17, JSON for 5, and text for 4³⁶. On the authors’ own benchmark, M3ToolEval, code had “a higher task success rate (12 out of 17 evaluated LLMs)”³⁶. The best model, gpt-4-1106-preview, gained “a 20.7% absolute improvement compared to the next best action format (text) while requiring 2.1 fewer interaction turns on average”³⁶. CMU’s slide shows a subset of eight models where code wins for all eight¹⁰; the full picture is the 8 of 17 and 12 of 17 mentioned above. All of these are models from 2022 to 2024, several generations before today’s.

Why might code be helpful? Look at the figure. If you need the weather for twenty cities and then the warmest, JSON means twenty calls. Even sent in parallel, all twenty results come back into the model’s context before it can compare them. With code, it is one short loop that also does the comparison¹⁰. Code also provides error messages for free: CodeAct lets the model use “automated feedback (e.g., error messages) implemented in most software to improve task-solving by self-debugging”³⁶. This idea is older than CodeAct. In 2022, Code as Policies had models write robot policy code that processes “perception outputs” and calls “control primitive APIs”³⁷.

The price is control. CMU calls code “a high-power tool”: expressive, but “Validation is less precise than for a narrow typed function”, and it can touch files, network and processes¹⁰. Its prescription: “Use sandboxing, resource limits, permissions, and audit logs.”¹⁰ A JSON call is easy to check against a schema. A program can do anything its sandbox allows.

Most libraries generate the schema from your function. Hugging Face’s agents course shows the trick. A @tool decorator reads the function’s signature and docstring and produces “Tool Name: calculator, Description: Multiply two integers., Arguments: a: int, b: int, Outputs: int”³⁹. All it needs is “that the tool implementation uses type hints, docstrings, and sensible function names”³⁹. LangChain’s StructuredTool.from_function “infers the args_schema from the function’s signature”³⁵. OpenHands auto-generates “LLM-compatible tool descriptions from Pydantic schemas”²⁰. FastAPI generates an OpenAPI schema for your whole web API, using JSON Schema for the data⁴⁰.

Schema generation copies what you wrote. A one-line docstring becomes a one-line description. This description is what the model reads to make its choice. FastMCP provides a clear warning. It can convert any OpenAPI spec into tools for an agent. By default, it converts “every endpoint” into a tool³⁸. Its documentation states: “in practice LLMs achieve significantly better performance with well-designed and curated MCP servers than with auto-converted OpenAPI servers”³⁸. It recommends generating servers this way for “bootstrapping and prototyping, not for mirroring your API to LLM clients”³⁸.

We will skip training in this lecture, so this section is short. This work comes from 2023 and 2024.

Toolformer, in February 2023, taught a model to call tools on its own. The model added possible API calls to text. A call was kept only if it helped by the paper’s test: “An API call is helpful to M if providing it with both the input and the output of this call makes it easier for the model to predict future tokens, compared to not receiving the API call at all, or receiving only its input”⁴². At run time, decoding pauses when it hits a special token. The API runs, and then decoding continues with the result inserted⁴².

Gorilla (May 2023) fine-tuned a LLaMA-based model to write API calls and reported that it “surpasses the performance of GPT-4 on writing API calls”⁴³. It also gave a useful definition: a hallucination is “an API call that is not a sub-tree of any API in the database – invoking an entirely imagined tool”, which is different from calling a real API incorrectly⁴³. ToolLLM (July 2023) scaled the data up, collecting “16,464 real-world RESTful APIs spanning 49 categories from RapidAPI Hub”⁴⁴.

Live APIs created a measurement problem. StableToolBench re-ran ToolBench, which calls real online APIs, and found “a notable decline in the performance of all methods over time”⁴¹. The status of the live APIs kept changing, which affected the results⁴¹. StableToolBench’s fix was a virtual API server, which allowed for reproducible results⁴¹.

1. Without looking, trace the get_weather call. Start from the definition you write and follow it to the “72°F” the model reads. Name each stage. For each stage, describe one way the call can fail there.
2. A teammate says, “We turned on strict mode, so tool calls are reliable now.” Write a two-sentence reply. Explain what strict mode guarantees and what it leaves to you.
3. Pick a tool from your own project. Write the error message it would show for a missing argument. Then, decide if two calls to this tool could safely run in parallel in the same turn.