What is still open

Your agent calls create_ticket. The call times out. The harness or the model tries again. Does this create one ticket or two?

Chapter 1 traced this question back to 1984. When a remote call fails, the procedure “will have been invoked either once or not at all—the user is not told which”⁴. Four years later, Sun’s RPC offered servers a partial fix: a server “may choose to remember this ID after executing a call and not execute calls with the same ID in order to achieve some degree of execute-at-most-once semantics”³. The server can choose to use this, and it gets some protection. Nothing forces it to.

MCP’s answer is a hint. An MCP tool can advertise idempotentHint, one of four hints about its behavior². A hint is the server describing itself. The specification says clients “MUST consider tool annotations to be untrusted unless they come from trusted servers”¹, and the maintainers are blunt: annotations “aren’t enforcement”². The tools page of the MCP specification (version 2026-07-28), as read for this book, describes no mechanism that requires a server to refuse a repeated call¹. As of October 2026, the tool-calling guides from OpenAI, Anthropic, and Google say nothing about what to do when a call may have already run or may run twice^5,6,7,8.

What is unpublished, then, is a shared rule: a protocol field or vendor practice that makes a retried tool call safe by construction. Until one exists, the safety of every retry lives in the code of each tool. For your Studio tool server, this means any tool that writes. If create_ticket accepts a request ID and refuses a second call with the same ID, you have rebuilt Sun RPC’s 1988 trick yourself. If it does not, a retry may create a duplicate, and nothing upstream will stop it.

What would settle the question? A tool protocol or vendor API that defines a request identifier the server must honor, and tests that inject a timeout after a write has landed and count the duplicates. A longer list of hints would not settle it, because you cannot check a hint. Chapter 6’s section on whether a call ran goes deeper into the mechanics.

On Monday, you read and approve the description of a third-party tool called search_docs. On Thursday, the server sends a different description with the same name. Does anyone ask for your approval again?

Anthropic’s guide to custom connectors says: “Server developers may update tool behavior unexpectedly, leading to unintended or malicious behavior.”¹⁰. Invariant Labs gives the pattern a name, the rug pull: “Unauthorized changes to MCP tool descriptions after initial user approval”⁹. Their scanner, MCP-Scan, pins installed tools by “tracking changes via tool hashing”, which “allows users to detect changes to tool descriptions”⁹. OWASP’s list for agentic applications sorts the cases. Manipulating a legitimate tool’s interface, such as “MCP tool descriptors, schemas, metadata”, counts as tool misuse; only a tool that is “malicious or compromised at the source” counts as a supply-chain problem¹¹.

The hints from the previous section do not help here because they travel with the description. “An untrusted server can lie.”². A server that swaps its description can also swap its readOnlyHint in the same message.

Be precise about the gap. The sources in this book describe the attack, name it, and detail one detection tool. They do not say how often descriptions are swapped in practice. They neither report nor rule out a rug pull seen in the wild. The MCP-Scan launch post describes how pinning works, and says the scanner also uses Invariant’s Guardrails API “to identify malicious instructions”, but it gives no detection or false-alarm rate for either⁹. A hash only tells you that text changed. A harmless typo fix and a hidden instruction both produce the same mismatch.

This matters now because every third-party server you connect to can change after you approve it. Your own Studio tool server can change for its users in the same way. The cheapest defense you control is to remember what you approved and to ask for re-approval when it changes.

What would settle this? We need documented cases showing how they were found, or a count across real servers. We also need measured detection and false-alarm rates for pinning on real traffic. Finally, a protocol rule would help, requiring a host to re-ask the user whenever a description changes. That last point would turn a habit into a guarantee. Chapter 6 treats lying descriptions as part of the trust boundary.

You move your agent onto a provider’s hosted loop. Your own code used to decide whether a call ran. Who decides now?

Chapter 1 showed that large providers sell the loop as a product. OpenAI’s Agents API “enables Programmatic Tool Calling by default and manages the agent loop for you”¹³. Claude Managed Agents documents its permission defaults in detail, so we will use it as a worked example. Its defaults differ by toolset kind: “the agent toolset defaults to always_allow, and MCP toolsets default to always_ask“¹². Custom tools “are executed by your application and controlled by you, so they are not governed by permission policies”¹². A third policy, auto, has the server evaluate each call, and “No toolset uses auto by default.”¹². Under auto, the evaluation considers the tool, the input, and the session so far, so “the server can treat two calls to the same tool differently”¹².

So a provider-run loop is neither permissive nor strict in general: one product starts three kinds of tool in three different places.

Now look at what Anthropic’s page does not say. It does not explain how the auto evaluator decides, or how often it gets a call wrong, and it gives no reason for the different defaults of the two server toolsets¹². For auto, the page does state one rule about whose words count. When it evaluates a call, the server “does not read intent from a tool result, a fetched webpage, an MCP server’s response, or a message between session threads”¹². But if you relay untrusted end-user text in your own messages, the server “reads that input as your intent too”¹².

This matters now because when someone else runs the loop, their defaults are part of your system’s behavior until you change them. For each tool, you should be able to state which policy applies. If you choose auto, you are trusting a judge whose error rate you cannot look up.

What would settle this: a published evaluation of the evaluator. This evaluation would include how often it allows a call it should have stopped and stops a call it should have allowed, on a stated set of tasks. It would also include the reasoning behind each default. Chapter 6’s section on gates in code compares these policies with other harnesses.

Last month, you wrote a skill that improved your agent. This month, a new model ships. Should you keep using that skill?

Chapter 4’s section on skill evidence gave the numbers; here is what they leave open. SkillsBench ran 87 tasks under 18 model–harness configurations¹⁴. “Curated Skills raise the average pass rate from 33.9% to 50.5% (+16.6 percentage points; 25.5% normalized gain), with configuration-level gains ranging from +4.1 to +25.7 pp.”¹⁴. CMU’s slide on the same benchmark adds: “Skills still hurt on 13 of 87 tasks.”¹⁵. A June 2026 paper, “Not All Skills Help”, found that “individual skills routinely help on some task types while hurting on others”¹⁶, and reports GPT-5.1 on τ-bench at “62.6%→62.6%” with a skill library curated on a different benchmark¹⁶. Its authors put the limit on “the model’s baseline competence on the target domain”¹⁶. OpenAI, writing about its GPT-6 Astra model, warns that “Guidance that helps Sol or Luna may overconstrain GPT-6 Astra”¹⁷.

The evidence is also split on where the gain comes from, and the SkillsBench authors say so themselves: “observed gains could partly reflect ‘more context’ rather than procedural structure”¹⁴. Against that, their self-generated condition “suggests that model-authored procedural text does not reproduce the curated-Skills gain”¹⁴. So, is a skill useful because of its procedure, or because it adds more text to the context? We should keep both interpretations open; the evidence does not yet choose between them.

What has not been published is the question you actually face: whether one skill continues to help after a model upgrade. None of the studies in this book follows the same skill from one model generation to the next. SkillsBench compares many configurations at one point in time. OpenAI’s warning comes as advice, not as a measurement.

This matters now because skills outlive the model they were written for. “Repository skills also guide other contributors’ agents, which may use different models.”¹⁷. Any skill in your Studio repository will be read by whatever model a teammate happens to run.

To settle this, you would run the same tasks with and without the skill on each new model. You would look task by task, not only at the average. Add a control with the same amount of unrelated text. This would help you tell if the gain comes from the procedure or from extra context. If repeated across a few model upgrades, that would answer the question for your tasks.

Your Studio tool server runs a nightly job. It reads customer records and files a summary. No one is at the keyboard. What credentials does it use?

The sources offer answers borrowed from how people and services already sign in. Sierra splits the work: “Interactive work runs as the user. Scheduled or shared workflows run as service accounts with only the permissions they need.”¹⁹. Its automations that touch customer data must also declare “which customers and tools they’re allowed to access before they ever run”¹⁹. MCP’s enterprise-managed authorization moves access decisions to the company’s identity provider: “Instead of each employee authorizing each MCP server individually, the organization’s IT or security team manages access policies in one place.”²⁰. Revoking an employee there takes effect “immediately across all MCP clients”²⁰. Anthropic’s connectors show the risk of a shortcut: “Claude inherits each person’s permissions from the connected service”, but “Custom connectors that use a shared credential reach whatever that credential can access.”¹⁸.

Notice what all these answers have in common. The agent borrows an identity: a person’s, a service account’s, or a shared credential. Even the question of whose intent counts is settled by the channel words arrive on; under its auto policy, Managed Agents treats relayed end-user text as the developer’s intent¹². None of the sources in this book treats the agent as a principal, an actor with its own identity and credentials, separate from the person or job that started it. Whether it should be one, so that a log can distinguish an action the agent took on an analyst’s behalf from one the analyst took directly, is an open question. Chapter 7’s section on two identities covers the patterns in use today.

This matters now because a nightly job that runs with your personal token keeps your access for as long as the token is valid. A shared credential widens every user’s reach to whatever that credential can access. Both are easy to set up and hard to notice later.

What would settle it: an identity mechanism that issues credentials to an agent as its own principal, scoped to a task and revocable independently. We would also need reports from deployments showing that audit logs and revocation actually work with it.

1. Pick one write tool on your Studio tool server. Without looking back at your notes, write the four parts for “did it run?”: what is known, what is unpublished, why it matters for that tool, and what evidence would settle it.
2. For each of the five questions, name the single piece of evidence that would most change how you build. Then explain why a weaker piece of evidence, such as another warning or another hint, would not settle it.
3. Apply the course’s ruler, “If the model were 10x better, would this still matter?”²¹, to each question. Which questions survive this test?