What every extra agent costs

In this lesson we count what a second and a third agent cost, and we decide when a design with more agents should replace one agent. In a pipeline, code runs three agents one after another, and each one gets what the one before it wrote. In a manager design, a model calls two workers as tools and decides when each one runs. The previous lesson builds both with fake models. Here both answer a task that one agent can also do, and we run them next to one agent that does the same work alone and count the difference.

The difference is called the orchestration tax: the cost that each extra agent adds. Handoffs and context sent again take more tokens. The extra calls take time, and each boundary between agents can lose information. Agents can also disagree or wait on each other, and a run with many agents is harder to debug. A design with more agents has to do better than one well-equipped agent by more than that tax. The Agent Engineer Course uses the same name for another cost, the time a person needs to review what many agents produce [1]. This lesson counts the cost inside the system.

The fixture is a small shop that answers customer emails about broken items. The one-agent design has three tools, get_order, get_policy and check_stock, and a loop. The pipeline runs a lookup agent that reads the email, looks the order up and writes notes, then a writer that writes the reply from those notes, then a checker that checks the reply against the notes. The manager calls the same lookup and writer agents as tools. Each model is fake: a function that reads its prompt and its input and writes text back. The runs print the same result each time. The orders, the policy and the emails are illustrative, written by the author. The files are in site/examples/building-agents/orchestration-tax/. Copy the orchestration-tax folder and cd into the copy. Each step below is python3 tax.py <step>, and no application programming interface (API) key is needed.

Where the tokens go

The tokens step runs one email, g06, through all three arrangements and counts the tokens that each model call reads and writes. The fixture counts a word or a punctuation mark as one token. That’s a rough stand-in for a real tokenizer, so compare the numbers with each other and not with a price list.

Example · run it

The tokens of one email

Run this, and compare what you see with the output below.

python3 tax.py tokens

Output

email g06, tokens per arrangement
                   calls   new handoff repeat written  total
single agent           2   218       0    162      70    450  1.0x
pipeline               4   237      44    300      65    646  1.4x
manager + workers      6   250     100    718     157   1225  2.7x

Output verified in CI from site/examples/building-agents/orchestration-tax/tokens.py.

Each piece of text that a call reads gets one of three labels. new is text that no call of the run has read yet: a prompt, the email, a tool result. handoff is text that another agent wrote, such as the notes of lookup. repeat is text that a call of the same run read before, or an agent’s own earlier output sent back to it. written counts what the models wrote.

The one-agent design has 162 repeated tokens too. A loop sends the whole conversation back to the model on each call. The second call reads the first one again. The extra agents add to that. Every agent in the pipeline starts with the same 59-token description of the shop, and each agent reads it again. The manager has three calls of its own, and each one reads the manager’s whole conversation so far, the email included. The manager also copies the notes of lookup into its call to the writer, and it writes the writer’s reply again as its own answer. Its later calls re-read the email, its own earlier calls and the worker results, and every agent re-reads the shop text, so 718 of its 1225 tokens are repeats.

The pipeline costs only 1.4 times the one-agent design here, partly because the writer and the checker never read the email. The next section shows what that costs.

The rest of the tax

Latency. For the same email, the one-agent design makes 2 model calls, the pipeline 4 and the manager 6. Each call waits for the one before it, and the customer waits for every call. When a response has to be fast, the Agent Engineer Course advises one agent, because more agents add overhead [1].

Information lost at a boundary. Email g06 asks for a refund and asks one more question: can the customer bring the toaster back to the shop? The trace step shows each call of the pipeline for that email.

Example · run it

The pipeline on email g06

Run this, and compare what you see with the output below.

python3 tax.py trace pipeline g06

Output

pipeline, email g06
call 1 lookup: reads new 136; writes 20
  wrote: get_order {"order_id": "1131"} / get_policy {"topic": "returns"}
call 2 lookup: reads new 50, repeat 156; writes 26
  wrote: order 1131: toaster, delivered 20 days ago / asks for: refund / policy: Returns and refunds are possible within 30 days of delivery.
call 3 writer: reads new 22, handoff 26, repeat 59; writes 18
call 4 checker: reads new 29, handoff 18, repeat 85; writes 1
  wrote: ok
reply: Dear customer, Your refund for the toaster is on its way. Kind regards, the shop
graded: fail, missing "weekdays"

Output verified in CI from site/examples/building-agents/orchestration-tax/trace_pipeline_g06.py.

The prompt of lookup asks for notes with the order, what the customer asks for and what the tools returned. The notes follow it, and the second question isn’t in them. The writer and the checker only get the notes, so no later agent can answer the question. The checker says ok, because it compares the reply with the notes, and the reply answers every point in them. The writer and the checker read 26 tokens of notes and not the whole conversation of lookup, which is why the pipeline was cheap. A handoff that summarizes saves tokens and can drop what a later step needs. Passing the whole conversation to every agent keeps it, and the Agent Engineer Course lists that as a mistake too, because the contexts grow fast [2]. The same course advises checking each agent’s output before the next agent gets it, because a bad result early in a pipeline goes on through every later step [2]. A check like that only finds what its input holds, and this checker never saw the email.

Agents that disagree or wait. Agents that depend on each other can wait for each other and never finish, send a task back and forth, or change the same thing in opposite ways [1]. The fixture has none of these, because code or the manager decides who runs, one at a time. Each new agent that can start work by itself adds a way to fail that a design with one agent doesn’t have.

Harder debugging. The g06 failure shows in the writer’s reply. It started one call earlier in another agent, in the notes of lookup, and the checker passed the reply because it compared it with those notes. With one agent, the email and the reply are in one conversation. With three, you need a log of every message between the agents to follow a request from start to end [1]. When a model decides the order, as the manager does, it is also harder to say why a run took the path it took [2].

What can’t one agent do?

Before you add a second agent, name what the first one can’t do. The Agent Engineer Course puts one agent with good tools ahead of a group of agents that are poorly coordinated. It advises starting with one agent and adding another only when the first is shown to be unable to do the work [2]. The reasons below come up often, and for each one there is a fix inside one agent to try first.

The context doesn’t fit. The task needs more text than the context window holds, or the context is full of old tool results before the agent reaches the question. The agent can load text just in time: it reads a file or searches when it needs a fact, and it doesn’t take everything in at the start. With compaction, the API writes a summary of the early part of a conversation, and the summary takes the place of those turns. A long agent task then uses fewer tokens and still fits in the window [3]. Claude Platform 101 teaches more patterns for managing the context of a long-running agent [4]. When the work is still too large after you try those patterns, a second agent can read a large part of it in its own context and send back a short summary.

The tool set is too large. An agent with many tools picks the wrong one more often [1]. A tool search tool keeps most definitions out of the context and loads only the tools the model searches for [5].

A step needs a different prompt. A skill puts its instructions into the context only when the task matches its description, so one agent can have many sets of instructions and hold only the one it uses [6]. A skill can’t remove text that the agent has already read. A reviewer that must judge a draft without the writer’s notes and reasoning needs a context that never held them. A step that reads untrusted input and must not hold a dangerous tool is the same case, because separate agents can have separate permissions [1].

The shop’s emails are short, the agent has three tools, and one prompt covers the work. The shop has none of these problems, so the pipeline and the manager can’t do anything that the one agent can’t.

Checkpoint · choice

Which reason names something one agent can't do?

A colleague proposes to split the agent into several agents. Which reason names something the one agent can’t do, even after the fixes that work inside one agent?

The agent has 60 tools, and in some runs it picks the wrong one.Long runs fill the context with old tool results before the answer.The reviewer has to grade each draft without the drafter's notes.The team itself splits the work into triage, research and writing.

CheckHintSkip

Measure the design against one agent

A design with more agents has to score better, and by more than it costs. The compare step runs the shop’s golden set of twelve emails through the one-agent design and through the pipeline. Each email has the phrases a correct reply contains and the ones it must not contain, the way a golden set records qualities in A golden set is the agent’s regression suite.

Example · run it

Both designs on the golden set

Run this, and compare what you see with the output below.

python3 tax.py compare

Output

golden set, 12 emails
                          passed  calls  tokens
single agent               9 of 12    24    4353  1.0x
pipeline (three agents)   10 of 12    48    8083  1.9x

Output verified in CI from site/examples/building-agents/orchestration-tax/compare.py.

The pipeline passes one email more. It makes twice the calls and uses 1.9 times the tokens. The totals don’t say which to keep, because they don’t say where the point came from. A difference of one can be the sum of many emails, some that only one design passes and some that only the other passes. The items step lists each email with its result in both designs, and the trace step shows how a run got there. Find what each design did on the emails where they differ. When the gain comes from something the one agent can take, such as a line in a prompt or one more tool, give it that and run the set again. When it comes from one of the reasons in the last section, weigh those emails against the tax.

Checkpoint · scenario

Three points more at twice the cost

On a golden set of 40 items, the design with three agents passes 34 and the single agent passes 31. The design with three agents uses twice the tokens per run. What do you do with these two totals?

Keep the three agents, because three more passes on 40 items is a clear gain.Keep the single agent, because twice the cost is too much for three points.Run both designs on a new set of 40 items and keep the one that scores higher.Find which items differ and why, and give the single agent what it lacked, if it can take it.

DecideHintSkip

Pitfall: A design that was never measured

A team splits its support agent into an agent that sorts the tickets, an agent that looks things up and a writer, because each role now has a short and clear prompt. The new design reads better, and every trace the team looks at seems reasonable, so it ships. Nobody runs the old single agent on the same golden set, so nobody sees the questions that the first agent’s note now drops, or that the bill for each run went up. A design with more agents is a claim to test like any other change. Run the single agent on the same golden set, and keep the extra agents only when they win by more than they cost.

Exercise

Run python3 tax.py items to see which of the twelve emails each design passes. For each email where the two differ, run python3 tax.py trace single <email> and python3 tax.py trace pipeline <email>, and read the prompts at the top of tax.py. Decide which design to keep, and write one line per email that differs, saying where the difference came from. The totals said the pipeline wins by one point, and the lines say whether that point is worth 1.9 times the tokens.

A good result names five emails. The pipeline passes g04, g05 and g08, which all ask for a replacement of an item that is out of stock. The prompt of lookup has the line “Check the stock before you offer a replacement.”, and the single agent’s prompt doesn’t, so the single agent promises a replacement that the shop can’t send. The single agent passes g06 and g09, which each ask for one more thing, a drop-off at the shop and a refund to a gift card. The notes of lookup drop that request, and no later agent sees the email. So the pipeline gains three emails from one prompt line and loses two at its first boundary. Keep the single agent and add the line to its prompt. With the line, the single agent passes 12 of 12 with 4864 tokens, against the pipeline’s 8083.

Which of the three reasons in “What can’t one agent do?” would change your answer for this shop, and what would it take for the shop’s emails to meet it?

Stretch: Add the sentence 'Check the stock before you offer a replacement.' word for word to SINGLE_PROMPT in tax.py (the fake model looks for that exact text) and run python3 tax.py compare again. Compare the new score and tokens with the pipeline's.

Recap

1. The orchestration tax is what each extra agent adds: tokens for handoffs and repeated context, calls to wait for, information lost at each boundary, new ways to fail and harder debugging.
2. A loop repeats its own context on every call, and each extra agent repeats the shared context again. A manager also re-reads its whole conversation on each of its calls.
3. A short handoff saves tokens and can drop what a later step needs.
4. Before you add an agent, name what one agent can’t do, and try the fixes inside one context first: loading just in time, compaction [3], tool search [5] and skills [6].
5. Add an agent only when one agent is shown to fail at the task [2]. Compare the design with the single agent on the same golden set, and read the items that differ before you decide.

You can now

• Justifies the coordination cost of more than one agent

Mark lesson finishedI know this, skip it

Habit

The next time you or a colleague propose a second agent, run the single agent on the same golden set first, and read the items where the two differ.

References

1. Addy Osmani, Ivar Soares Urdalen, Leo Simons. Multi-agent systems: architectures, roles, the orchestration tax. Agent Engineer Course. Course. AEC-07
2. Addy Osmani, Ivar Soares Urdalen, Leo Simons. Orchestrators: code- versus model-driven, patterns, anti-patterns. Agent Engineer Course. Course. AEC-18
3. Anthropic. Compaction overview. Claude Platform documentation. Reference. Claude docs compaction
4. Anthropic. Claude Platform 101. Claude Academy. Course. Academy claude-platform-101
5. Anthropic. Tool search tool. Claude Platform documentation. Reference. Claude docs tool-search-tool
6. Anthropic. Agent Skills. Claude Platform documentation. Reference. Claude docs agent-skills-overview