Writing the plan before taking the steps
In this lesson the fake model writes a numbered plan for a goal before any tool runs. We print the plan and tick off each step as it runs. Then step three returns a fact the plan didn’t expect, and we compare a run that follows the old plan with a run where the model revisits the plan after each step. At the end we look at what the written plan gives a person who reads it before the first step, and at what it costs.
The tools belong to the small shop from
Reasoning before each action, with three new
ones. check_stock returns how many of an item the shop has,
send_replacement books a replacement for an order, and
email_customer sends the customer a message. The fake model and the
code that runs the plan are written by hand. The runs show how the
pattern behaves, and they don’t show how well a real model plans. The complete
program is site/examples/building-agents/planning/agent.py in the
repository. Copy the planning folder, and each step below is
python3 agent.py <step>.
A plan before the first step
Section titled “A plan before the first step”The pattern is called planning: the agent writes out the steps to a goal before it takes them, as a list it then works through and updates. The lesson on planning in the Agent Engineer Course compares it to a packing list that you write before a trip and tick off as you pack [1].
The goal is “Replace the broken kettle from order 1042.” The fake model’s first reply is the whole plan. Each step has a line of text that a person can read and the tool call that does the step.
return [ make_step(f"Look up order {order_id}", "get_order", order_id=order_id), make_step("Read the returns policy", "get_policy", topic="returns"), make_step(f"Check that a {item} is in stock", "check_stock", item=item), make_step(f"Send a replacement {item}", "send_replacement", order_id=order_id), make_step( f"Email the customer that a new {item} has shipped", "email_customer", order_id=order_id, message=f"Your new {item} has shipped.", ),]show_plan prints the list with a box in front of each step, and an x
in the box of each step that has run.
The plan for the kettle
Section titled “The plan for the kettle”Run this, and compare what you see with the output below.
python3 agent.py plangoal: Replace the broken kettle from order 1042. plan v1: 1. [ ] Look up order 1042 2. [ ] Read the returns policy 3. [ ] Check that a kettle is in stock 4. [ ] Send a replacement kettle 5. [ ] Email the customer that a new kettle has shipped
Output verified in CI from site/examples/building-agents/planning/plan.py.
No tool has run yet. The plan is text in the fake model’s reply, and the shop is as it was.
Working through the list
Section titled “Working through the list”The plan runs one step at a time. For each step the code calls the tool
the step names, keeps the result and prints a step line. After a step
has run, its box gets an x. The progress step runs the first two
steps and prints the plan at that point.
The plan after two steps
Section titled “The plan after two steps”Run this, and compare what you see with the output below.
python3 agent.py progressstep 1: get_order(order_id='1042') -> {'ok': True, 'item': 'kettle', 'days_since_delivery': 12}
step 2: get_policy(topic='returns') -> {'ok': True, 'text': 'Returns and refunds within 30 days of delivery.'}
plan v1:
1. [x] Look up order 1042
2. [x] Read the returns policy
3. [ ] Check that a kettle is in stock
4. [ ] Send a replacement kettle
5. [ ] Email the customer that a new kettle has shippedOutput verified in CI from site/examples/building-agents/planning/progress.py.
The printed plan shows where the run is and what comes next, in words. A loop without a plan shows only the calls it has made so far. The Agent Engineer Course lists this progress view among the reasons to plan, next to steps that a plan keeps from being skipped [1].
Research on prompts measured the second reason. The Plan-and-Solve paper replaced the prompt “Let’s think step by step” with one that asked the model to understand the problem, write a plan and then carry it out. The authors wanted fewer answers that skip a reasoning step. With GPT-3 the plan-first prompt did better on every set of math word problems they tested, and its average accuracy was 2.5 percentage points higher, 72.9 against 70.4 [2]. In a sample of 100 problems, the wrong answers that skipped a step went from 12 with the old prompt to 10 with the plan-first prompt, and to 7 with a longer version of it [2]. That plan was written and followed inside one reply, with no tools, and the model didn’t revise it after it started.
A step that changes the facts
Section titled “A step that changes the facts”follow runs every step of the first plan in order. It makes one model
call for the plan. After that it runs the tools in order. Here is
the whole run.
Following the first plan
Section titled “Following the first plan”Run this, and compare what you see with the output below.
python3 agent.py followstep 1: get_order(order_id='1042') -> {'ok': True, 'item': 'kettle', 'days_since_delivery': 12}
step 2: get_policy(topic='returns') -> {'ok': True, 'text': 'Returns and refunds within 30 days of delivery.'}
step 3: check_stock(item='kettle') -> {'ok': True, 'item': 'kettle', 'in_stock': 0, 'restock_days': 21}
step 4: send_replacement(order_id='1042') -> {'ok': True, 'replacement_for': '1042', 'status': 'backordered'}
step 5: email_customer(order_id='1042', message='Your new kettle has shipped.') -> {'ok': True, 'sent_to': 'customer of 1042'}
plan v1:
1. [x] Look up order 1042
2. [x] Read the returns policy
3. [x] Check that a kettle is in stock
4. [x] Send a replacement kettle
5. [x] Email the customer that a new kettle has shipped
model calls: 1Output verified in CI from site/examples/building-agents/planning/follow.py.
Every step reports 'ok': True, and every box has an x. Step three
said that the shop has no kettles for 21 days. The plan was written
before anyone knew that, so step four booked a replacement that went on
back order, and step five told the customer a new kettle has shipped.
The plan was right when it was written and wrong after step three.
work runs the same plan in a different way. After each step it calls
the model again with the plan and every result so far, and the model
either keeps the plan or returns a new one. The steps that have run stay
as they are, and the model may change the steps still to come. Here is the
fake model’s reply to that call.
latest = results[-1]if "in_stock" in latest and latest["in_stock"] == 0: order_id = plan[0]["args"]["order_id"] item, days = latest["item"], latest["restock_days"] return [ *plan[: len(results)], make_step(f"Refund order {order_id}", "issue_refund", order_id=order_id), make_step( f"Email the customer that the {item} is refunded", "email_customer", order_id=order_id, message=f"The {item} is out of stock for {days} days, so we refunded you.", ), ]return NoneWhen the model returns a changed plan, work gives it the next
version number, so the first plan is v1 and the first revision is v2.
show_plan prints a plan vN: line and then one numbered line per
step, with [x] for a step that has run and [ ] for an open one. The
revise_only step runs the first three steps, gives the model the
results and prints only the plan it gets back.
Predict the revised plan
Section titled “Predict the revised plan”In the lesson, a fake model writes a five-step plan to replace a broken kettle from order 1042: look up the order, read the returns policy, check the stock, send a replacement, and email the customer that a new kettle has shipped. After each step the model gets the plan and the results back. When a stock result says the shop has none in stock, it keeps the steps that have run and replaces the rest with a refund of the order and an email that the item is refunded. Step three returns in_stock 0 and restock_days 21. The plan v1 steps read: Look up order 1042; Read the returns policy; Check that a kettle is in stock; Send a replacement kettle; Email the customer that a new kettle has shipped. The reply writes its new steps as Refund order, then the order id, and Email the customer that the item is refunded, with the item name in place of item. The output is the plan the model returns: a line plan vN: with the next version number after v1, then one line per step with its number, [x] for a step that has run or [ ] for an open one, and the step text.
What is the plan after the model revisits it? Predict the output, then run it.
python3 agent.py revise_onlyplan v2: 1. [x] Look up order 1042 2. [x] Read the returns policy 3. [x] Check that a kettle is in stock 4. [ ] Refund order 1042 5. [ ] Email the customer that the kettle is refunded
Output verified in CI from site/examples/building-agents/planning/revise_only.py.
Which steps have run and stay as they are, and which rule in the reply matches the result of step three?
The first three steps keep their x, and the version number tells a
reader that the plan changed. Steps four and five of plan v2 run next,
and the customer gets a refund and a message that says why. The
revise step prints the plan before and after, with the result of step
three between them.
The plan before and after
Section titled “The plan before and after”Run this, and compare what you see with the output below.
python3 agent.py revisebefore:
plan v1:
1. [x] Look up order 1042
2. [x] Read the returns policy
3. [x] Check that a kettle is in stock
4. [ ] Send a replacement kettle
5. [ ] Email the customer that a new kettle has shipped
step 3 returned: {'ok': True, 'item': 'kettle', 'in_stock': 0, 'restock_days': 21}
after:
plan v2:
1. [x] Look up order 1042
2. [x] Read the returns policy
3. [x] Check that a kettle is in stock
4. [ ] Refund order 1042
5. [ ] Email the customer that the kettle is refundedOutput verified in CI from site/examples/building-agents/planning/revise.py.
The revised run has a cost. The cost step runs the kettle goal with
follow and with work and counts the calls.
The calls of each run
Section titled “The calls of each run”Run this, and compare what you see with the output below.
python3 agent.py costfollow: 1 model call, 5 tool calls work: 6 model calls, 5 tool calls
Output verified in CI from site/examples/building-agents/planning/cost.py.
Both runs make five tool calls. work makes a model call for the plan
and one more after each step, and one of those five calls changed the
plan. The Agent Engineer Course suggests a light plan of a few steps,
run step by step, with a check of the plan after each major step
[1]. A cheaper work could call the model again only when a
result isn’t what the plan expected, and the exercise asks for that as
a stretch.
The moment to correct the plan
Section titled “The moment to correct the plan”Before step one runs, the plan is five lines of text and the shop is
unchanged. A person who reads it then can fix it for the price of
reading it. Say the customer’s message said “I want my money back”. The
person reads step four, “Send a replacement kettle”, and changes it
before any tool runs. After a run of follow, the same fix means a
cancelled back order and a second email that corrects the first.
Some agent products stop at this moment on purpose. Claude Code has a plan mode for it. In that mode the agent may look around the project first, and it may not change your files until you accept the plan it shows you. The one exception is an interactive terminal session in which bypass permissions are available. You can also reply that it should keep planning and say what to change [3]. The written plan is what makes that stop useful: a person can say no to a step while saying no costs nothing.
When is the fix cheapest?
Section titled “When is the fix cheapest?”An agent that sends a monthly newsletter writes a plan of three steps before it starts: pick the list of all customers as the audience, send the newsletter to that audience in batches of 1,000, and report how many were sent. The team wanted it sent to subscribers only, and a person on the team can read the plan and the progress of each step.
The plan’s first step picks the wrong audience. When can the person fix it at the lowest cost?
Which moves in the run can't be undone?
When to write the plan first
Section titled “When to write the plan first”Write the plan first for a task with steps that depend on each other, where a wrong early step wastes the steps after it. A plan also helps when a person or code should read the steps before they run. The kettle goal is such a task: five steps, two of them change things for the customer, and a person may want to approve them. The Agent Engineer Course advises against a plan for a simple task. Its lesson on design patterns names tasks of one step as a poor fit [4], and its lesson on planning prefers a reactive approach for tasks of fewer than three steps [1]. For those, the plan costs a call and says what the first call would have done anyway.
A written list is also one choice among several for keeping track. Claude Code has tools for a task list, and on newer models it leaves them out by default. The documentation says a newer model keeps track of work with many steps without the list, and that the tool definitions take up space in the context [5]. The plan in this lesson is written for someone to read it: the person who approves it, and the code that runs it step by step.
Which task gets a written plan first?
Section titled “Which task gets a written plan first?”The lesson has a fake model that writes a numbered plan for a goal before any tool runs, and then works through it and revisits it after each step. The plan costs one model call up front and one after each step.
Which of these tasks gains the most from a plan that the agent writes before it takes the first step?
Where would a wrong early step waste the steps after it, and where would a person want to read the steps before they run?
The exercise uses a second goal, “Replace the broken toaster from order
1090.”, with the same plan. Its step three returns a fact that
revise_plan has no rule for. The exercise step prints the plan before
and after the model revisits it.
The toaster before the exercise
Section titled “The toaster before the exercise”Run this, and compare what you see with the output below.
python3 agent.py exercisebefore:
plan v1:
1. [x] Look up order 1090
2. [x] Read the returns policy
3. [x] Check that a toaster is in stock
4. [ ] Send a replacement toaster
5. [ ] Email the customer that a new toaster has shipped
step 3 returned: {'ok': True, 'item': 'toaster', 'in_stock': 6, 'recalled': True}
after:
the model kept plan v1Output verified in CI from site/examples/building-agents/planning/exercise.py.
Exercise
The toaster’s step three returns in_stock: 6 and recalled: True, and
the model keeps plan v1, so step four would send the customer another
recalled toaster. In your copy of the
planning folder, change revise_plan so that a recalled item replaces
steps four and five with a refund and an email that names the recall,
and keep the rule for an item that is out of stock. Run
python3 agent.py exercise. The output is the plan before and after the
revision. Writing the rule yourself shows how much of the pattern is the
check after each step.
A good result: plan v2 of the toaster has steps one to three ticked, a
refund of order 1090 as step four and an email about the recall as step
five, and python3 agent.py revise prints the same kettle plans as
before. Compare with python3 model_answer.py in the same folder. Which
other result of step three should change the rest of the plan, and
which step would it change? The opening comment of model_answer.py
gives one answer.
Stretch: Give each step an expected result, such as in_stock above 0 for the stock check, and change work so that it calls the model again only when a result differs from what the step expected. Run cost again and compare the model calls.
Recap
- Planning has the agent write the steps to a goal before it takes them, as a list it works through and ticks off [1].
- A written plan shows a reader where the run is, and in research on math problems a plan-first prompt led to fewer answers that skipped a step [2].
- A plan can go stale when a step returns a new fact. Give the model the plan and the results after a step, and let it change the steps still to come, at the cost of a model call each time.
- Before the first step, the plan is text and nothing has changed, so a person can correct it cheaply, as Claude Code’s plan mode lets you do [3]. For a task of one or two steps a plan costs a call and gives little.
You can now
- Picks a design pattern for a task and says why
Write the plan first, or skip it?
Section titled “Write the plan first, or skip it?”The lesson has a fake model that writes a numbered plan for a goal before any tool runs, and then revisits it after each step. A plan costs model calls, and it gives a person the steps to read before anything changes.
Does the task have several dependent steps, or steps that a person should approve before they run?
What went wrong with the plan?
Section titled “What went wrong with the plan?”A team's agent writes a four-step plan to book a meeting room: find a free room, check the room size, book the room and send the invitation. Its code then runs the four steps in order without calling the model again. Step two returns that the room holds four people, and the meeting has ten.
The agent books a room for four and sends ten people an invitation to it. Every step reported success. What is the cause?
When did the plan last see a result?
When does the person read the plan?
Section titled “When does the person read the plan?”An agent that cleans up a shared drive writes a plan of six steps before it starts: list the folders, find files older than two years, move them to an archive folder, delete the empty folders, delete the files in the archive folder that are larger than 1 GB, and email a summary. A person on the team can read the plan.
The agent has written its plan and is about to start. When and how should the person on the team use the plan?
At which point has nothing on the drive changed yet?
References
Section titled “References”- Addy Osmani, Ivar Soares Urdalen, Leo Simons. Planning and reasoning: the loop, plan-then-execute, hierarchy. Agent Engineer Course. Course.
AEC-06 - Lei Wang, Wanyu Xu, Yihuai Lan and 4 others. Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL 2023), 2609-2634. Paper.
Wang 2023 - Anthropic. Choose a permission mode. Claude Code documentation. Reference.
Claude Code permission modes - Addy Osmani, Ivar Soares Urdalen, Leo Simons. Agentic design patterns: ReAct, reflection, tool use, planning. Agent Engineer Course. Course.
AEC-04 - Anthropic. Tools reference. Claude Code documentation. Reference.
Claude Code tools reference