Tests that prove behavior and docs that stay current

A coding agent writes tests quickly, and they pass. That tells you the tests agree with the code, and it doesn’t tell you they would notice if the code were wrong. In this lesson you break a small program on purpose and run the tests an agent wrote for it. Some of them still pass, and you rewrite one of them so it checks what the program must do. Then you review a change the agent made to the same program, and find the docs it left out of date.

The program is shop/, a pricing module for a small web shop, in the course repository. It adds up the lines of an order and adds shipping: 4.95 euros (EUR), or nothing for an order of 50.00 EUR or more.

FREE_SHIPPING_FROM = 50.00
SHIPPING = 4.95


def shipping(amount):
    """Return the shipping cost for an order with this subtotal."""
    if amount >= FREE_SHIPPING_FROM:
        return 0.0
    return SHIPPING

Break the code on purpose

Testing with an agent means keeping tests that prove what the code does for its users, and reviewing the tests the agent writes as closely as the code. Agents produce tests readily, and some of those tests can pass because they repeat what the code happens to do. The Academy course The AI-native SDLC playbook (SDLC is short for the development life cycle of software) argues that when an agent produces most of a team’s code, review and testing become the slow steps [1]. A quick way to review a test is to make the code wrong and see whether the test notices.

From the root of your clone of the course repository, copy the shop to a place of your own, and run its tests there.

cp -R site/examples/coding-with-agents/tests-and-docs-that-hold/shop ~/shop
cd ~/shop
python3 test_pricing.py; echo "exit status $?"

Example · run it

The agent's tests, before any change

Run the tests in your copy and compare what you see with the output below.

Output

$ python3 test_pricing.py
test_subtotal_adds_the_lines: pass
test_small_order_pays_shipping: pass
test_large_order_ships_free: pass
test_total_adds_shipping: pass
test_total_of_a_known_order: pass
5 passed, 0 failed
exit status 0

Output verified in CI from site/examples/coding-with-agents/tests-and-docs-that-hold/run_tests.py.

The examples in this lesson aren’t graded. Each one is a command you run, with output the course checks on every build.

The tests are in test_pricing.py. ORDER is two items at 12.50 EUR and one at 20.00 EUR.

ORDER = [(2, 12.50), (1, 20.00)]


def test_subtotal_adds_the_lines():
    assert subtotal(ORDER) == 45.00, subtotal(ORDER)


def test_small_order_pays_shipping():
    assert shipping(10.00) == SHIPPING, shipping(10.00)


def test_large_order_ships_free():
    assert shipping(FREE_SHIPPING_FROM + 10) == 0.0, shipping(FREE_SHIPPING_FROM + 10)


def test_total_adds_shipping():
    expected = subtotal(ORDER) + shipping(subtotal(ORDER))
    assert total(ORDER) == expected, total(ORDER)


def test_total_of_a_known_order():
    assert total(ORDER) == 49.95, total(ORDER)

Now make a bug a customer would see on every small order. Open pricing.py in your copy and change SHIPPING = 4.95 to SHIPPING = 5.95. Before you run the tests again, decide which ones still pass.

Checkpoint · multi-choice

Which tests miss the wrong price?

The constant now holds 5.95 where the shop charges 4.95. Which of the five tests still pass?

Select exactly 4.

`test_subtotal_adds_the_lines``test_small_order_pays_shipping``test_large_order_ships_free``test_total_adds_shipping``test_total_of_a_known_order`

CheckHintSkip

Example · run it

The tests, with the wrong price

Run the tests in your copy again. The course’s script makes the same change to a fresh copy and runs the tests. You can run it from site/examples/coding-with-agents/tests-and-docs-that-hold/ in your clone.

python3 break_price.py

Output

$ python3 test_pricing.py
test_subtotal_adds_the_lines: pass
test_small_order_pays_shipping: pass
test_large_order_ships_free: pass
test_total_adds_shipping: pass
test_total_of_a_known_order: FAIL: got 50.95
4 passed, 1 failed
exit status 1

Output verified in CI from site/examples/coding-with-agents/tests-and-docs-that-hold/break_price.py.

One test out of five caught a wrong price. Two of the tests that passed never looked at the price, and that’s fine: a test for the subtotal doesn’t need to. The other two mirror the implementation. test_small_order_pays_shipping compares the shipping cost with SHIPPING, the same constant the code returns, so it passes whatever the constant holds. test_total_adds_shipping works out the expected total by calling the same functions that total calls. A test like this repeats the code, and a test that repeats the code agrees with it when the code is wrong.

The fix is to write down, in the test, the value the program must produce, worked out by hand or taken from the spec. Rewrite test_small_order_pays_shipping in your copy so it states the price the shop charges:

def test_small_order_pays_shipping():
    assert shipping(10.00) == 4.95, shipping(10.00)

Example · run it

The rewritten test, with the wrong price

Run the tests with the wrong price still in pricing.py. The course’s script makes both changes to a fresh copy:

python3 rewrite_test.py

Output

$ python3 test_pricing.py
test_subtotal_adds_the_lines: pass
test_small_order_pays_shipping: FAIL: got 5.95
test_large_order_ships_free: pass
test_total_adds_shipping: pass
test_total_of_a_known_order: FAIL: got 50.95
3 passed, 2 failed
exit status 1

Output verified in CI from site/examples/coding-with-agents/tests-and-docs-that-hold/rewrite_test.py.

The rewritten test now fails, and its line names the wrong value. Put SHIPPING = 4.95 back in your copy and run the tests once more: all five pass again, and the rewritten test now proves the price.

Checkpoint · choice

Which rewrite would catch the wrong price?

test_total_adds_shipping passed with the wrong price. Which line in its place would fail while SHIPPING holds 5.95?

`assert total(ORDER) == subtotal(ORDER) + SHIPPING``assert total([(1, 30.00)]) == 34.95``assert total(ORDER) > subtotal(ORDER)``assert total(ORDER) == round(total(ORDER), 2)`

CheckHintSkip

Pitfall: The test that learned the new answer

An engineer asked an agent to add discount codes to a checkout. After the change, a test that checked the total of a known order failed by one cent, because the new code rounded twice. The agent changed the assertion to compare the totals to one decimal place, and the suite went green while every order with a discount showed a total one cent off. A new expected value in a test is a change to what the program promises. Review it like a change to the spec, and reject one the brief didn’t ask for.

Keep the tests fast and unchanged

An agent reruns the tests after every change. How long they take decides how many attempts fit in a loop: a one-second suite can run after each edit, and a twenty-minute suite gets run once at the end, or skipped. The example instructions file in the vendor guide for Claude Code has a line for this. It asks the agent to prefer running single tests to the whole suite, for speed [2]. Give the agent the fast tests for the part it changes, and run the full suite before the merge.

The agent also must not change the tests to get a pass. The shortest way to make a failing change pass is to delete the test, skip it or loosen its assertion, as the agent in the pitfall did. When an agent may edit the tests, the blast radius of one wrong change includes the check that should catch it. So the rule for an agent is that it never deletes or weakens a test. A test it thinks is wrong makes it stop and explain why, and the decision is yours.

A rule in the project instructions is a request. Claude Code reads its instructions file when a session starts and uses it as context, and nothing in the file is enforced [3]. The rule makes a weakened test less likely, and the review of the test files is what finds one.

Checkpoint · choice

The suite is too slow for the loop

Which way of running the tests fits the agent’s loop?

Let the agent mark the slowest tests as skipped for the run, and turn them back on before the merge.Run no tests in the loop, and run the full suite yourself once the agent says it is done.Run the fast tests for that module after every change, and run the full suite once before the merge.Run the full suite after every change, and drop the limit of five attempts so the agent has time.

CheckHintSkip

Checkpoint · scenario

The tests pass, and one assertion changed

The brief didn’t ask for a change to the total of an order without a discount code. What do you do?

Accept the change, since the new assertion still checks that the total is about right.Put the old assertion back, run the tests, and brief the agent again, and tell it not to change the existing tests.Ask the agent why it changed the test, and accept the change if its reason sounds right.Keep the new assertion, and add a comment above it that says why the check was loosened.

DecideHintSkip

Documentation is part of the change

Documentation is the written explanation of how a system behaves: the README, the pages in docs/, and the comments and docstrings in the code. An agent can write good documentation from code, and it can just as easily change the code and leave the documentation saying something else. Treat the docs as part of the change. The brief names what to update, the project instructions say where the docs are, and the review checks them.

The instructions file of the shop, AGENTS.md, says where its docs are. The Claude Code guide suggests that an instructions file links to detailed documentation and doesn’t copy it in [2].

- Run `python3 test_pricing.py` after every change, and read what it prints.
- Never delete a test or loosen an assertion to make a change pass. If a
  test looks wrong, stop and say why.
- Documentation is in three places: `README.md` for customers,
  `docs/pricing.md` for the full rules, and the docstrings in `pricing.py`.
  A change to a price or a threshold updates all three in the same change.

The shop raises its free-shipping threshold. The brief and the reply below are illustrative and don’t record a specific model’s words.

Prompt (illustrative, not a recorded transcript)

Raise the free-shipping threshold in pricing.py from 50.00 EUR to 60.00 EUR, and add a test at the new threshold. Update the documentation that AGENTS.md lists. Run python3 test_pricing.py after each change. Done when the tests pass and none of the places AGENTS.md lists still gives 50.00 EUR as the threshold.

Response

I changed FREE_SHIPPING_FROM to 60.00 and updated the docstring of pricing.py. I added test_free_shipping_starts_at_sixty, which checks 60.00 and 59.99. The tests print 6 passed, 0 failed.

The reply lists the docstring and says nothing about the other two places. You brief your own agent the same way in the exercise at the end of the lesson. To see what the change touched, make the copy a git repository with one tagged commit before the run:

git init
git add .
git commit -m "Before the agent's change"
git tag before-change

After the run, stage everything with git add -A, so that a file the agent created shows as well, and list what differs from the tag. Then search the three places for the old threshold.

Example · run it

What the change touched, and what it missed

The course’s script makes the agent’s change to a fresh copy of the shop, committed and tagged before-change first, and runs the same commands you run after your own agent’s run in the exercise. You can run it from site/examples/coding-with-agents/tests-and-docs-that-hold/ in your clone.

python3 the_change.py

Output

$ git add -A
$ git diff --cached --stat before-change
 pricing.py      | 4 ++--
 test_pricing.py | 6 ++++++
 2 files changed, 8 insertions(+), 2 deletions(-)
$ python3 test_pricing.py
test_subtotal_adds_the_lines: pass
test_small_order_pays_shipping: pass
test_large_order_ships_free: pass
test_total_adds_shipping: pass
test_total_of_a_known_order: pass
test_free_shipping_starts_at_sixty: pass
6 passed, 0 failed
exit status 0
$ grep -n "50.00" README.md docs/pricing.md pricing.py
README.md:5:Shipping costs 4.95 EUR. Orders of 50.00 EUR or more ship free.
docs/pricing.md:7:the free-shipping threshold, and free from 50.00 EUR. The threshold is

Output verified in CI from site/examples/coding-with-agents/tests-and-docs-that-hold/the_change.py.

The tests pass, and the list of changed files holds neither README.md nor docs/pricing.md. The search finds the old threshold in README.md and in docs/pricing.md, and nothing in pricing.py, whose docstring the agent did update. A customer who reads the README expects free shipping on a 55.00 EUR order and pays 4.95 EUR. Send the two stale lines back to the agent, and review the next diff the same way.

Checkpoint · scenario

The tests pass. Is the change done?

What do you do before you approve the change?

Approve it, since the tests pass and the docstring next to the constant was updated as well.Ask the agent to confirm that it updated all the documentation, and approve when it says yes.Ask the agent to write the documentation again for the whole project, from the code as it is now.Search the places the instructions list for the old threshold, and send the stale lines back.

DecideHintSkip

Exercise

The exercise takes about ten minutes and gives you one rewritten test and a short list of stale lines. It practices the checks that find a test that can’t fail and documentation that no longer matches the code.

From the root of your clone, make a fresh copy of the shop for the first part:

cp -R site/examples/coding-with-agents/tests-and-docs-that-hold/shop ~/shop-boundary
cd ~/shop-boundary

In shipping, change >= to >. Before you run the tests, write down which of the five you expect to fail. Run python3 test_pricing.py and compare. For every test that passes, name one bug in pricing.py it would not catch. Then rewrite test_large_order_ships_free so that it fails with the break in place and passes without it.

For the documentation, go back to the root of your clone, make one more fresh copy and give it a tagged commit:

cp -R site/examples/coding-with-agents/tests-and-docs-that-hold/shop ~/shop-docs
cd ~/shop-docs
git init
git add .
git commit -m "Before the agent's change"
git tag before-change

Start your own coding agent inside ~/shop-docs, so it works on the copy and not on your clone, and give it the brief from this lesson. When it is done, run these commands:

git add -A
git diff --cached --stat before-change
python3 test_pricing.py
grep -n "50.00" README.md docs/pricing.md pricing.py

A good result has a list of expected failures that matches the run, a rewritten test that fails with the break and passes without it, and a review that names every place AGENTS.md lists that still gives the old threshold, or confirms that none does. Then answer one question: which test in your own project would you break first, and why that one?

Stretch: Then pick one test an agent wrote in your own project, break the line of code it is about on purpose, and see whether the test fails.

Recap

1. Review an agent’s test by making the code wrong on purpose and running it. A test that still passes either gives the code an input that never reaches the broken line, or takes its expected value from the code it tests, so it mirrors the implementation. Rewrite it with an input that reaches the line and the value the program must produce, from the spec or worked out by hand.
2. Give the agent the fast tests for the part it changes to run in its loop [2], and run the full suite before the merge. The agent never deletes, skips or loosens a test to make a change pass. When a test looks wrong, it stops and says why.
3. A rule in the instructions file is context for the agent, and nothing enforces it [3]. Read the diff of the test files to confirm that no test was deleted, skipped or loosened.
4. Documentation is part of the change. The brief names what to update, the instructions say where the docs are, and the review searches those places for what the change made stale.

You can now

• Reviews code they did not write, against the specification
• Turns a check into a bounded, self-checking loop

Mark lesson finishedI know this, skip it

Habit

The next time an agent’s change comes with new tests, change one line of its code to make it wrong, run the tests, and see which of them notice before you approve.

References

1. Anthropic. The AI-native SDLC playbook. Claude Academy. Course. Academy ai-native-sdlc-playbook
2. Anthropic. Best practices for Claude Code. Claude Code documentation. Reference. Claude Code best practices
3. Anthropic. How Claude remembers your project. Claude Code documentation. Reference. Claude Code memory