Cost, caching and pinning the model

In this lesson we give an agent loop a bill. The loop is the one from Building your first agent: send the message list to the model, run the tool it asks for, append the result, and repeat until it answers. This time every call to the model goes through a provider that counts tokens and charges for them. We add a cost record per run and a token budget that stops a run. Then we cache the part of the prompt that never changes, and retry a call that the provider turns down with a rate limit. The last section pins the model, so that its behavior changes only on a date you choose.

The provider is fake. It runs inside the program and never reaches the network, and no application programming interface (API) key is needed. Its token counter treats every word and every punctuation mark as one token, which is a rough stand-in for a real tokenizer. The prices are made up, and the cache prices follow the Claude API’s rule for most models, which is described below. The files are in site/examples/building-agents/cost-and-provider-operations/. Copy the folder, and run each step from inside the copy as python3 ops.py <step>.

What one run costs

The agent is a help desk assistant for a small office, with three tools: get_ticket, search_kb for the knowledge base and read_log for last night’s log of an office service. Task t1 is a question about a laptop that will not start. The run makes three calls: the model asks for the ticket, then for the knowledge base article, and then it answers.

A real provider reports the tokens of each call in its response. The Claude API returns input_tokens and output_tokens, and two more fields for the prompt cache, cache_creation_input_tokens and cache_read_input_tokens [1]. The price is per million tokens, output tokens cost several times as much as input tokens, and the tool definitions you send count as input [2]. The fake provider returns the same four fields, and one function turns them into dollars:

PRICE_INPUT = 3.00  # dollars per million tokens, made up
PRICE_OUTPUT = 15.00
PRICE_CACHE_WRITE = PRICE_INPUT * 1.25
PRICE_CACHE_READ = PRICE_INPUT * 0.1

def dollars(usage: dict) -> float:
    return (
        usage["input_tokens"] * PRICE_INPUT
        + usage["cache_creation_input_tokens"] * PRICE_CACHE_WRITE
        + usage["cache_read_input_tokens"] * PRICE_CACHE_READ
        + usage["output_tokens"] * PRICE_OUTPUT
    ) / 1_000_000

The loop keeps a CostRecord per run, with the number of calls, the tokens and the dollars, and adds each response’s usage to it. With log=print it prints one line per call.

Example · run it

The bill of one run

Run this, and compare what you see with the output below.

python3 ops.py cost

Output

call 1: input 260, output 23, $0.001125
call 2: input 353, output 26, $0.001449
call 3: input 443, output 28, $0.001749
stop: end_turn
answer: Ticket 311 is open. Hold the power button for 15 seconds, then charge it for 10 minutes.
calls: 3
tokens: 1133
cost: $0.004323

Output verified in CI from site/examples/building-agents/cost-and-provider-operations/cost.py.

The input grows with every call. Call 2 sends everything call 1 sent, plus the model’s tool request and the ticket, and call 3 adds the article. A loop pays for its whole history again on every turn, so the number of turns and the size of each tool result decide the bill. The system prompt and the tool list are 226 of the 260 input tokens of call 1, and they are in every call. A run costs less than half a cent here, and the number that matters is that cost times the runs per day. Track it per run and per task type, and alert on it, so a change in cost shows up the day it happens [3].

A budget that stops the run

A step limit catches a loop that repeats too often, and Stopping the loop on purpose adds a character budget next to it. When a run has few turns and one of them is large, such as a tool result much longer than expected, only a token budget stops it. The Agent Engineer Course sets a hard token cap per session for this reason [3]. The loop checks the budget before each call, and when the record has reached it, the run stops with its own reason:

for _ in range(max_steps):
    if budget is not None and record.tokens >= budget:
        return outcome("budget_spent", None)
    try:
        response = call_provider(provider, model, messages, cache, no_sleep, log, record.calls + 1)
    except ProviderError as err:
        return outcome("provider_error", None, f"{err.status} {err.kind}: {err.message}")
    record.add(response["usage"])
    reply = response["reply"]
    if "answer" in reply:
        return outcome("end_turn", reply["answer"])
    ...

outcome returns a dict with the stop reason, the answer, and the record’s calls, tokens and cost, in that order. The cost is a string with a dollar sign and six decimals, such as '$0.004323'. call_provider and the provider_error stop are the subject of a later section. record.tokens counts every token of every call, input and output.

Checkpoint · predict

What does the budget stop return?

Task t1 runs again, now with budget=600. Use the per-call numbers from the run above, and predict what this prints.

print(run("t1", budget=600))

Then run it:

python3 ops.py budget

What is printed?

Output verified in CI from site/examples/building-agents/cost-and-provider-operations/budget.py.

CheckHintSkip

The run spent 662 tokens, more than the budget of 600, because the check runs before a call and a call can’t be stopped halfway. Set the budget with that in mind. The caller gets the stop reason, and the tokens and the cost it has already paid for, and can decide what to tell the user.

Caching the part that never changes

The system prompt and the tool list are the same on every call of every run. Prompt caching lets the provider process that prefix once and read it back on later calls. On the Claude API the cache follows the order of the request, the tools first, then the system prompt, then the messages. A cache hit needs an exact match of everything up to the cache breakpoint, and a cached prefix lasts five minutes by default [1]. For most models, the price of writing the prefix to the cache is 1.25 times the input price, and the price of reading it back is 0.1 times the input price [2]. The API caches a prefix only above a minimum length, which depends on the model [1].

The fake provider caches the system prompt and the tool list together when a run passes cache=True. The first call writes them to the cache, and later calls read them.

Example · run it

The same run with the prefix cached

Run this, and compare what you see with the output below.

python3 ops.py cache

Output

call 1: input 34, cache write 226, output 23, $0.001295
call 2: input 127, cache read 226, output 26, $0.000839
call 3: input 217, cache read 226, output 28, $0.001139
stop: end_turn
answer: Ticket 311 is open. Hold the power button for 15 seconds, then charge it for 10 minutes.
calls: 3
tokens: 1133
cost: $0.003272

Output verified in CI from site/examples/building-agents/cost-and-provider-operations/cache.py.

Compared with the first run, call 1 costs more by the extra price of the cache write, while calls 2 and 3 cost less, and the run drops from $0.004323 to $0.003272. The token count stays 1133: caching changes the price of the tokens, and the model still reads all of them. The more calls share one prefix, the more a run saves. With the Claude API, a run that starts while the prefix is still in the cache reads it on its first call as well. Tokens read from the cache also don’t count toward the input-tokens-per-minute rate limit for most Claude models [4].

Checkpoint · choice

The time in the system prompt

A teammate adds the current time, to the second, at the start of the system prompt, so that the model knows when it answers. Caching is on. What happens to the cost of the prefix?

Every call writes the prefix to the cache again, so the prefix costs more than it did without caching.Only the first call of each run writes the prefix, and later calls in the run read it back.The fake provider still reads the prefix from the cache, because it looks the prefix up by model id.The calls fail, because a cached prefix must not change between calls of one run.

CheckHintSkip

Calling the provider reliably

A provider refuses calls, and the loop must know which refusals to wait out. The Claude API names its errors by Hypertext Transfer Protocol (HTTP) status and type. Retry a 429 rate_limit_error, a 500 api_error, a 504 timeout_error and a 529 overloaded_error after a wait [5]. A 429 for a rate limit comes with a retry-after header that says how many seconds to wait [4]. Report the rest to the caller: a 400 invalid_request_error or a 401 authentication_error fails again on every retry. So does a 429 that says the monthly spend cap is reached, which has no retry-after header [5]. The official software development kits (SDKs) already retry transient failures twice by default, with exponential backoff, and honor retry-after [5]. A loop you write yourself needs the same rules.

The fixture’s call_provider wraps every call. It passes a timeout, so a call that hangs becomes an error the loop can handle. A timeout or a retriable status leads to a retry. Before the retry the loop waits for the retry-after seconds when the provider sends them, or for 1, 2 and then 4 seconds when it doesn’t. Any other error goes back to the caller.

RETRIABLE = {429, 500, 504, 529}

def call_provider(provider, model, messages, cache, sleep, log, number, max_retries=3, timeout=30.0):
    attempt = 1
    while True:
        try:
            return provider.create(model, SYSTEM, TOOL_LIST, messages, cache=cache, timeout=timeout)
        except CallTimeoutError as err:
            status, text, wait_for = "timeout", str(err), None
        except ProviderError as err:
            if err.status not in RETRIABLE:
                raise
            status, text, wait_for = f"{err.status} {err.kind}", err.message, err.retry_after
        if attempt > max_retries:
            raise ProviderError(0, "gave_up", f"{status} after {attempt} attempts")
        wait = float(wait_for) if wait_for is not None else 1.0 * 2 ** (attempt - 1)
        log(f"call {number}: {status}: {text}; wait {wait:.1f}s and retry")
        sleep(wait)
        attempt += 1

run turns a ProviderError into the stop reason provider_error, with the error in the result. The fixture never really waits: its sleep does nothing, and the log line says how long a real loop would wait. The fake provider bills nothing for a call it refuses.

Checkpoint · predict

A rate limit on the third call

The fake provider now refuses the third call once. The step prints only the retry lines of the log, then the calls, tokens and cost of the run. Predict what it prints. The run without errors made 3 calls for 1133 tokens and $0.004323.

limited = Provider(
    refuse={3: ProviderError(429, "rate_limit_error", "rate limit exceeded", retry_after=2)}
)
result = run("t1", provider=limited, log=retries_only)
print(f"calls: {result['calls']}, tokens: {result['tokens']}, cost: {result['cost']}")

Then run it:

python3 ops.py rate_limit

What is printed?

Output verified in CI from site/examples/building-agents/cost-and-provider-operations/rate_limit.py.

CheckHintSkip

The run ends with the same calls, tokens and cost as the run without the error, and the log has one retry line. A person reading the log sees that the limit was hit, and the user sees nothing. The errors step shows the other two paths. The second call times out once and is retried after the default wait of one second. Then the provider refuses the third call with a 400, and the loop stops at once.

Example · run it

A timeout, then a terminal error

Run this, and compare what you see with the output below.

python3 ops.py errors

Output

call 1: input 260, output 23, $0.001125
call 2: timeout: no reply within 30s; wait 1.0s and retry
call 2: input 353, output 26, $0.001449
stop: provider_error
answer: None
calls: 2
tokens: 662
cost: $0.002574
error: 400 invalid_request_error: messages: bad format

Output verified in CI from site/examples/building-agents/cost-and-provider-operations/errors.py.

The 400 is not retried, because the same request would fail the same way three more times. The caller gets the error, the 662 tokens and the cost already spent, and can report the problem to a person.

Pinning the model

Each Claude model id names one fixed version of the model. From the 4.6 generation on, an id such as claude-sonnet-4-6 is itself that version, and an updated model ships under a new id. Earlier models also have aliases on the Claude API: claude-sonnet-4-5 points to the newest dated version with that number, such as claude-sonnet-4-5-20250929. The same page notes that the serving systems around a model can change over time, so small differences can appear even on a fixed id [6]. A fixed id doesn’t last forever either. Each id has its own retirement date, with at least 60 days’ notice for a public model, and after that date requests to it fail [7]. Pinning lets you choose when the model changes and plan the move. You still have to move before the retirement date.

Choosing which model to pin is a trade between cost, speed and quality, and a simple step can go to a smaller, cheaper model [8] [3]. The fixture pins fake-helper-2-1. It also has an alias, fake-helper-latest, which the fake provider moved to fake-helper-3-0 on 2026-06-01. The alias step runs task t1 on the alias one day before that date and on that date.

Example · run it

The same code on two dates

Run this, and compare what you see with the output below.

python3 ops.py alias

Output

2026-05-31: fake-helper-2-1, calls 3, cost $0.004323
2026-06-01: fake-helper-3-0, calls 4, cost $0.006597

Output verified in CI from site/examples/building-agents/cost-and-provider-operations/alias.py.

Nobody changed the code or the prompt. The newer model makes one more call and writes a longer answer, and the run costs about half as much again.

Pitfall: An alias in the configuration

A team writes an alias such as fake-helper-latest into its agent’s configuration, because it always gives them the newest model. On a date the provider chose, the alias moves. The agent’s answers get longer, it makes an extra call on some tasks, and the cost per run jumps, with nothing in the team’s own history to explain it. Rule: pin an exact model id, and treat a new model as a change like a new prompt. It passes the evaluation suite first, then goes to a small share of the traffic, and reaches everyone only when its numbers hold.

Exercise

Copy the folder and add a table step to ops.py. For each of the five tasks in TASKS it calls run and prints one row with the task id, the calls, the tokens and the cost. The last line is the average cost of the five tasks. Find the task whose cost is far above the others, and run it with log=print to see which call makes the difference. Name the tool that causes it. A cost table per task type is the view that finds this in production, where no one reads each run.

A good result: the other four tasks cost between $0.002364 and $0.004323, one task costs $0.010635, and the average of the five is $0.004804. So the costly task is more than twice the average you printed, and about three times the average of the other four. Its trace shows a small first call and a much larger second call, because the tool returned a long result and the loop sent all of it to the model. Would a token budget have stopped that run, and what budget would you set for this agent?

Stretch: Change read_log so that it returns only the lines that report a stop or an error, and run the table again. How far does that task's cost drop, and what could the model no longer see?

Recap

1. A loop sends its whole history again on every call, so the number of turns and the size of each tool result decide what a run costs. Keep a cost record per run from the usage the provider reports [1] [3].
2. A token budget stops a run with a named reason, and the caller still gets the tokens and the cost already spent. The check runs before a call, so a run can end a little over the budget.
3. Prompt caching writes the unchanging prefix once and reads it back at a fraction of the input price, but only while the prefix matches exactly [1] [2].
4. Retry rate limits, overload and timeouts with a wait, and use retry-after when the provider sends it. Report every other error to the caller [5] [4].
5. Pin an exact model id, and roll out a new model like any other change. An alias moves on the provider’s date, and a pinned id retires on a date you are told about in advance [6] [7].

You can now

• Manages cost and rolls out changes without breaking users

Mark lesson finishedI know this, skip it

Habit

Before an agent gets a new tool, look at the size of one real result from it. The loop sends that result to the model again on every later call of the run.

References

1. Anthropic. Prompt caching. Claude Platform documentation. Reference. Claude docs prompt-caching
2. Anthropic. Pricing. Claude Platform documentation. Reference. Claude docs pricing
3. Addy Osmani, Ivar Soares Urdalen, Leo Simons. From prototype to production: eval-gated deploys, rollout, cost. Agent Engineer Course. Course. AEC-11
4. Anthropic. Rate limits. Claude Platform documentation. Reference. Claude docs rate-limits
5. Anthropic. Claude API errors. Claude Platform documentation. Reference. Claude docs errors
6. Anthropic. Model IDs and versioning. Claude Platform documentation. Reference. Claude docs model-ids-and-versions
7. Anthropic. Model deprecations. Claude Platform documentation. Reference. Claude docs model-deprecations
8. Anthropic. Claude Platform 101. Claude Academy. Course. Academy claude-platform-101