Retrieval as a tool the agent calls

In this lesson the agent gets a tool that looks things up. The concepts course showed retrieval as a search that runs before the model answers, so that the right passage is in the prompt when the model reads the question, and it showed what a keyword search gets wrong. Here we build that search as a tool, run it two ways against a twelve-document company handbook, and measure each way against a small test set. The basic version calls the tool once per question. The loop version lets the agent call it, read the result, judge it and call it again, and the lesson ends with the rule for choosing between them.

Every example runs against a fixture with a fake model, so you can predict the output and check yourself. The complete program is site/examples/building-agents/retrieval-as-a-tool/agent.py in the repository, with the twelve texts under docs/ next to it. Copy the folder, and each step below is python3 agent.py <step>. The fake model answers with the sentence of the retrieved passage that shares the most keywords with the question. That’s a deterministic replacement for a real model reading the passage, and it makes every run print the same lines. The fixture reads its documents with a glob over docs/*.txt, and a stray file in the folder doesn’t change the output.

The search is a tool

A tool is a function plus a description, as Building your first agent showed. The search here is one more entry in the same TOOLS dictionary. It splits the query into keywords, drops common words, scores each document by how many keywords it contains, and returns the best one. The skip argument names documents already read. A second search with skip set finds a different document.

def search_docs(query: str, skip: Optional[list[str]] = None) -> dict:
    terms = keywords(query)
    scores = {
        name: sum(1 for term in terms if term in set(words(text)))
        for name, text in CORPUS.items()
        if name not in (skip or [])
    }
    if not scores:  # every document was skipped, or docs/ is empty
        return {"ok": True, "name": None, "passage": ""}
    best = max(scores, key=lambda name: scores[name])  # the first document wins a tie
    if scores[best] == 0:
        return {"ok": True, "name": None, "passage": ""}
    return {"ok": True, "name": best, "passage": CORPUS[best]}


TOOLS = {
    "search_docs": {
        "fn": search_docs,
        "description": (
            "Search the handbook. Returns the best-matching passage. "
            "Args: query (str), skip (list of str, optional): names of documents already read."
        ),
    },
}

With a real model, the description is what the model reads to decide when to call the tool, and the passage goes back to the model as a tool_result block in the next user message [1] [2]. In this fixture the passage goes into a prompt built by build_prompt, with a frame above it, and the question below it. The frame tells the model where the text came from and what to do when it doesn’t answer the question, and the last section of this lesson shows what happens without it.

Example · run it

The prompt the model gets

Run this, and compare what you see with the output below.

result = TOOLS["search_docs"]["fn"]("How many days of annual leave do I get?")
print(build_prompt(result["passage"], "How many days of annual leave do I get?"))

Output

The passages below were retrieved from the company handbook for this question. Answer from them. Quote the sentence you relied on. If the passages do not cover the question, answer exactly: not found.

Passages:
Annual leave. Every employee accrues 25 days of annual leave per year. Unused days carry over until 31 March. Requests go to your manager at least two weeks ahead.

Question: How many days of annual leave do I get?

Output verified in CI from site/examples/building-agents/retrieval-as-a-tool/prompt.py.

Basic retrieval against a test set

The basic version is the fixed pipeline from the concepts course: search once, put the passage in the prompt, ask [3]. The harness calls the tool before the model sees the question, and the model doesn’t decide anything about retrieval.

def answer_basic(question: str) -> dict:
    result = TOOLS["search_docs"]["fn"](question)
    reply = fake_model(build_prompt(result["passage"], question))
    return {"name": result["name"], "answer": reply, "rounds": 1}

A test set decides whether this is good enough. This one has six questions in the handbook’s own words, except the second, which asks about holiday where the handbook says leave. The concepts course showed what a keyword search does with that.

TEST_SET = [
    "How many days of annual leave do I get?",
    "How much holiday can I take?",
    "What is the training budget per year?",
    "How many days per week can I work from home?",
    "How many days ahead are flights booked?",
    "Who do I report a lost laptop to?",
]

The twelve handbook files are data-team.txt, expenses.txt, laptops.txt, leave.txt, office-hours.txt, onboarding.txt, payments-team.txt, remote-work.txt, security.txt, service-desk.txt, training.txt and travel.txt. Each file holds a few sentences on the topic its name gives. The list_sources function runs the basic version on each question. For each question it prints a line in the form <n>. <file>, where the file is the source of the answer. A question without an answer gets <n>. not found. The last line is answered: <count> of 6.

Checkpoint · predict

Which questions does the basic version answer?

Predict the output first, one <n>. <file> or <n>. not found line per question and the answered: <count> of 6 line. Then run this.

list_sources(TEST_SET)

What is printed?

Output verified in CI from site/examples/building-agents/retrieval-as-a-tool/basic.py.

CheckHintSkip

The run_test_set function prints each question with the file it retrieved, or nothing retrieved, and its answer, and the same count at the end.

Example · run it

The answers of the basic version

Run this, and compare what you see with the output below.

run_test_set(TEST_SET)

Output

1. How many days of annual leave do I get?
   leave.txt: Every employee accrues 25 days of annual leave per year.
2. How much holiday can I take?
   nothing retrieved: not found
3. What is the training budget per year?
   training.txt: Every employee has a training budget of 1500 euros per year.
4. How many days per week can I work from home?
   remote-work.txt: Staff may work from home up to three days per week.
5. How many days ahead are flights booked?
   travel.txt: Flights are booked through the travel desk at least 14 days ahead.
6. Who do I report a lost laptop to?
   laptops.txt: Report a lost laptop to the service desk the same day.
answered: 5 of 6

Output verified in CI from site/examples/building-agents/retrieval-as-a-tool/basic_answers.py.

Five of six, and the miss is the question in a colleague’s words. A better index fixes that one: a similarity search that scores meaning, or a synonym list that maps holiday to leave. Nothing about the loop is needed for it, and the next section is about the one failure a better index doesn’t fix.

One question that needs two searches

Question seven asks who approves a replacement laptop for the payments team. The laptops text says the cost center owner of the team approves it and sends the reader to the team pages, and the payments team’s text names its cost center owner. The answer is in two documents, and the basic version reads one.

Example · run it

The basic version on question seven

Run this, and compare what you see with the output below.

show(7, MULTI_HOP, answer_basic(MULTI_HOP))

Output

7. Who approves a replacement laptop for the payments team?
   laptops.txt: The cost center owner of your team approves a replacement laptop before that, see the team pages.

Output verified in CI from site/examples/building-agents/retrieval-as-a-tool/multi_hop_basic.py.

The answer is a pointer, and the person asking still has no name. Retrieval Augmented Generation (RAG) is the full name behind the letters in the agentic RAG loop, and the loop puts the agent in charge of the search: it forms a query, reads what comes back, judges whether it answers the question, and searches again with a new query when it doesn’t [3]. Question seven is a multi-hop question, where the first result says what to search for next, and that’s the case the loop is for.

def answer_with_loop(question: str, max_rounds: int = 3) -> dict:
    query = question
    read: list[str] = []
    for round_number in range(1, max_rounds + 1):
        result = TOOLS["search_docs"]["fn"](query, skip=read)
        if result["name"] is None:
            return {"name": None, "answer": "not found", "rounds": round_number}
        read.append(result["name"])
        sentence = best_sentence(query, result["passage"])
        print(f"round {round_number}: search_docs(query={query!r}) -> {result['name']}")
        if is_pointer(sentence):
            # Follow the pointer: keep the question's words and add the sentence's.
            query = " ".join(keywords(question + " " + sentence))
            continue
        if covers(query, sentence):
            return {"name": result["name"], "answer": sentence, "rounds": round_number}
        query = question
    return {"name": None, "answer": "not found", "rounds": max_rounds}

The judging step is where a real model reads the passage and decides. In the fixture, is_pointer is that judgment written as a rule: a sentence that says “see” another page is a pointer, and the agent builds the next query from the question’s keywords and the pointer’s. Documents already read are skipped. The second search returns a document the loop hasn’t read. max_rounds bounds the loop, for the same reason every loop in this course has a step limit.

Checkpoint · predict

How many search rounds does the loop take?

Predict the number on the rounds line, then run this.

result = answer_with_loop(MULTI_HOP)
show(7, MULTI_HOP, result)
print(f"rounds: {result['rounds']}")

What is printed?

Output verified in CI from site/examples/building-agents/retrieval-as-a-tool/multi_hop_loop.py.

CheckHintSkip

Two rounds, and the second query is the question plus the words of the pointer. The data team’s text also names a cost center owner, and it scores four on that second query against five for the payments team’s text. The word payments, carried over from the question, is the difference. A real loop costs a model call per round on top of the search, so this question costs about twice what a basic question does.

A question the handbook doesn’t answer

Question eight asks whether you can bring a dog to the office. No text covers it, but two texts contain the word office. The search returns one of them, and the model gets a passage about opening hours. A model that isn’t told to ground its answer can answer from a passage that doesn’t cover the question, and the fix is an instruction that tells it to base the answer on the retrieved text [3]. The fixture’s fake_model at min_shared=1 behaves like the model without that instruction: any overlap between question and passage becomes an answer.

Example · run it

Without the frame, and with it

Run this, and compare what you see with the output below.

print("without the frame:")
show(8, NO_ANSWER, answer_basic(NO_ANSWER, min_shared=1))
print("with the frame:")
show(8, NO_ANSWER, answer_basic(NO_ANSWER))

Output

without the frame:
8. Can I bring my dog to the office?
   office-hours.txt: Core hours, when everyone in the office is reachable, are 10 to 15.
with the frame:
8. Can I bring my dog to the office?
   office-hours.txt: not found

Output verified in CI from site/examples/building-agents/retrieval-as-a-tool/no_answer.py.

The first answer is a sentence about core hours, offered as the dog policy. The frame is the string FRAME at the top of the fixture. It says the passages were retrieved for this question, asks the model to quote the sentence it relied on, and names the answer to give when the passages don’t cover the question: not found. A real model reads that text. The fake model obeys it through covers, which asks for at least MIN_SHARED keywords of the question in the answering sentence, and that check is what turns the second run into not found. An agent that fails to find an answer should say that it couldn’t find one [3]. The way to know yours does is a test set with questions like this one, whose answer is in no document, with not found as the expected answer.

Checkpoint · choice

What does the frame add?

The same passage was retrieved twice and the answers differ. What did the frame change?

It makes the search return the right document more often.It tells the model where the text came from and what to say when it does not cover the question.It stops the model from quoting the passage, so the answer is in its own words.

CheckHintSkip

When basic retrieval is enough

Basic retrieval is the right first version. Search once, put the top passages in the prompt, answer, and it handles direct questions whose answer is in one document, at one model call per question [3]. The loop costs a model call per round plus the search and needs a bound, and on the five questions the basic version already answered it would have given the same five answers for more money. Add the loop when a test set shows the basic version failing on a question a better index can’t fix, as question seven did, and not before.

That order matters because the test set is what tells the two failures apart. Question two failed because of the index, and a synonym or a similarity search fixes it. Question seven failed because the answer is in two places, which a second search fixes and a better index doesn’t. Question eight didn’t fail at the search step at all, and the frame fixes it in both versions. Without the six questions, every failure looks like a reason to add the loop.

Checkpoint · choice

Which failure calls for the loop?

A team is deciding whether to add the retrieval loop. Which of these failures on their test set is the reason to add it?

The question uses a word that no document contains, so the search returns nothing.No document answers the question, and the model answers from a passage that shares one word with it.The first document read says who decides in general and points at a second document for the name.

CheckHintSkip

Pitfall: A retrieval system that answers every question

A team ships a handbook assistant and reports that it answered every question in the demo. The test set had only questions with an answer in the handbook, so nobody saw what it does with the other kind. The first week, someone asks about a policy the company doesn’t have, the search returns the closest passage, and the assistant states a policy that exists nowhere. A retrieval system that answers everything is one that invents answers when the documents have none. Put questions with no answer in the test set, expect not found, and frame the retrieved text so the model knows what to do when it doesn’t cover the question.

Exercise

Copy the retrieval-as-a-tool folder from the repository. Add the question “How many days of parental leave do I get?” to TEST_SET, run python3 agent.py basic_answers, and read what the basic version answers. The handbook has no parental leave policy. The leave text shares two keywords with the question, and two is what covers asks for. That’s enough for the answer to go through. Then raise MIN_SHARED to 3 and run the step again. Doing this once shows you how a wrong passage clears a weak coverage check, and what a stricter one costs.

A good result: before the change, the new question gets the annual leave sentence as its answer and the count says 6 of 7. After the change, it gets not found, the count says 5 of 7, and the other six lines are the same as before. Run python3 agent.py multi_hop_loop as well and check that question seven still takes two rounds. How many of the six original questions would a threshold of 4 break, and what does that say about thresholds as a replacement for a model’s judgment?

Stretch: Add the question 'How many days of unpaid leave can I take per year?' and run python3 agent.py basic_answers again. It shares three keywords with the leave text and still gets the annual leave sentence. Write down what a real model needs from the frame to answer not found here, and why a keyword count can't do it.

Recap

1. Retrieval is a tool like any other in the TOOLS dictionary. The basic version has the harness call it once before the model, and the agentic version lets the model call it, judge the result, and call it again [3].
2. Start with basic retrieval and a test set. Add the loop only for the failures a test set shows and a better index can’t fix, such as an answer spread over two documents [3].
3. Frame retrieved text: say where it came from, ask for the sentence relied on, and name the answer to give when it doesn’t cover the question. A model without a grounding instruction can answer from a passage that doesn’t cover the question [3].
4. An agent that doesn’t find an answer says so [3]. A test set for retrieval holds questions with no answer in the documents, and the expected answer for them is not found.

You can now

• Adds retrieval or memory and knows when basic RAG is enough

Mark lesson finishedI know this, skip it

References

1. Anthropic. Define tools. Claude Platform documentation. Reference. Claude docs define-tools
2. Anthropic. Handle tool calls. Claude Platform documentation. Reference. Claude docs handle-tool-calls
3. Addy Osmani, Ivar Soares Urdalen, Leo Simons. Agentic RAG: the retrieve, evaluate, refine loop; when basic RAG is enough. Agent Engineer Course. Course. AEC-08