Finding information with an assistant
In this lesson we use a chat assistant to find our way around a text we have never read. We paste it and ask the question a newcomer would ask, first as a plain request and then as a brief that names the source and asks for a quote behind each claim. Then we ask the same question with no text at all and watch what comes back. By the end you have a rule for finding information with an assistant that you can apply to any document someone hands you.
You need any chat assistant you already have access to. Every response shown here is illustrative. The author wrote it to show the pattern, and your model gives a different text.
The text
Section titled “The text”Here is the material. It is a fictional guideline from a fictional company, kept short on purpose so that you can check every answer against it. Read it once now, so that you can judge the answers later.
Brindlewick Instruments: expenses and travel guideline
This guideline applies to everyone who spends their own money on behalf of Brindlewick Instruments and claims it back. It replaces the 2024 version.
Claiming. Submit a claim through the expense form on the intranet within 14 days of the date on the receipt. Each claim needs a receipt for every amount above 10 euros. For smaller amounts a one-line note of what was bought and why is enough. Claims older than 90 days are not paid. Approved claims are paid with the next salary run. A claim approved after the 20th of the month is paid with the salary of the month after.
Company cards. Team leads hold a company card for team costs. Everyone else pays first and claims afterwards. Do not ask a team lead to put a personal travel cost on their card.
Meals when traveling. Brindlewick doesn’t pay a daily allowance. Meals on a business trip are reimbursed against receipts up to 35 euros per day. Alcohol is not reimbursed. Meals with a client are booked as client entertainment and need the client’s company name on the claim.
Travel. Take the train for any trip under four hours door to door. Flights need written approval from your manager before booking, and only economy fares are reimbursed. Book through the travel desk where you can, so that the cost never touches your own account. Taxis are reimbursed when there is no public transport option at that hour, and the claim says so.
Own car. Use your own car only when no train option exists for the trip. Mileage is reimbursed at 23 cents per kilometer, plus parking against receipts. Fines and damage are never reimbursed.
Home office. In your first three months you may claim up to 250 euros once for a desk or chair for working from home, with the receipt attached to the claim. Monitors, keyboards and headsets come from the IT desk, so do not buy your own.
Questions. The finance team answers expense questions in their office hours on Tuesday and Thursday mornings. For anything this guideline does not cover, ask before you spend.
The question we work with is the one a new hire asks in their first week: “What do I need to know about expenses before my first trip?” You don’t know this text, so you can’t judge an answer to it by feel. That’s the normal situation when you ask an assistant to find information. The point of this lesson is to ask in a way that lets you check the answer anyway.
Ask for a summary
Section titled “Ask for a summary”Before you read on, predict what you get back when you paste the guideline and type one line above it. How long is the answer, and how do you tell whether it is right?
Summarize this.
[the guideline, pasted in full]
This guideline explains how employees of Brindlewick Instruments claim back expenses. Claims go through the intranet form within 14 days and need receipts, and claims older than 90 days are not paid. Payment comes with the next salary run. Only team leads have company cards. Meals on business trips are reimbursed against receipts up to a daily limit, and alcohol is excluded. Train is the default for shorter trips, flights require manager approval and must be economy, and taxis are only allowed when there is no public transport. Mileage in your own car is reimbursed at a fixed rate when no train is available. New hires can claim a one-off amount for a home office desk or chair within their first three months. Questions go to the finance team during their office hours on Tuesdays and Thursdays.
You probably got something like this. It is a fair paragraph. “Need receipts” drops the 10-euro line, and “a daily limit” and “a fixed rate” leave out the numbers, which are what a newcomer asks about next. The bigger problem is how you found that out. You had to read the whole guideline again and hold it against the paragraph. The summary saved you nothing.
That result is the model’s default when the prompt leaves everything unsaid, as you saw when you rebuilt a one-line request into a prompt. “Summarize this” doesn’t say who is asking, what they need, or how you plan to check the answer. The model produces the average summary and leaves the checking to you.
Ask for the three things and where they come from
Section titled “Ask for the three things and where they come from”Now write the same question as a brief. The goal is the newcomer’s question. The context is the guideline, pasted below a marked boundary, and the brief says that the guideline is the only source. The format is three items, each with the sentence from the text it comes from, quoted word for word. The last line tells the model what to do when the text has no answer. A missing answer is the first place a guess appears when you leave that line out.
I am new at Brindlewick Instruments and about to make my first business trip. Using only the guideline below the line, give me the three things I must know about expenses before I travel. Below each one, put the sentence from the guideline it comes from, quoted word for word. Where the guideline has no answer, tell me that instead of guessing.
[the guideline, pasted in full]
- Pay first, claim within two weeks, with receipts. “Submit a claim through the expense form on the intranet within 14 days of the date on the receipt.”
- Take the train unless the trip is long, and get flights approved first. “Flights need written approval from your manager before booking, and only economy fares are reimbursed.”
- Meals are covered against receipts, up to a limit, and no alcohol. “Meals on a business trip are reimbursed against receipts up to 35 euros per day, and alcohol is not reimbursed.”
Compare this with the paragraph. Each claim now comes with the place in the text it rests on, and checking it is a search for one sentence. Try it on your own result: search the guideline for each quoted sentence.
Do the same on the illustrative answer and the third quote fails. The guideline has “Meals on a business trip are reimbursed against receipts up to 35 euros per day” and then, as a separate sentence, “Alcohol is not reimbursed”. The model joined two sentences into one and presented the result as a quote. The fact is right, and the quote isn’t in the text. Without the brief you would never have seen the join, and next time the join could be between a rule and something the model added. Asking for the source of each claim makes the answer checkable, and the check takes a minute.
Which ask can you check?
Section titled “Which ask can you check?”The lesson pastes a fictional company expense guideline into a chat assistant and asks what a new hire must know before a first business trip.
You are about to paste the expense guideline into an assistant. Which ask gives you an answer you can check against the text in a minute?
For each ask, how would you tell a right answer from a confident wrong one without reading the whole guideline again?
Fix the ask
Section titled “Fix the ask”A colleague sends this to their assistant, gets a confident list of six rules, and later finds that two of them are not in the policy. Rewrite the ask so that every rule in the answer can be checked against the policy.
Using only the attached travel policy, list the rules for booking a flight. Quote the sentence from the policy that each rule comes from, word for word. If the policy does not say something, say that it does not, and do not fill it in from general knowledge.
How would the colleague have caught the two invented rules before relying on them?
The same question from memory
Section titled “The same question from memory”Now ask the newcomer’s question without the guideline. Open a fresh chat and type only the question, with a specific detail in it that you already know the answer to.
I am starting a new job and about to make my first business trip. What is the deadline for submitting expense claims, and how much can I spend on meals per day?
Policies vary between companies, but the common rules are these. Expense claims are usually due within 30 days of the expense, and many companies allow up to 60 days. For meals, most companies set a per diem of around 50 to 75 euros per day for domestic travel, and you normally do not need to submit receipts for meals under the per diem. Check your employee handbook to confirm the exact figures.
Read that answer as the newcomer would. It has a hedge at the start and a hedge at the end, and between them it gives four specific numbers and a rule about receipts. Every one of them is wrong for Brindlewick, where the deadline is 14 days, there is no per diem, the meal limit is 35 euros, and a meal above 10 euros needs a receipt. The prose is as confident as the answer that came with quotes. In the lesson on how models work you saw why. The specifics are where a model has the fewest examples behind its prediction, so it fills the gap with something of the right form. The model has never seen the guideline, and a general answer about companies is the most likely text for a question that names none.
A confident answer with no source
Section titled “A confident answer with no source”A colleague asked a chat assistant a factual question about the company's expense rules without giving it the company's guideline, and got a confident specific answer.
A colleague asks an assistant “how many days do I have to submit an expense claim?” without pasting anything, and gets “30 days”. What do you tell them to do before they rely on it?
What did the assistant have in front of it when it answered, and what would change that?
Which lines make it checkable?
Section titled “Which lines make it checkable?”A learner pastes a company's travel guideline into a chat assistant and asks what a new hire must know before booking a train.
Which two lines in the ask give you an answer you can check against the guideline in a minute?
Which lines tie the answer to the pasted text, so that you can check it there?
One quote isn't in the text
Section titled “One quote isn't in the text”A learner asked an assistant to answer from a pasted expense guideline and to quote the sentence behind each point. One quoted sentence isn't in the guideline, though the point it supports is right.
One of the quotes in the answer isn’t in the guideline, and the point it supports is right. What do you do?
What does a quote that isn't in the text tell you about how the answer was made?
The rule
Section titled “The rule”Finding information with an assistant is fast when you give it the material and slow when you don’t, because without the material the time moves to checking. The brief you wrote in this lesson has one line for each part of that: the source and that it is the only one, the sentence behind each claim, and what to do when the text is silent. With the quotes in hand, checking what you will repeat takes a minute.
The same brief works for a contract, a set of meeting notes, a long email thread or a product manual. When the assistant searches the web, the quote becomes a link, and the check is opening it. Claude, for example, adds a citation to each answer it builds from a web search [1]. With pasted text or with a link, before you repeat an answer you should be able to name where it came from.
Exercise
Open a fresh chat and paste the Brindlewick guideline from this page. Send the brief from the second section, asking for the three things a newcomer must know with a quote for each. Then search the guideline for every quote and write down, for each of the three, whether it is there word for word, there in different words, or not there at all. This takes about five minutes and leaves you with a three-line table. Doing it once by hand is what turns the check into a habit. A good result is a table where every row has a verdict you could show to someone else. What would you have done with the answer if you hadn’t asked for the quotes?
Stretch: Ask a fourth question that the guideline does not answer, such as whether a hotel breakfast counts as a meal, and see whether the assistant says so or fills it in.
Recap
- An assistant is fast at orientation in a text you supply, and unreliable on specifics from memory alone.
- “Summarize this” gives you a paragraph you can only check by reading the source again.
- A brief for finding information gives the assistant the material, marks it as the only source, and asks for the sentence behind each claim and for a plain “the text doesn’t say” where there is none.
- Check the quotes. A quote that is not in the text, even when the fact is right, shows where the model joined or added something.
- A confident detail from memory is the most likely text for the question, and the model didn’t look it up. Confidence in the prose says nothing about the fact [2].
You can now
- Writes a brief with goal, context, limits and done-criteria
References
Section titled “References”- Anthropic. Enable and use web search. Claude Help Center. Reference.
Claude support enabling-and-using-web-search - Anthropic. AI capabilities and limitations. Claude Academy. Course.
Academy ai-capabilities-and-limitations