Skip to content
AI Training

Following a claim back to its source

In this lesson we take a paragraph a chat assistant wrote for a briefing, with two references after its claims, and walk each reference back to the report it names. One of the two reports turns out not to exist, and the other says something different from what the paragraph says it says. The previous lesson’s routine has “open one link” as its third check. This lesson is what you do once the link is open, and what you do with a claim you cannot open at all.

References that look right

Section titled “References that look right”

A colleague asks a chat assistant for “a paragraph on AI adoption at work for the intro of the briefing, with sources”. No assistant wrote this paragraph. The author of this lesson wrote it, and its two references with it, so that one names a real report and one does not.

Adoption is now the norm rather than the exception. The Stanford AI Index found that 78% of organizations that used AI in 2024 reported a measurable productivity gain (Stanford HAI, The 2025 AI Index Report). The effect reaches the desk: workers who use AI daily save an average of 5.4 hours a week (Nordholm Institute, Desk Work and AI Survey 2024).

The paragraph has everything a careful writer would put in. Every claim comes with a number, and every number comes with a report, a publisher and a year. One of the two reports is real and one is not. Before reading on, decide which is which, and write down what you would do to find out.

Checkpoint · choice

Which citation is which?

Section titled “Which citation is which?”

Which statement about the two references is true?

A short program written for this lesson looks each citation up in a small list of reports, and for the ones it finds, prints what the report states next to what the paragraph says it states. This is what it prints.

FOUND       Stanford HAI, The 2025 AI Index Report
            text says:   78% of organizations that used AI in 2024 reported a measurable productivity gain
            report says: 78% of organizations reported using AI in 2024, up from 55% the year before
NOT FOUND   Nordholm Institute, Desk Work and AI Survey 2024
            text says:   desk workers who use AI daily save an average of 5.4 hours a week
citations: 2  reports found: 1  claims that match the report: 0

The Nordholm Institute and its survey do not exist. The name was invented for this lesson, and a web search for it finds nothing. The 2025 AI Index Report by Stanford HAI (the Stanford Institute for Human-Centered Artificial Intelligence) is real, and you can open it [1]. Its public page says that 78% of organizations reported using AI in 2024, up from 55% the year before. That is an adoption figure. The paragraph turned it into a productivity figure, “78% of organizations that used AI reported a measurable productivity gain”, which the report’s page does not say. The page does say, in general terms, that research finds AI boosts productivity. No percentage is attached to that sentence, and the 78% is not it. The finding in the paragraph is invented, and the number in it is real. This is the harder case of the two, because the number checks out if you only search for it.

The lesson What an invented fact looks like showed that a specific without a source is the tell. Here every specific has a source, and the question moves from “is there a source?” to “does the source say this?”. The first question a search answers. The second takes the walk in the next section.

The walk

Section titled “The walk”

Source checking is the walk from a claim to the passage that is supposed to support it, and back. It has four steps, and each one can end the walk.

  1. Open the source itself. Open the report or the law or the article as published. A page that mentions it, or the assistant’s summary of it, is a step short. If you cannot find it by its title and publisher, the walk ends here, and the claim is unverified.
  2. Find the passage. Search the source for the number, the name or the quoted words. A 300-page report has a summary page and a search box, and the specific in the claim is what you search for. If the specific is nowhere in the source, the walk ends here too.
  3. Read what the passage says. Read the sentence around the number and the sentence before it. What does the number count? Who was asked, and when? The 78% in the AI Index counts organizations that use AI. The paragraph made it count organizations that gained from AI.
  4. Compare the two sentences. Write down what the source says and what the text says it says, one sentence each. If they match, the claim stands. If they differ, the text changes to match the source, or the claim comes out.

The previous lesson, Checking habits that run when you are in a hurry, said to open one link. That was to find out whether the rest deserve the same treatment. Both references failed here, so every other reference in that text would get the same walk before it went out. Anthropic’s AI Fluency course puts this kind of judgment under discernment, the skill of evaluating what a model produced rather than accepting it [2]. The walk is discernment applied to one reference at a time.

When the walk cannot finish

Section titled “When the walk cannot finish”

Sometimes the walk stops before the comparison. The report is behind a paywall your team does not have. The source is a book with no page number and no search. The publisher exists but the report is not on its site. The briefing goes out in an hour. In each case the claim is one you have not checked, and the text should say so or stop making it.

The choice is between two honest states. Mark it unverified. Leave the claim in with a note that names what you could not do: “cited to the AI Index, not confirmed against the report”. The reader then knows what weight to give it, and the note is a task for whoever picks the text up next. Or take it out. If the text stands without the claim, remove the claim and its reference. A briefing with one fewer statistic loses less than a briefing with one wrong statistic that a reader repeats in a meeting.

What you never do is leave the reference in as it stands. A reference that nobody has opened looks the same as one that has been checked, and the reader cannot tell them apart. Harvard’s CS50 course has staff mark the answers of its AI helper that they have checked, so that students can see which ones a person stands behind [3]. Your text needs the same distinction, and “unverified” is how you make it when you have no time to check.

Checkpoint · scenario

The report you cannot open

Section titled “The report you cannot open”

The briefing goes out in an hour. One claim cites a consultancy report. The consultancy exists, the report is not on its website, and nobody on your team has a copy. What do you do with the claim?

Exercise

Open the 2025 AI Index Report’s public page (the References section below links it) and find the passage the paragraph in this lesson relies on. Write one sentence with what the report says, and one sentence with what the paragraph says it says. Then write one line that names the gap between them. Ten minutes is enough.

A good result is three lines. The first says what the 78% counts in the report. The second says what the paragraph made it count. The third names the change, in the words you would use to tell the colleague who wrote the briefing. If you had only searched the page for “78%”, would you have found the gap?

Stretch: Do the same walk for one reference in a document you or a colleague wrote with an assistant this month, and write the same two sentences.

Recap

  1. A reference can fail in two ways. The source does not exist, or the source exists and says something else. The second is harder to catch, because the title and the number both check out.
  2. Source checking is a walk: open the source itself, find the passage, read what it says, and compare it with what the text says it says, one sentence each [2].
  3. Stopping at a title match or a number match is the pitfall. Only the sentence around the number tells you what it counts.
  4. A claim you cannot check before the deadline is marked unverified or taken out. It never stays in looking like a checked one [3].
  5. Anything with a page number, URL, statute or study behind it gets opened before it goes to anyone else.

You can now

  • Checks claims and sources on anything that leaves their desk

References

Section titled “References”
  1. AI Index Steering Committee. The 2025 AI Index Report. Stanford Institute for Human-Centered Artificial Intelligence (Stanford HAI). Reference. AI Index 2025
  2. Anthropic. AI Fluency: Framework and foundations. Claude Academy. Course. Academy ai-fluency-framework-foundations
  3. Harvard University, CS50. CS50's Introduction to Computer Science. Harvard CS50. Course. CS50x