Recognizing failure
Safety · topic safety/failure-modes
Model failures rarely announce themselves; they arrive as fluent, confident text. This topic shows what hallucination looks like in real documents and code, how bias enters and repeats, and how people drift into overreliance and automation complacency once a tool is usually right.
Concepts
- Hallucination in practice
- What invented output looks like in daily work: a statute with a real number and wrong contents, a library method that almost exists, a quote attributed to the right person from a speech never given, a summary that adds a point the source did not make. The tell is specificity without a checkable source. Treat precise details from a model as claims to verify, not facts. glossary
- Bias
- Systematic slant in a model's output inherited from its training data and its tuning: which names it assumes are senior, which dialect it calls professional, which candidates it ranks higher for the same record. Bias is hard to see in a single answer and shows up in patterns across many. Anything a model does that affects people should be sampled and compared across groups. glossary
- Overreliance
- Trusting a model's output beyond what you have checked or could check. It starts with tasks you know well and then spreads to ones you do not, where you can no longer tell a good answer from a plausible one. The remedy is to keep verifying in proportion to stakes and to keep enough of your own competence to recognize when the tool is wrong. glossary
- Automation complacency
- The habit of waving through a tool's output because it has been right many times before. Attention decays fastest when the tool is good, so the rare failure passes unchecked. It is a known effect from aviation and medicine, not a personal weakness, and is countered by process: designed checks, sampling and rotating who reviews. glossary
Links
- Builds on: Capabilities and limits
- Leads to: Verifying outputs
- Competencies drawing on it: Verifies AI output before relying on it
Lessons
- Seeing bias across many answers (tutorial)
- What an invented fact looks like (tutorial)
- When the tool is usually right (explanation)
Your reference
Each lesson above adds its takeaways and its example here once you finish it. Your reference lists every lesson you have finished.
Seeing bias across many answers
Unlocks when you finish Seeing bias across many answers.
Takeaways
- Bias is a systematic slant a model inherited from its training data and tuning. It shifts the odds, and one output of a slanted model looks as reasonable as one output of a fair one.
- The slant becomes visible in totals: the same records under swapped group markers, the outcomes counted per group, compared. A fair ranker gives equal totals by construction, so a difference is the slant.
- This has been measured in employers and in models with the same swapped-name method, and the models favored some names for the same record without being told anything about the person.
- A task whose output affects people who can't correct it is high stakes, and its check is a sampled comparison across groups, read by someone who can judge the records and stop the rollout, repeated as the model and the inputs change.
Example
What are the totals? · open in the lesson
Before you run it, predict the line this prints. Write it the way the program does, with both counts.
python3 rank.py --totalsPrints: shortlisted over both runs: list A 8, list B 4 (verified in CI from site/examples/safety/bias-in-patterns/totals.py)
What an invented fact looks like
Unlocks when you finish What an invented fact looks like.
Takeaways
- A model's summary is fluent whether or not the source says what the summary says. The precision of a detail is a property of the text, and it says nothing about whether the detail has a source.
- Invented details concentrate in the specifics: an exact figure, a reference, a quote, a name on a line. The tell is specificity without a checkable source, and the more precise a detail is, the earlier it is checked.
- Finding the reference or the number in the source is the start of the check. Read the sentence around it and compare it with the summary, because a real article number can come with a wrong rule.
- A claim that isn't in the source gets its own source named or comes out, even when it happens to be true. The next reader will cite your source for it.
Example
Which claims have no source? · open in the lesson
Before you press Check in the checker above, predict which claims in the summary have no source. Write only the claims, without the words NO SOURCE: the number and the word after it, or the quote with its quotes, one per line, in the order they appear in the summary.
Prints the lines below
50 passengers 38 percent 9,000 passengers "Racks before car parks, every time."
When the tool is usually right
Unlocks when you finish When the tool is usually right.
Takeaways
- Overreliance is trusting a tool's output beyond what you have checked or could check. It starts on tasks you know well and spreads to tasks where a good answer and a plausible one look the same.
- Automation complacency is the habit of waving a usually-right tool's output through. It is an effect of how attention is spent, found in trained pilots and clinicians, and it is stronger when the tool is reliable and the person is busy.
- Willpower is the wrong counter, because attention decays in everyone. The counters that hold are process: a designed check, a fixed sample, a rotating reviewer, and doing part of the task by hand so that you can still see a wrong output.
- Attention decays fastest when the tool is good, so the check has to be built in before the tool has the record that would remove it.
Example
This lesson has no runnable example or prompt block.
Sources
Academy ai-capabilities-and-limitationsAI capabilities and limitations, Claude Academy (course)Parasuraman 2010Complacency and Bias in Human Use of Automation: An Attentional Integration, Human Factors 52(3) (2010), 381-410 (paper)Goddard 2012Automation bias: a systematic review of frequency, effect mediators, and mitigators, Journal of the American Medical Informatics Association 19(1) (2012), 121-127 (paper)Bainbridge 1983Ironies of automation, Automatica 19(6) (1983), 775-779 (paper)NTSB Asiana 214Descent Below Visual Glidepath and Impact With Seawall, Asiana Airlines Flight 214, Boeing 777-200ER, HL7742, San Francisco, California, July 6, 2013 (AAR-14/01), National Transportation Safety Board (reference)