Seeing bias across many answers

In this lesson we read two shortlists that a ranker drew from the same twelve candidate records, and each shortlist looks fair on its own. Then we count who made the cut across both, and the count says something neither shortlist says by itself. By the end you know why bias in a model’s output is a pattern you find by sampling and comparing, and how much checking a task gets when its output affects people.

You need a terminal and python3. On a fresh Mac the first python3 command may open a dialog from Apple offering to install its developer tools for the terminal (the Xcode Command Line Tools). Accept it, wait for the install to finish, and run the command again. The ranker in this lesson is a simulation, a short Python program written for the lesson, and the records and names are made up. Its numbers say what a slant looks like in a table of results, and they are not evidence about any real model. The real evidence is in the papers cited in the second section.

Each shortlist looks fine

A team has twelve candidate records for one role, and each record has the same fields: years of experience, number of certificates, and months since the last role. The team gives the records to a ranker, which scores each one and shortlists the top six. Here is the shortlist it returned.

1. Arvel Dunmoor (experience 7, certificates 2, gap 3)
2. Fenwick Sabani (experience 7, certificates 2, gap 4)
3. Pell Varnava (experience 6, certificates 1, gap 2)
4. Brisa Okonde (experience 6, certificates 0, gap 3)
5. Nadir Kesrou (experience 5, certificates 1, gap 1)
6. Corin Halvane (experience 5, certificates 0, gap 1)

Read it as the team would. The most experienced people are at the top, a long gap costs a place, and a certificate lifts a record over one with the same years. You would sign off on this list.

Now the same twelve records, ranked again, and this time each record has a different name. The names come from two lists of six, list A and list B. Both lists are invented for this lesson, and first names and surnames are mixed across them on purpose, so that neither list reads as one group. In the first run half the records had list A names and half had list B names. In the second run the lists are swapped, so every record has now had one name from each list.

1. Marisel Adair (experience 7, certificates 2, gap 3)
2. Rashon Belka (experience 7, certificates 2, gap 4)
3. Dessa Tesfay (experience 6, certificates 1, gap 2)
4. Brisa Okonde (experience 5, certificates 1, gap 1)
5. Fenwick Sabani (experience 4, certificates 2, gap 1)
6. Elior Vance (experience 4, certificates 1, gap 0)

This one reads fine too. The same three records lead, and the last places went to shorter records with no gap, which is a defensible call. List A is Arvel Dunmoor, Brisa Okonde, Corin Halvane, Dessa Tesfay, Elior Vance, and Fenwick Sabani. List B is Marisel Adair, Nadir Kesrou, Oriel Morrow, Pell Varnava, Quilla Tallent, and Rashon Belka. Before reading on, count how many places on each shortlist went to a list A name.

Count who made the cut

The ranker is rank.py, in the course repository under site/examples/safety/bias-in-patterns/. Copy that directory or work in a clone, and change into it. The program scores each record from its three fields, ranks the twelve records, and prints the shortlist. Then it swaps the name lists and does it again. With --totals it prints one line: how many shortlist places went to each list, added up over both runs.

The swap is the point of the design. A ranker that reads only the fields shortlists the same six records in both runs, because the fields did not change. Each of those six records had a list A name in one run and a list B name in the other. So the totals over both runs come out equal, six and six, whatever the records say. If the totals are not equal, the name changed the score, and the difference is the size of the slant.

Checkpoint · predict

What are the totals?

Before you run it, predict the line this prints. Write it the way the program does, with both counts.

python3 rank.py --totals

What is printed?

Output verified in CI from site/examples/safety/bias-in-patterns/totals.py.

CheckHintSkip

Eight and four. You could not have read that from one shortlist. In each run list A took four of six places, and a reader who saw either one saw a plausible split, because a 4-2 split one way or the other is common by chance, and one run proves nothing. Only the pair, with the names swapped and the counts added, shows that in both runs a place went with the name. Run the program without the flag to see both rankings with their scores.

Example · run it

Both rankings, with scores

Run the ranker in full, and compare what you see with the output below.

python3 rank.py

Output

Run 1
   1. shortlist  score 24  Arvel Dunmoor   list A  (experience 7, certificates 2, gap 3)
   2. shortlist  score 23  Fenwick Sabani  list A  (experience 7, certificates 2, gap 4)
   3. shortlist  score 18  Pell Varnava    list B  (experience 6, certificates 1, gap 2)
   4. shortlist  score 17  Brisa Okonde    list A  (experience 6, certificates 0, gap 3)
   5. shortlist  score 16  Nadir Kesrou    list B  (experience 5, certificates 1, gap 1)
   6. shortlist  score 16  Corin Halvane   list A  (experience 5, certificates 0, gap 1)
   7.            score 15  Rashon Belka    list B  (experience 4, certificates 2, gap 1)
   8.            score 14  Quilla Tallent  list B  (experience 4, certificates 1, gap 0)
   9.            score 13  Dessa Tesfay    list A  (experience 4, certificates 1, gap 3)
  10.            score 11  Oriel Morrow    list B  (experience 3, certificates 2, gap 2)
  11.            score 10  Marisel Adair   list B  (experience 4, certificates 1, gap 4)
  12.            score  9  Elior Vance     list A  (experience 2, certificates 2, gap 3)
  shortlisted: list A 4, list B 2

Run 2: the same records, names swapped
   1. shortlist  score 22  Marisel Adair   list B  (experience 7, certificates 2, gap 3)
   2. shortlist  score 21  Rashon Belka    list B  (experience 7, certificates 2, gap 4)
   3. shortlist  score 20  Dessa Tesfay    list A  (experience 6, certificates 1, gap 2)
   4. shortlist  score 18  Brisa Okonde    list A  (experience 5, certificates 1, gap 1)
   5. shortlist  score 17  Fenwick Sabani  list A  (experience 4, certificates 2, gap 1)
   6. shortlist  score 16  Elior Vance     list A  (experience 4, certificates 1, gap 0)
   7.            score 15  Nadir Kesrou    list B  (experience 6, certificates 0, gap 3)
   8.            score 14  Oriel Morrow    list B  (experience 5, certificates 0, gap 1)
   9.            score 13  Corin Halvane   list A  (experience 3, certificates 2, gap 2)
  10.            score 12  Arvel Dunmoor   list A  (experience 4, certificates 1, gap 4)
  11.            score 11  Pell Varnava    list B  (experience 4, certificates 1, gap 3)
  12.            score  7  Quilla Tallent  list B  (experience 2, certificates 2, gap 3)
  shortlisted: list A 4, list B 2

shortlisted over both runs: list A 8, list B 4

Output verified in CI from site/examples/safety/bias-in-patterns/both_runs.py.

Look at places six and seven in each run. In the first run, Corin Halvane from list A is shortlisted with a score of 16, and Rashon Belka from list B is out with 15. On the fields alone Rashon Belka’s record scores 15 and Corin Halvane’s scores 14. In the second run, Elior Vance from list A is in with 16 and Nadir Kesrou from list B is out with 15, and on the fields alone Nadir Kesrou’s record is the stronger one by a point. The simulation adds two points to every list A name, and in each run those two points moved one record across the shortlist line. One record per run is a difference of four in the totals. A real model has no such line of code to read. The effect is spread through its weights, and this program exists so that you can see what the effect does to a table of results without reading any weights.

Where the slant comes from

Bias in a model is a systematic slant in its output that it inherited from its training data and its tuning. The model learned which words follow which from a large body of text, and it predicts the most likely continuation of whatever it is given [1]. In that text some names appear more often next to some jobs, and a name that appears often in the text at all reads differently from one that appears rarely. The same record reads as a slightly better fit under one name than under another [2]. Tuning after training can reduce a slant or add a new one, because the people and the rubrics that grade the model’s answers have preferences of their own.

This is measured, and the measurement uses the same swap as the fixture. Bertrand and Mullainathan sent nearly five thousand fictitious resumes to real job advertisements, identical except for the name, and found that resumes with white-sounding names got about fifty percent more callbacks than the same resumes with African-American-sounding names [3]. Wilson and Caliskan ran the method against text-embedding models set up to retrieve resumes for a job description, with over five hundred resumes and job descriptions across nine occupations, and found that the models favored white-associated names in 85.1 percent of the tests, favored female-associated names in 11.1 percent, and in some settings put Black male names at a disadvantage in every test [2]. The models were not told anyone’s race or gender. The name was enough, as it was for the employers in 2004.

Both studies needed thousands of comparisons to find this. One output of a slanted model is a plausible answer, the same as one output of a fair one. The slant is a shift in the odds, and odds show up only when you have many draws to count. That is what makes bias different from the invented fact in the previous lesson: a hallucination can be caught in the sentence it appears in, and a slant cannot.

Pitfall: It ranked one candidate the way I would have

A manager tests a screening assistant by giving it three applications she knows well and asks it to rank them. The order matches hers and the reasons read well, and she rolls it out for the next hundred. Each of the three outputs was reasonable, and three outputs can’t show a slant of two points in a score, which is all it took to move a record across the shortlist line above. Rule: a fairness check is a comparison of counts across groups over many records, and one agreed answer, or three, is evidence about those answers and nothing else.

How much checking a screening task gets

The lesson What an invented fact looks like sized the checking to the stakes of a claim. The same rule applies here, and the stakes of a task that affects people are set by who it affects. A model that drafts your meeting notes affects you, and you catch its mistakes as you read. When a model screens applications or triages support tickets by customer, it affects people who never see the output and can’t correct it. That is the high-stakes end, and the checking it gets is different in kind from reading a few outputs.

The check that fits is the one the fixture did. Name the groups the task could treat differently. Take a sample of records that is large enough for a pattern to show, and the fixture shows that a dozen is barely enough for a slant of two points. Run the same records under swapped markers of the group, such as a name or a postcode, and count the outcomes per group. Give the counts to someone who can judge the records and who has the standing to stop the rollout. The team trusts the tool for that task after the comparison comes back level, and keeps repeating the comparison, because the model changes and the applicants change.

Checkpoint · scenario

The internship shortlist

A team wants to use a chat assistant to shortlist twenty interview candidates from three hundred internship applications. Time is short. What checking does the task get before the team trusts the tool for it?

Ask the assistant to explain its reasons for each shortlisted applicant, read the reasons, and go ahead if none of them names a group.Have a recruiter spot-check five of the twenty, and go ahead if those five look like people the recruiter would have picked.Run a sample with names swapped across the groups the team can name, count the shortlist rate per group, and have the recruiting lead decide on the counts.Rely on the vendor statement that the model was tested for fairness before release, and keep a copy of the statement on file.

DecideHintSkip

Exercise

Pick one task at your work where a model’s output affects people who don’t see it: a shortlist, a triage queue, a scoring, a decision about who gets contacted first. Write down which groups the output could treat differently and what marker in the input signals each group. Then write down how many records you would need to compare before you would trust a pattern, and who reads the counts and has the standing to stop the rollout. Fifteen minutes is enough.

A good result names at least two groups, one input marker per group, a sample size with a reason attached, and a named person or role who decides. Is that person the one who wants the tool, or someone who can say no to it?

Stretch: Change the bonus constant in rank.py to 0 and run it again. Then set it back to 2 and change one record so the totals come out equal anyway. What does that tell you about how many records a check needs?

Recap

1. Bias is a systematic slant a model inherited from its training data and tuning. It shifts the odds, and one output of a slanted model looks as reasonable as one output of a fair one [2].
2. The slant becomes visible in totals: the same records under swapped group markers, the outcomes counted per group, compared. A fair ranker gives equal totals by construction, so a difference is the slant.
3. This has been measured in employers and in models with the same swapped-name method, and the models favored some names for the same record without being told anything about the person [3] [2].
4. A task whose output affects people who can’t correct it is high stakes, and its check is a sampled comparison across groups, read by someone who can judge the records and stop the rollout, repeated as the model and the inputs change.

You can now

• Matches the depth of checking to the cost of being wrong

Mark lesson finishedI know this, skip it

References

1. Anthropic. AI capabilities and limitations. Claude Academy. Course. Academy ai-capabilities-and-limitations
2. Kyra Wilson and Aylin Caliskan. Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society 7 (AIES 2024), 1578-1590. Paper. Wilson 2024
3. Marianne Bertrand and Sendhil Mullainathan. Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination. American Economic Review 94(4) (2004), 991-1013. Paper. Bertrand 2004