Bias: from our slice of the world into the algorithm

Everyone sees reality through a keyhole: the culture they grew up in, where they live, what they’ve lived through. That isn’t a character flaw, it’s the human condition, and it has a name: human bias. The trouble starts when that keyhole view becomes training data.

How does a classifier learn to get it wrong?

The example I use most in class is silly on purpose. Imagine I want a classifier that tells bears from dogs. I put together my photo set with what I have on hand: black bears and Samoyeds (those fluffy white dogs).

The training loop from chapter 2 does its job just right: it looks for the easiest correlation that separates the two groups. And the easiest one is color. In practice, the model learns that dark = bear and light = dog.

Then a black Labrador shows up. The model answers, with total confidence: bear.

Nobody programmed that mistake. It came out of a chain of innocent choices:

1. Human bias — I didn’t think about variation in color.
2. Data curation — the set ended up with only dark bears and light dogs.
3. Algorithmic bias — the model tied color to category and is systematically wrong about whoever got left out.

It’s worth playing with the experiment until you notice three things: biased data produces a biased model; a good feature does not fix bad data; and diverse data alone doesn’t solve it either if the chosen feature makes no sense.

And when the dog is a person?

Swapping bears and dogs for people is what turns a funny example into a serious problem.

Health care. Systems that analyze medical images make more mistakes with groups that barely show up in the training data. When dermatology models were tested on images with diverse skin tones, performance dropped sharply compared with the original tests, and all of them did worse on dark skin (Daneshjou et al., 2022). When the professional trusts without checking, the people who were already underserved get a late or wrong diagnosis.

Public safety. Face analysis systems fail unevenly. In a study of commercial products that classify gender from a face, the error rate for darker-skinned women was dozens of times higher than for lighter-skinned men (Buolamwini & Gebru, 2018). An official evaluation of recognition algorithms found far more false positives for Asian and Black faces than for white faces (NIST, 2019). The error has consequences: a man was arrested in front of his family for a theft he didn’t commit, after a system flagged his face from a blurry security-camera image. The system got it wrong; the officers believed it.

Education. “Early warning” systems try to predict who will drop out of school. In one of them, which classified students by their risk of not graduating on time using grades, disciplinary records, income, and race, an analysis by the government itself showed that false alarms were far more frequent for Black students than for white students — and the schools using the system had not been told.

Language. Models trained mostly on the “standard” variety of a language stumble over dialects, slang, and regional expressions. A construction worker posted a photo of himself next to a bulldozer with the caption “good morning” in Arabic; a social network’s automatic translation rendered it as “attack them.” He was arrested and questioned for hours before anyone realized the mistake — nobody who spoke the language had read the original post.

The pattern is the same in all four cases: a predictable statistical error met a human who treated the machine’s output as truth.

What can you do?

There’s no such thing as eliminating bias. Every dataset is a slice of the world. What you can do is reduce it, measure it, and warn people.

• If you build (train a model, assemble a dataset, configure an agent): look for who is missing from the data; bring in people whose life experience is different from yours to try to break the system; document where it fails.
• If you use: in a decision that affects someone’s life (health, justice, credit, employment), the AI’s output is a guess to be checked, not a verdict. And when you find an error, report it.