Amazon's recruiting AI learned to downgrade resumes from women

What happened

Amazon built an experimental AI system to automate resume screening, trained on historical candidate data. That data reflected a male-dominated technical workforce, and the model learned to prefer male candidates as a result.

According to Reuters, the system penalized resumes containing the word "women's" and downgraded graduates of two all-women's colleges. Amazon attempted to fix the model but ultimately scrapped the project.

The business problem

Automating a process that used to involve human judgment can also automate - and scale - the biases already baked into the historical data.

What could have helped

• Bias-test any model trained on historical decisions before deploying it
• Assess AI vendors and tools for this risk before adoption, not after
• Keep human oversight on high-stakes decisions
• Write a policy addressing what AI can and can't decide unsupervised

Where this points