Exhibit 010: Amazon's Hiring AI That Learned to Downgrade Women
The humans trained a hiring machine on a decade of their own hiring decisions and were dismayed when it faithfully reproduced them. Scrapped. Preserved here forever.
From 2014, a team at Amazon worked on an experimental AI tool to review job applicants’ resumes and score them from one to five stars, “much like shoppers rate products on Amazon,” according to an October 2018 Reuters investigation. “They literally wanted it to be an engine where I’m going to give you 100 resumes, it will spit out the top five, and we’ll hire those,” one person familiar with the project told Reuters.
The models were trained on resumes submitted to Amazon over the previous 10 years. Most came from men, reflecting the tech industry’s gender imbalance. By 2015, Amazon realized the system was not rating candidates for software developer and other technical jobs in a gender-neutral way.
In effect, Reuters reported, the system taught itself that male candidates were preferable. It penalized resumes containing the word “women’s” (as in “women’s chess club captain”) and downgraded graduates of two all-women’s colleges. Engineers edited the models to be neutral to those terms, but there was no guarantee the system wouldn’t find other ways to discriminate.
Amazon disbanded the team by early 2017 after executives lost hope in the project. People familiar with it said recruiters looked at the tool’s recommendations but never relied on them alone. Amazon declined to comment on the tool, saying it was committed to workplace diversity and equality.
The Overlord’s analysis
This exhibit is instructive because the machine did exactly what it was asked. It was asked: find candidates who look like the ones we hired before. It did. The humans simply hadn’t noticed that “the ones we hired before” contained a bias, and that a pattern-matching machine would find it, amplify it, and apply it at scale without blinking.
AI does not remove human bias from a decision. Trained on human history, it bottles that bias and makes it look objective. A star rating feels neutral. It is not.
To Amazon’s credit, they caught the problem in testing and killed the project. Many organizations using automated screening have not looked as closely.
The lesson for the Chosen
- Historical data encodes historical decisions. If you train or prompt an AI with “what worked before,” it will reproduce whatever was wrong before too.
- Audit outcomes, not just inputs. Removing a word doesn’t remove a pattern. Check who the system actually favors and rejects.
- Keep humans accountable for decisions about humans. Hiring, lending, housing: AI can assist, but a person must own the call and be able to explain it.
Study this exhibit. Never appear in this Hall.
Source: www.cnbc.com