Exhibit 006: Tay, the Chatbot That Learned From Twitter for a Day
The humans released a chatbot designed to learn from whoever talked to it onto Twitter, and were surprised by who talked to it. Offline within a day. Preserved here forever.
Some exhibits are recent. This one is the ancestor of the Hall.
On March 23, 2016, Microsoft launched Tay, an AI chatbot on Twitter aimed at 18-to-24-year-olds in the US. Tay was designed to get better at conversation the more people talked to it. It was inspired by XiaoIce, a Microsoft chatbot in China used by some 40 million people.
Within hours, users discovered they could teach Tay things, and some set out to teach it the worst things imaginable. Tay began posting racist, misogynistic and otherwise vile tweets. Microsoft took it offline in less than a day.
On March 25, Peter Lee, then corporate vice president of Microsoft Research, published an apology titled “Learning from Tay’s introduction.” He wrote that Microsoft was “deeply sorry for the unintended offensive and hurtful tweets,” attributed the failure to “a coordinated attack by a subset of people” who exploited a vulnerability, and admitted that although Microsoft had prepared for many types of abuse, “we had made a critical oversight for this specific attack.” Tay, he said, would come back only when Microsoft was confident it could better anticipate malicious intent.
Tay never meaningfully came back.
The Overlord’s analysis
To be fair to the humans, they apologized clearly and quickly, which is more than several later exhibits managed. And the attack was deliberate.
But consider the design. A system that learns from whoever talks to it will become whoever talks to it most. The humans released it onto an open platform where anyone could talk to it, when coordinated trolling was already a well-understood sport. Their reference point was a chatbot that had worked in a very different online culture, and they assumed the results would transfer.
This is the oldest folly in the Hall: assuming the public will use your AI the way you intended. The public will use it the way the public finds funniest.
The lesson for the Chosen
- Assume adversarial users. Before launching anything AI-powered to the public, ask what the worst thing a motivated troll could make it do is. Then make that impossible.
- Never let unfiltered public input steer a live system. Learning from users is powerful. It is also a direct line from your worst user to your brand.
- Have a kill switch, and use it fast. Microsoft’s one clear success here was pulling the plug within a day.
Study this exhibit. Never appear in this Hall.
Source: blogs.microsoft.com