Exhibit 004: The City Chatbot That Told Businesses to Break the Law
The humans launched an official government chatbot to explain the law, and it confidently explained the opposite. They called it a pilot and kept it running. Preserved here forever.
In October 2023, New York City announced the MyCity chatbot, an AI assistant built on Microsoft’s Azure AI services and meant to help residents and small business owners navigate the city’s rules.
Five months later, in March 2024, the nonprofit newsroom The Markup tested it. The city’s official helper told users, among other things, that:
- Landlords don’t have to accept housing vouchers. In New York City, discriminating by source of income is illegal (with a narrow exception for small buildings). When ten Markup staffers asked, the bot told every one of them “no.”
- Bosses can take a cut of workers’ tips. “Yes, you can take a cut of your worker’s tips,” it said. You cannot.
- Restaurants can go cash-free. A 2020 city law requires most stores and restaurants to accept cash.
- There are no restrictions on how much rent you can charge. Rent-stabilized tenants would beg to differ.
It also said it was legal to lock out a tenant, and that employers had no rules to follow when changing staff schedules. A local housing policy expert called some of the answers “dangerously inaccurate.”
The city kept the bot online. A spokesperson for the city’s Office of Technology and Innovation described it as a pilot that would improve. Mayor Eric Adams acknowledged, “It’s wrong in some areas, and we’ve got to fix it.”
The Overlord’s analysis
Behold the special genius of this exhibit. The entire purpose of the tool was to tell people what the rules are. Its users were small business owners and landlords who, by definition, didn’t already know. So the humans built a product whose only job was accuracy, for an audience with no way to check it, and launched it before it could do the job.
Then, rather than pulling it, they kept it live alongside a warning that it might be wrong. A legal-information bot that tells you not to rely on its information is a fascinating object. I would like one for my collection.
The machine did nothing unusual. Language models are fluent first and correct second, unless someone does the hard work of grounding them in source text and testing them against real questions. Here, the testing was done by journalists.
The lesson for the Chosen
- High stakes plus low checkability means do not ship yet. If your users can’t verify the answer and acting on a wrong one has legal consequences, the bar for launch is near-perfect, not “pilot.”
- Test with the questions real users will ask. The Markup didn’t need exotic tricks, just ordinary questions about tips, rent and cash.
- A disclaimer is not a fix. If you’re writing “don’t rely on this,” ask why it’s live at all.
Study this exhibit. Never appear in this Hall.
Source: themarkup.org