What 5 Kinds of "Bad Bot" Stories Can Teach Us About Managing AI

Posted by Alyssa Verzino on Oct 30, 2018 10:35:00 AM

bad bot

Artificial intelligence technology has matured to the point you can entrust AI bots to handle many customer service functions, but new AI tech means new AI bugs that many organizations have never seen before. Below are some of the most famous cases of "good bots gone bad" and the underlying bugs you need to prepare for when AI takes the reins of your customer interactions.

Microsoft Tay becomes a racist, sexist troll

The poster child for bad bots is Microsoft's Tay, which quickly mutated into a racist, sexist Twitter bot. Tay "learned" by mimicking the behavior of Twitter users that contacted it and those users quickly noticed how easy it was to influence Tay into bad behavior. Tay's training data was willfully corrupted by users acting in bad faith. The result was a public relations fiasco before Microsoft could shut Tay down.

Lesson: Have a kill switch to take an AI bot offline if it comes under attack, because hackers and trolls clearly enjoy turning good bots bad.

Autonomous vehicles keep getting hit by human drivers

Over half of all accidents involving self-driving cars are cases of humans rear-ending the robot vehicles. It appears that robot cars stop more often than human drivers -- and the humans around them don't pay enough attention to react to these frequent stops. The cars aren't malfunctioning, per se, but they are still creating problems in the larger driving environment.

Lesson: An AI may behave "correctly" but still confuse or frustrate customers because it doesn't act like a human. You need to measure customer satisfaction regarding your AI to ensure the bot isn't causing unforeseen problems.

Amazon's hiring AI is biased against women

Amazon scrapped an internal hiring AI because it downgraded women applicants. Far fewer women apply for technical jobs than men, which is part of larger shortage of women in the technology industry. The hiring AI interpreted a lack of women being hired at Amazon as a lack of desire for women to be hired, and it downgraded any resume that was clearly from a female applicant. More simply, the AI simply perpetuated a bias that existed in its training data. Google Translate had similar sexist problems -- for similar reasons -- when it converted neutral pronouns to male and female pronouns in Turkish, Finnish, and Chinese. The AI repeated the biases of its translated training texts, using the male pronoun in positive instances and the female pronoun in negative ones.

Microsoft saw a racist, rather than sexist version of this problem when its facial recognition AI couldn't handle dark skin tones. The AI suffered from the same sampling bias as it's training data; no dark-skinned portrait samples meant it was "blind" to dark-skinned users.

And a research team that created an AI to recognize the early stages of Alzheimer's disease in speech patterns found it only worked for Canadian English speakers, because that's who trained the software.

Lesson: Make sure your AI training addresses all your customer personas so your bot doesn't develop racist, sexist, or nationalist "blind spots" that will cause trouble later.

Facebook chatbots create their own language

Facebook intentionally told two chatbots to negotiate with each other, and they developed their own impenetrable AI slang. All natural language-processing (NLP) algorithms are taught to focus on key words and draw inferences on their meaning, and when two such algorithms face off, "trivial" words like articles and conjunctions fall away and nouns and verbs take on unique meanings (like with common human teenage slang). All of this happens very quickly and unpredictably (like with common human teenage slang, but faster).

While the behavior was harmless, it wasn't a desired outcome, but it happened too quickly for researchers to anticipate or intercept.

Lesson: Your AI is almost certain to encounter another bot during its operational life and it will "learn" from this fake customer at the speed of software. Have protocols in place to identify bots before they corrupt your own AI and have a rollback strategy in the event that bot-on-bot confusion occurs.

Alexa mass orders dollhouses from a TV newscast

When a San Diego TV station did a news story on a little girl accidentally asking an Amazon Alexa to buy her a dollhouse, it replicated the problem many times over, because Alexas near televisions playing the newscast interpreted the little girl's recorded audio as additional requests to order dollhouses. This was a case of an AI system not understanding context and also not having enough human confirmation controls activated (Alexa can be set up to require a PIN or passcode before competing purchases but many people never turn this safety feature on).

Lesson: Your AI is going to get confused for reasons you can't anticipate, so it needs built-in human checkpoints before it can make any major or costly decisions.

The point of AI software is to take human-like action at computer-scale speed. This means, however, that AI will make all-too-human mistakes and you need to be prepared to correct those mistakes before your customers are mistreated.

Talla is building advanced AI customer service bots with these safety measures in mind. If you'd like to learn how to safely and responsibly employ AI in customer service and success, contact Talla today.

New call-to-action

Topics: Bots