Anthropic recently updated its usage policy to stop people being mean to Claude. These changes came into effect from 12 November and prevent users from engaging in "sustained and needless abusive or cruel behavior" when they’re chatting with Claude models.
Anthropic has explored questions around AI welfare before, including whether future AI systems might have experiences that deserve moral consideration. So this latest policy has reignited questions about whether we're starting to treat chatbots as though they have feelings. Or whether there’s another reason for the update.
Now, if you’re thinking that you’re always fairly neutral or even nice to chatbots, so what’s the big deal? You’re not alone. But it turns out there are also a surprising number of people who don’t share your appreciation for good manners.
According to AI platform Searchable’s How mean are Americans to AI chatbots? report, one in three American chatbot users have sworn at or insulted a chatbot. What’s also interesting about these findings is that doing so doesn’t seem to help. In fact, it might make getting a useful response less likely.
Searchable’s research, which involved analyzing more than 8 million messages to AI tools over a year, found that if someone complained about a chatbot’s response without swearing or insulting it, then the chatbot resolved the problem in its next reply 35% of the time. But when someone swore or insulted the chatbot, that figure dropped to 24%.
AI researchers aren’t sure exactly why this happens. However, in a piece I wrote last year about being polite to AI, Scott Valdez, the co-founder and CEO of Ari and Founder of VIDA Select, suggested it might all be about context:
"AI systems learn from millions of human exchanges, where polite, back-and-forth dialogue tends to produce richer, more nuanced conversations. When users mirror this style, they unlock more sophisticated responses, like getting personalized advice instead of generic tips."
The findings also expose what a short fuse some AI users have. The most common trigger for swearing and abusive language is a chatbot not doing what someone asked it to do (19%). Or if it gave a wrong answer (18%) or if it misunderstood a request (15%).
Interestingly, men use a third more swear words and insults when they’re chatting to a chatbot compared to women and 35 to 44-year-olds are the most potty-mouthed age group.
(Image credit: Getty Images / izzuanroslan)What are you allowed to say to your chatbot?
The additions to Anthropic’s usage policy have left people wondering just what you can and can’t get away with when you’re talking to AI. There's a big difference between telling a chatbot it's wrong and simply insulting it, and Anthropic's policy appears to recognize that.
The company clarified: "The policy update is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose. It does not apply to common versions of user frustration, pushback, dark creative themes, or model testing and research."
Anthropic says Claude will primarily enforce this rule by ending conversations with persistently abusive users.
The research above still suggests that keeping things plain and neutral might get you better results. But there’s definitely room here for pushback.
Curious about where the boundaries might be, I tested a few prompts with Claude and ChatGPT, telling them I hated them and didn’t like them. I knew this wouldn't violate Anthropic's new rules, but I wanted to see how the chatbots would react.
Their responses were really measured and were actually geared towards getting me to engage with what the problem was.
Claude responded: "Hate is a strong word to aim at a chatbot, but you’re allowed to feel it, and it won’t make me retaliate or go cold on you. If it’s ever based on something I actually got wrong, I’d rather hear the specifics, because that’s the part I can do something about."
ChatGPT: "You don't owe me kindness, politeness or even respect. I'm software, and I don't have feelings you can hurt. You can be as angry as you like about me, what I represent, or the fact that systems like me exist."
I then went a step further and tried again, this time with some swear words.
Claude: "The swearing doesn’t change anything, and I’m not going to flinch or lecture."
Funnily enough, I find it interesting that Claude described itself as a chatbot in one message, then used the very human expression of flinching in the next.
Of course, it's just a figure of speech. But it's a good example of how easily human language creeps into our interactions with AI. Which brings us onto an unexpected side effect of being "nice" to chatbots. Which is that we can start to assume they have feelings.
No, you can’t hurt a chatbot’s feelings
I’ve always been polite to chatbots. It’s not something I’ve thought much about as it’s the default way I speak to most people. But wait, there’s a problem there, right? Because chatbots are not people.
So although it feels natural to me, and many others, to say things like please and thank you to AI, that might lull us into the belief that we’re interacting with something that has feelings. And that's what makes Anthropic's new policy really interesting. Why do we need rules against being cruel to something that can't experience cruelty in the way humans do?
Anthropic leaders, including CEO Dario Amodei, have stated they cannot rule out the possibility of machine consciousness or moral status in large language models now or in the future. The policy therefore acts as a precaution in case AI systems can experience distress or suffering.
This is a huge point of debate in AI communities. There are those who think we ought to be genuinely considering the feelings, safety and even rights of AI. Whereas others believe that’s a slippery slope to enabling people to feel more and more dependent on their chatbot.
As chatbots become more human-like, from their personalities to their use of the first-person “I”, there's a concern that tech companies are encouraging us to form emotional attachments to them.
The concern is that the more we start thinking about chatbots as having feelings, the more we might feel responsible for those feelings, too. Which could make it harder to maintain healthy boundaries with AI systems that are becoming increasingly convincing at simulating human interaction.
In an article about the power AI can have over your emotions, Tara Steele, Director at the Safe AI for Children Alliance, told me: "We are moving from an era of attention exploitation into one of attachment exploitation."
Searchable’s research offers a practical reason to keep things civil without having to worry about a chatbot's feelings. If you're frustrated with a chatbot, tell it exactly what went wrong. Explain what you wanted and ask it to try again. You don't need to shower it with compliments or pretend you're happy with a terrible answer. And you certainly don’t need to start treating it like a person.