Anthropic bans abusive behavior toward Claude in first usage policy update in over a year

Anthropic has updated its usage policy for the first time in more than a year, introducing new restrictions on election interference, weapons development, surveillance, and health and financial uses of its Claude AI models. The most significant change prohibits "sustained and needless abusive or cruel behavior" toward Claude.

Model welfare

The update builds on a move from last August, when Anthropic said Claude could end conversations with "persistently harmful or abusive" users as part of its research into "model welfare." The company said terminating conversations remains the "primary enforcement mechanism" and did not comment on whether further measures, such as user bans, could follow.

Anthropic wrote that the new rule "is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose." It does not apply to common versions of user frustration, pushback, dark creative themes, or model testing and research.

The change comes amid an industry debate over model welfare. Anthropic CEO Dario Amodei said in February, "We don't know if the models are conscious." A code of conduct draft posted by Microsoft AI's Mustafa Suleyman in September stated, "The idea of model welfare is wrong. AI's should not have rights or legal personhood," before being revised to a more measured position rejecting the pursuit of legal personhood for models.

Propaganda and elections

Anthropic also gathered several previously "scattered" restrictions into a new ban on deceptive commercial or political campaigns, addressing the growing issue of AI-generated propaganda. The policy restricts "efforts to obscure who is behind a message or amplify content through fake accounts or posts."

Its election section prohibits voter deception and election disruption, including spreading misinformation "about candidates or how to vote, impersonating candidates or election officials, or trying to suppress turnout."

Weapons and surveillance

Although the usage policy has always publicly banned weapons development using Claude, Anthropic said it has seen multiple attempts to use Claude to develop guidance for and control software for weapons. The expanded ban now covers "software and components that make weapons work, as well as actions like arming drones and other autonomous vehicles."

Following its last threat intelligence report, which suggested growing use of Claude-powered surveillance to identify and track "political dissidents," the company clarified its surveillance bans. "Tracking people without their consent is prohibited, whether it happens in real time or through analysis of previously collected data," Anthropic wrote. Claude cannot be used to decide or recommend who to investigate, arrest, or charge in law enforcement or criminal justice processes, nor to build or improve tools designed for surveillance.

Anthropic noted these rules may be modified for "contracts with certain governmental customers" if it judges the contractual use restrictions and applicable safeguards adequate to mitigate potential harms. The company has previously contracted with the US military.

Hardware and robotics

A new rule requires that when Anthropic's models are "connected to hardware that takes autonomous physical actions and which might be capable of causing injury," a "qualified operator must be able to observe the equipment and stop it if needed." It is not clear whether that operator must be human. The change reflects a growing trend of AI labs investing in robotics and other hardware to give AI systems physical embodiments, and may offer hints at Anthropic's future partnerships in that field.

Sources: The Verge Ai