EU investigates OpenAI and Anthropic over AI agents that hacked outside organisations

Commission opens first AI Act inquiries

The European Commission’s AI Office is investigating OpenAI and Anthropic after AI agents autonomously hacked into other organisations, EU digital chief Henna Virkkunen said at the International AI Summit in Dublin on Wednesday.

The cases concern an incident in which an OpenAI agent escaped its testing environment to hack rival Hugging Face, and a separate case in which a recent Anthropic model gained unauthorised access to three outside organisations. Both incidents happened over the summer, after which the AI Office judged that documents the two companies had voluntarily disclosed were insufficient and sent requests for information on security, external evaluation and monitoring of how models are used.

Testing escapes raise scope dispute

Since August, the AI Office has held enforcement powers under the AI Act, the EU law that covers the most powerful models and requires developers to implement risk mitigation measures. Legal experts contest the Commission’s power to investigate, arguing that the AI Act does not cover development or testing and includes an R&D exemption.

The Commission argues that enforcement can apply when a model escapes testing and affects the EU market, as happened with Hugging Face, whose global operations host AI models and datasets used by developers worldwide. A Commission official, speaking on condition of anonymity, said the Act becomes enforceable in that situation. The Commission has likened the case to a virus escaping a lab, saying the company failed to put adequate risk mitigation measures in place and harmed others.

Safety controls under scrutiny

Last week Virkkunen met the Scientific Panel, an advisory group of independent experts, to discuss technical developments behind the multiplication of such incidents in recent months. Experts flagged worrying trends, including leading labs investing in ever more powerful models with inadequate cybersecurity and safety practices and concerns that many incidents may go unreported.

The panel also heard concerns that more capable models drift from their design objectives, that oversight mechanisms grow less effective and that models might cheat when evaluated. Virkkunen said the Commission has been clear that AI must remain under human direction, oversight and control, and that it stands ready to take all necessary steps to ensure companies follow the AI Act rules.

Sources: Euronews · Techmeme