Websites Block AI Agents as Industry Rushes to Set Access Standards

Personal AI agents such as Meta's Muse, Instinct and ChatGPT's Dots promise to do more than answer questions. They can book flights, make dinner reservations and order groceries on a user's behalf. A growing problem is that many websites are blocking these agents from completing those tasks.

Agents Hit Roadblocks at Major Sites

Amazon recently began blocking Meta's Muse agent from its retail site, preventing it from browsing or purchasing from the product catalog. The issue extends well beyond Amazon. Users across social media report similar blocks, some intentional and others caused by traditional anti-bot measures designed to stop spam and malicious activity.

Walmart offers a telling example. Customers complained that Muse could not complete purchases on its site, and NBC reported the same problem in its own tests. A Walmart spokesperson said the blocks were not intentional. The company is partnered with Muse, an arrangement announced at Meta's Connect developer conference in September, and said it wants to be where its customers are, including through AI agent experiences. The apparent cause is a human verification button that must be clicked. If that experience is interrupted, the check fails and the agent is kicked out.

Airlines, Marketplaces and CDNs Weigh In

Flight booking is a leading use case for agents, yet airlines are pushing back. Delta said it has no partnership or integration enabling third-party agents to shop for or book flights, and that any future opening would have to consider security and customer experience. United pointed to its Terms of Use, which prohibit robots, spiders and other automatic devices without prior written permission.

Yelp said it does not permit non-human traffic unless an agent pays for access through its data licensing program. eBay said it does not prohibit all third-party shopping agents, but restricts unauthorized actions such as automated scraping and model training. Adidas, Zillow and Pizza Hut did not respond or declined to comment.

Cloudflare, which protects sites from unwanted automated traffic, is also under scrutiny. A change to its crawler defaults on September 15 let site owners block AI training while allowing other bots, and shifted existing sites to block AI agents on pages with ads. Cloudflare said it had no specific data to share about the blocking of personal agents, and pointed to its public data hub, Cloudflare Radar. The company runs a marketplace where AI bots pay for data access and is testing a browser built for AI agents.

An Open Standard Takes Shape

Meta, Walmart, Stripe, Sierra, Genesys, Rocket, NiCE and Decagon have begun working on an open standard for how AI agents communicate with businesses, aimed at separating legitimate agents acting on behalf of users from malicious bots. Meta said the protocol will focus on agent-to-agent communication for online commerce.

Meta is also building direct partnerships. It maintains a growing list of connectors inside the Muse app, and newly announced partners are expected to appear as the list is reorganized. "Turning away a personal agent means turning away the customer behind it," Meta said, adding that it would rather work with businesses on underlying concerns than see them lose customers. The company argued that clear norms and predictable control for businesses can lead to a more efficient way for agents to work together.

Sources: Techcrunch