Cloudflare and OpenAI pilot brings hundreds of millions of new pages into ChatGPT search

Cloudflare and OpenAI share early results from AI discovery pilot

Cloudflare and OpenAI launched a research pilot in July 2026 to find out whether signals from Cloudflare's network could help ChatGPT users discover new content more efficiently. According to a joint announcement, the pilot brought previously unknown pages into OpenAI's search index, giving ChatGPT more sources to draw on when answering questions. The participating cohort represented about 1% of Cloudflare's total network, and through it OpenAI discovered and indexed hundreds of millions of new web pages.

Filtering noise before crawling

The collaboration surfaced practical lessons about what makes a page worth discovering. The partners observed how easily automated traffic can make a nonexistent page look interesting, including bots repeatedly requesting files that do not exist, sites redirecting almost any address to their homepage, and addresses that serve copies of the same content. Recognizing these patterns lets the system cut wasted work before a crawler fetches a page. The pilot is also testing signals that flag potentially harmful pages, such as a fake login form planted on an otherwise legitimate site. Flagging the single compromised page rather than the whole domain keeps people safe without penalizing the site owner.

Less crawl load, more relevant answers

For website owners, the signals reduce unnecessary crawler traffic. Filtering keeps crawlers away from pages that do not exist or that redirect elsewhere, while change signals tell them when a page is worth another visit and when it is not. The partners note a balance to get right: finding new pages can initially increase crawling as those pages are evaluated and indexed, but the goal is to stay current while reducing total work. OpenAI independently evaluates and ranks how pages appear based on relevance, and ChatGPT Search links back to sources so people can explore the original content. The work complements OpenAI's GPT-6 and Intelligent UI efforts to make ChatGPT more visual and interactive for shopping and learning.

Control and next steps

Cloudflare stresses that domain owners' choices matter throughout: a signal about a page does not grant permission to access it. The research uses metadata about the web for search and discovery rather than sharing page content for model training, and site owners can use Cloudflare's AI Crawl Control to manage how AI systems access their websites. The next phase will improve page selection, test faster and more reliable change signals, and measure what happens after content is discovered, as both companies work to translate better discovery into real business outcomes.

Sources: Cloudflare