AI Scrapers Consume a Fifth of git.kernel.org's Capacity

14 CPU cores permanently rendering git commits

Konstantin Ryabitsev, who maintains the kernel.org infrastructure, has published hard numbers on what AI scrapers cost git.kernel.org. Across five geo-distributed nodes, 14 CPU cores are doing nothing but rendering git commits as HTML at any given moment. In his wording, the site spends more CPU cycles producing output for scrapers than it spends on all other kinds of legitimate access combined, including git clones.

The load is not a spike but a constant "background radiation": a permanent share of capacity devoted to output useful for a single purpose — feeding a learning model.

Why kernel.org is worth scraping

Linux development happens in the open. Both the git repositories and the mailing-list archives can be cloned, and the content is guaranteed to be free of machine-generated text — an unusually clean corpus for training. Ryabitsev compares training a model on model output to a "digital prion disease", which is why an archive whose entire history predates large language models is especially valuable.

Scrapers nevertheless ignore the efficient route. Instead of cloning linux.git — about 1.48 million commits, with roughly 922 forks of it hosted on the server — they render each commit as an HTML page and parse the result. Because cgit also serves patches, plain renders and diffs between arbitrary commits, a single fork produces an astronomically large number of valid URLs; across hundreds of forks the address space runs into the billions.

From fail2ban to proof-of-work

Blocking started with fail2ban and user-agent inspection, moved to bans by IP and then by whole ASN as bots began impersonating ordinary browsers, and collapsed once crawlers started arriving from millions of residential and mobile addresses in short bursts — a pattern driven by proxy SDKs embedded in consumer devices such as smart TVs.

The next defence was Anubis, a proof-of-work challenge that asks a client to compute a hash with a given number of leading zeroes. It worked immediately, but a few months later the bots were solving difficulty 4; raising it to 5 bought more months, and today they solve that too. Legitimate users pay the price: a few seconds of computation on a phone, which then runs uncomfortably warm.

Legitimate traffic is about 2%

Today git.kernel.org receives roughly six million requests a day asking for random commits. About 66% are turned away by the Anubis challenge, while 33% solve the maths and reach the main site. Under generous assumptions, legitimate requests are only around 2% of traffic. Of the 90 cores spread across the five nodes, 14–16 render commits for scrapers — about 20% of total capacity on average, though the real graph arrives in waves and is far spikier than a flat line.

The project's response is to turn off features in order to reduce the number of crawlable URLs and to gate operations that are expensive to run, so anonymous visitors should expect to lose some functionality. All of the data will still be available for download to anyone who asks — just with more hoops to jump through. Ryabitsev notes there is no simple solution while new model vendors and monetised proxy networks keep appearing.