A crawler name in your logs is a claim: 38% of bot-named requests from one Google Cloud network asked for .env files

An 82-hour analysis of one small site's access log shows that the crawler name in a user agent string proves nothing about who sent the request. In a report on dev.to, site owner Reese Calder compares the bot named in each request with the network the request actually arrived from.

What was measured

The site is small: one Cloudflare Worker answering on two hostnames. Since September 23 the Worker has also stored the ASN Cloudflare passes in request.cf.asn for each request. The window runs from 03:02 UTC on September 23 to 13:00 UTC on September 26: 82 hours and 18,846 requests.

For each request three things were kept: the crawler name in the user agent (22 names of known crawlers and fetchers), the network, and whether the path was a credential or config file (.env, .git, id_rsa, wp-config and similar). No documented crawler asks for those files.

What came out

Of the 18,846 requests, 5,180 (27.5%) came from one network, AS396982, which is Google Cloud. 2,696 of them (52%) asked for one of the files, while every other network together sent 296 such requests out of 13,666. 1,395 requests carried a crawler name and came from that Google Cloud network; 533 of them (38.2%) asked for a credential or config file.

None of the 867 requests that carried a crawler name and came from any other network asked for such a file, and 754 of those (87%) came from a network matching the crawler's operator: Applebot from Apple, bingbot from Microsoft, ClaudeBot from Anthropic. Seven names, Google-Extended (56 requests) among them, only ever showed up from that one network.

Why it matters

Google-Extended is the clean example: Google's documentation says the token has no separate HTTP user agent string and is used only in robots.txt, so a request whose user agent says Google-Extended is somebody's claim.

On September 22 the same log showed 40 to 45 hits each for five different crawler names, which looked like one client cycling through names. A bot report built on the user agent counts those as crawler visits, and a firewall rule that matches only the user agent string acts on whoever sends the string.

After publishing, the author changed his Worker: since 13:27 UTC on September 26 a request that names a crawler and asks for one of those files is labelled "other bot", and a request naming Google-Extended is labelled "spoofed crawler name".

Limits

The author stresses the data covers one small site and 82 hours. The log does not store the client IP, so nothing was compared with operators' published IP ranges, and the operator column is his own assumption. The file test only catches requests that reveal themselves, so 533 is a floor, and Google Cloud also hosts bots that identify themselves honestly (AgenstryBot, ProwlBot), so the 862 requests from that network that did not ask for such a file are not shown to be anything.