AI bots swarm university libraries, threaten public access

Academic library websites are being slowed or crashed by data-scraping bots used by AI and third-party companies to gather content for large language models. Library staff say the traffic is disrupting access for students, researchers and the public, and experts warn the problem is getting worse. One university reported more than 11,000 searches for Finnish music from thousands of addresses in a single day.

Bots target open collections

Libraries, museums and other cultural heritage institutions are prime targets because they hold quality, vetted information that is generally not hidden behind paywalls or logins. Workers at San Jose State University and the University of Victoria say the bots are trying to access archives that include academic, historical and government documents. The bots are used by both AI and third-party companies to scour websites and retrieve data, often to feed models such as Claude, Gemini and ChatGPT.

Good bots and bad bots

Eric Hellman of Project Gutenberg says search engines and archiving sites use bots that follow robots.txt, voluntary instructions that help them avoid disrupting a site. He says AI scraping bots sometimes ignore those rules and can request thousands of documents a second in every available format. Because one company or bot farm can work from thousands of IP addresses, Hellman says blocking individual addresses is no longer useful.

Libraries weigh access barriers

Some libraries have set up CAPTCHA challenges or other verification systems, but those measures can also affect good bots. Some have begun requiring login credentials, a move staff say they want to avoid because it puts barriers on human access to material they believe should be easily accessible. Michael Weinberg of GLAM-E says bot traffic now creates a disincentive for institutions to make parts of their collections available online.

AI firms and opt-out signals

Major AI firms have argued that scraping publicly available data is permissible under fair use and in the public's best interest. Anthropic, OpenAI, Google and Microsoft recently expressed support for honouring a single standardized opt-out signal in a website's robots.txt file that could block their web crawlers. Compliance would be voluntary, and the agreement leaves out problematic bots from third-party companies and smaller AI startups.

Sources: cbc.ca