Three days ago, Drew DeVault – founder and CEO of SourceHut – published a blogpost called, “Please stop externalizing your costs directly into my face”, where he complained that LLM companies were crawling data without respecting robosts.txt and causing severe outages to SourceHut.

I went, “Interesting!”, and moved on.

Then, yesterday morning, KDE GitLab infrastructure was overwhelmed by another AI crawler, with IPs from an Alibaba range; this caused GitLab to be temporarily inaccessible by KDE developers.

I then discovered that, one week ago, an Anime girl started appearing on the GNOME GitLab instance, as the page was loaded. It turns out that it’s the default loading page for Anubis, a proof-of-work challenger that blocks AI scrapers that are causing outages.

By now, it should be pretty clear that this is no coincidence. AI scrapers are getting more and more aggressive, and – since FOSS software relies on public collaboration, whereas private companies don’t have that requirement – this is putting some extra burden on Open Source communities.

So let’s try to get more details – going back to Drew’s blogpost. According to Drew, LLM crawlers don’t respect robots.txt requirements and include expensive endpoints like git blame, every page of every git log, and every commit in your repository. They do so using random User-Agents from tens of thousands of IP addresses, each one making no more than one HTTP request, trying to blend in with user traffic.

Due to this, it’s hard to come off with a good set of mitigations. Drew says that several high-priority tasks have been delayed for weeks or months due to these interruptions, users have been occasionally affected (because it’s hard to distinguish bots and humans), and – of course – this causes occasional outages of SourceHut.

Drew here does not distinguish between which AI companies are more or less respectful of robots.txt files, or more accurate in their user agent reporting; we’ll be able to look more into that later.

Finally, Drew points out that this is not some isolated issue. He says,

All of my sysadmin friends are dealing with the same problems, [and] every time I sit down for beers or dinner to socialize with sysadmin friends it’s not long before we’re complaining about the bots. […] The desperation in these conversations is palpable.

Which brings me back to yesterday’s KDE GitLab issues. According to Ben, part of the KDE sysadmin team, all of the IPs that were performing this DDoS were claiming to be MS Edge, and were due to Chinese AI companies; he mentions that Western LLM operators, such as OpenAI and Anthropic, were at least setting a proper UA – again, more on this later.

The solution – for now – was to ban the version of Edge that the bots were claiming to be, though it’s hard to believe that this will be a definitive solution; these bots do seem keen on changing user agents to try to blend in as much as possible.

Indeed, GNOME has been experiencing issues since a last November; as a temporary solution they had rate-limited non-logged in users from seeing merge requests and commits, which obviously also caused issues for real human guests.

The solution the eventually settled to was switching to Anubis. This is a page that presents a challenge to the browser, which then has to spend time doing some math and presenting the solution back to the server. If it’s right, you get access to the website.

According to the developer, this project is “a bit of a nuclear response, but AI scraper bots scraping so aggressively have forced my hand. I hate that I have to do this, but this is what we get for the modern Internet because bots don’t conform to standards like robots.txt, even when they claim to”.

However, this is also causing user issues. When a lot of people open the link from the same place, it might happen that they get served some higher-difficulty exercise that will take some time to complete; there’s one user reporting one minute delay, and another – from his phone – having to wait around two minutes.

Why? Wel