Indexing of Linux kernel > 6.14-rc6 is not available

[jinghao-jia]

The indexing of the kernel source somehow stopped again at 6.14-rc6.

---

Request date: 2025-04-08 18:27:19.044003
Path: /linux/v6.14/source
Query string:
Method: GET
Status code: 404 Not Found
Error title: File not found
Error details: This file does not exist.

[tleb]

Yes, indeed. As many other open-source projects, we are constantly being hit by waves of LLM crawlers. We have not put in place a defense layer such as Anubis as we'd prefer avoiding that.

Our first reaction method when the service is under pressure is to disable indexing which takes a lot of CPU load. Once #372 is merged, it should be much easier. Worst case, we move indexing to another server, but that isn't a great situation to be in.

---

Sources about the LLM crawlers topic:

• Mastodon post about git.kernel.org
• Article from Sourcehut CEO
• Codeberg issue, which references many more sources.

There are many others. Basically, any service hosting Git content which generates millions of pages and is somewhat expensive to serve is at the mercy of those crawlers.

[tleb]

Update: we deployed Anubis and have been indexing the kernel for four days by now. It should finish soon, at least we hope.

[jinghao-jia]

Update: we deployed Anubis and have been indexing the kernel for four days by now. It should finish soon, at least we hope.

Thanks! I have noticed the change (Anubis) when accessing Elixir yesterday.

[fstachura]

Elixir finally finished indexing. Sorry for the delay