Wiki server overwhelmed by scrapers

[flexibeast]

Hi all,

A heads-up, quoting the "News" column on the "Main Page" of the wiki:

The wiki has been enduring heavy scraper bot activity recently, to the point that the site has been repeatedly overwhelmed. These overloads have been causing errors such as 500 Internal Server, Bad Gateway (502), Gateway Timeout (504), or long or incomplete page loading. The infrastructure team has been working to keep the servers running through all this, and hopefully the issues will be contained.

The core services powering Gentoo, connectivity, parts, repair, remote hands, hosting, and power all need to be paid for by Gentoo - any donations to help keep the lights on are appreciated.

[s0ulslack1]

Maybe an rss type setup could help?

[flexibeast]

Maybe an rss type setup could help?

Sorry, i'm not sure i follow .... ? What does RSS have to do with things like (say) AI bots scraping content to use as training data?

If a bot is merely wanting to know the latest changes, it could simply examine the "Recent Changes" page. i'm not aware that this is available as an RSS/Atom feed; if it's not, i imagine there's a MediaWiki plugin that could provide one. But at any rate, the format of the "Recent Changes" page is consistent and could be 'parsed' (in the broad sense) for the required information.

[Banana]

if the server serving is overwhelmed any other solution served by this server is also affected. RSS as another solution will not help, only if it is server elsewhere.

[NeddySeagoon]

Bugs and the forums get hit with the same problem.

[gorg86]

The last 2 weeks I had to read through a lot of wikis and that was very annoying.
I'm not in charge of the infrastructure ofc, but I'm certain that is being caused by bots that scrape for AI.
We need to find a way to block those bad actors.
Other sites are suffering from this, too.

[Chiitoo]

We are playing whack-a-scraper a lot in the background, but can't quite keep up with it.

The other option is something like Anubis, which will be very restrictive, though I want to get back to trying it out and configure it so all the real people will get in.

[gorg86]

I while back I read a post on the level1techs forum (I think), where someone managed to solve the AI bot issue without breaking stuff.
I'll try to dig it up.
EDIT:
Close enough, it was Geoffrey McRae and he had some help: https://x.com/geoffrey_mcrae/status/2004062396697268619
I'm not a fan of Cloudflare...

[Zucca]

Close enough, it was Geoffrey McRae and he had some help: https://x.com/geoffrey_mcrae/status/2004062396697268619
I'm not a fan of Cloudflare...

There has been some talk about Crowd Sec before. It is, for sure, an interesting project.