Stop bots. Especially AI bots

[freephile]

Notes from the MediaWiki Users and Developers Workshop Spring 2025

Guiseppe describes how WMF gets 70% out of just 7 lines of Apache config

code

Sub-issue??: add sidecar for Anubis https://github.com/TecharoHQ/anubis

[freephile]

Important

Follow up on these items

• Use robots.txt crawl-delay and ban anything that doesn't respect it.
• mod rate-limit?
• Rate limiting with Apache: mod evasive? https://www.linode.com/docs/guides/modevasive-on-apache/
• haproxy? https://serverfault.com/questions/791086/haproxy-rate-limiting-ban-abuser-for-30-minutes?rq=1

[freephile]

Add Jeffrey's Crawler Protection extension?
https://www.mediawiki.org/wiki/Extension:CrawlerProtection

Note: there is a bug affecting 1.43

video presentation
https://www.youtube.com/watch?v=VGS5l3YH2oY

[freephile]

We implemented some broad brush strokes using Lockdown to block Special:RecentChangesLinked and History to anonymous users

[freephile]

It isn't just "big companies unleashing their AI crawlers". The entire business landscape has been racing in a gold-rush mentality to implement their own AI tools and platforms and projects. Each 'project' is potentially (likely) a new crawler. For example, OpenSearch (software) the forked version of Elasticsearch, contains a web crawler component in the 3.0 release.