Avatar

robots.txt and other txt files? (HTML, JS, another tricks)

by felix ⌂, Sunday, October 18, 2026, 12:04 (9 hours, 44 minutes ago) @ numberonespinelapologist

I used to have something like that, but in the end grew tired of playing whack-a-mole and switched to a much simpler robots.txt found on the SDF website:

User-agent: Ariadne
User-agent: Wibybot
User-agent: MojeekBot
User-agent: DuckDuckBot
User-agent: search.marginalia.nu
Disallow:

User-agent: *
Disallow: /

It does keep out a lot of bots, but on the plus side it's opt-in. Anything that ignores it is clearly malicious, even if it's one of the big names.


Complete thread:

 RSS Feed of thread