robots.txt and other txt files? (HTML, JS, another tricks)
I used to have something like that, but in the end grew tired of playing whack-a-mole and switched to a much simpler robots.txt found on the SDF website:
User-agent: Ariadne
User-agent: Wibybot
User-agent: MojeekBot
User-agent: DuckDuckBot
User-agent: search.marginalia.nu
Disallow:
User-agent: *
Disallow: /
It does keep out a lot of bots, but on the plus side it's opt-in. Anything that ignores it is clearly malicious, even if it's one of the big names.