robots.txt and other txt files? (HTML, JS, another tricks)

by numberonespinelapologist, Sunday, October 18, 2026, 14:41 (7 hours, 8 minutes ago) @ felix

Yeah that definitely is a much simpler approach haha, on the list I found there are hundreds of bots already.
Seems like it is a difficult balance between keeping bad bots out and being findable by others. I applied to join a bunch of webrings in the hope that it helps with the second part, at least.


Complete thread:

 RSS Feed of thread