robots.txt and other txt files? (HTML, JS, another tricks)

by singletona082, Friday, March 21, 2025, 20:10 (12 days ago)

I get Robots.txt is supposed to help with automated web crawlers, but anyone have a 'for dummies' overview?

Are there any other *.txt files I could include at the top level that could do neat things?


Complete thread:

 RSS Feed of thread