<?xml version="1.0" encoding="utf-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/">
<channel>
<title>Web1.0 Hosting Forum - robots.txt and other txt files?</title>
<link>https://forum.w10.host/</link>
<description>Save the Old Web!</description>
<language>en</language>
<item>
<title>robots.txt and other txt files? (reply)</title>
<content:encoded><![CDATA[<p>Yeah that definitely is a much simpler approach haha, on the list I found there are hundreds of bots already.<br />
Seems like it is a difficult balance between keeping bad bots out and being findable by others. I applied to join a bunch of webrings in the hope that it helps with the second part, at least.</p>
]]></content:encoded>
<link>https://forum.w10.host/index.php?id=252</link>
<guid>https://forum.w10.host/index.php?id=252</guid>
<pubDate>Sun, 18 Oct 2026 14:41:25 +0000</pubDate>
<category>HTML, JS, another tricks</category><dc:creator>numberonespinelapologist</dc:creator>
</item>
<item>
<title>robots.txt and other txt files? (reply)</title>
<content:encoded><![CDATA[<p>I used to have something like that, but in the end grew tired of playing whack-a-mole and switched to a much simpler robots.txt found on the SDF website:</p>
<p>User-agent: Ariadne<br />
User-agent: Wibybot<br />
User-agent: MojeekBot<br />
User-agent: DuckDuckBot<br />
User-agent: search.marginalia.nu<br />
Disallow:</p>
<p>User-agent: *<br />
Disallow: /</p>
<p>It does keep out a lot of bots, but on the plus side it's opt-in. Anything that ignores it is clearly malicious, even if it's one of the big names.</p>
]]></content:encoded>
<link>https://forum.w10.host/index.php?id=250</link>
<guid>https://forum.w10.host/index.php?id=250</guid>
<pubDate>Sun, 18 Oct 2026 12:04:56 +0000</pubDate>
<category>HTML, JS, another tricks</category><dc:creator>felix</dc:creator>
</item>
<item>
<title>robots.txt and other txt files? (reply)</title>
<content:encoded><![CDATA[<p>I know this thread is old, but i am replying just in case someone else is looking for an answer:</p>
<p>I found this useful site in which someone is maintaining a big list of bots and a ready made robots.txt file.</p>
<p><a href="https://baccyflap.com/res/robots/">https://baccyflap.com/res/robots/</a></p>
]]></content:encoded>
<link>https://forum.w10.host/index.php?id=247</link>
<guid>https://forum.w10.host/index.php?id=247</guid>
<pubDate>Fri, 16 Oct 2026 21:28:27 +0000</pubDate>
<category>HTML, JS, another tricks</category><dc:creator>numberonespinelapologist</dc:creator>
</item>
<item>
<title>robots.txt and other txt files? (reply)</title>
<content:encoded><![CDATA[<p><a href="http://web1.0hosting.net/faq/">http://web1.0hosting.net/faq/</a> have you read this?</p>
<p>if there are no answers to the questions there, I would like to hear a more specific question.</p>
<p>regarding robots.txt, the simplest instructions are on the Internet, we did not post them because they are the same for everyone and everywhere</p>
]]></content:encoded>
<link>https://forum.w10.host/index.php?id=147</link>
<guid>https://forum.w10.host/index.php?id=147</guid>
<pubDate>Fri, 21 Mar 2025 20:35:09 +0000</pubDate>
<category>HTML, JS, another tricks</category><dc:creator>Turboblack</dc:creator>
</item>
<item>
<title>robots.txt and other txt files?</title>
<content:encoded><![CDATA[<p>I get Robots.txt is supposed to help with automated web crawlers, but anyone have a 'for dummies' overview?</p>
<p>Are there any other *.txt files I could include at the top level that could do neat things?</p>
]]></content:encoded>
<link>https://forum.w10.host/index.php?id=145</link>
<guid>https://forum.w10.host/index.php?id=145</guid>
<pubDate>Fri, 21 Mar 2025 20:10:53 +0000</pubDate>
<category>HTML, JS, another tricks</category><dc:creator>singletona082</dc:creator>
</item>
</channel>
</rss>
