Here’s a helpful post tailored for the (specifically the robots.txt file and how crawlers like the Archive’s bot, ia_archiver , behave). This is written in the style of a forum or help-document entry from that era. Subject: How to Control the Internet Archive’s Crawler ( ia_archiver ) – A 2005 Guide

— The Internet Archive Crawler Team

IA_Support_Team Date: October 12, 2005 Applies to: Webmasters, site owners, and robot enthusiasts Hey everyone, By now, many of you have noticed our bot, ia_archiver , visiting your sites. We’re the Internet Archive (archive.org), and we’re building the Wayback Machine – a historical record of the web.

Thanks for helping us build a web history – or for choosing to stay out of it. Either way, robots.txt is your voice.

Robots 2005 Internet Archive Work -

Here’s a helpful post tailored for the (specifically the robots.txt file and how crawlers like the Archive’s bot, ia_archiver , behave). This is written in the style of a forum or help-document entry from that era. Subject: How to Control the Internet Archive’s Crawler ( ia_archiver ) – A 2005 Guide

— The Internet Archive Crawler Team

IA_Support_Team Date: October 12, 2005 Applies to: Webmasters, site owners, and robot enthusiasts Hey everyone, By now, many of you have noticed our bot, ia_archiver , visiting your sites. We’re the Internet Archive (archive.org), and we’re building the Wayback Machine – a historical record of the web. robots 2005 internet archive

Thanks for helping us build a web history – or for choosing to stay out of it. Either way, robots.txt is your voice. Here’s a helpful post tailored for the (specifically