Robotstxt
Web Robots (also known as Web Wanderers, Crawlers, or Spiders),
are programs that traverse the Web automatically. Search engines such
as Google use them to index the web
content, spammers use them to scan for email addresses, and they have
many other uses. by alexa.com
