Everything you ever wanted to know about robots.txt and then some.
_Then in 2007, they announced that all three of them would support the Sitemap directive in robots.txt files. And yes, that important piece of internet history from the blog of a formerly 125 Billion dollar company now only exists because it was archived by Archive.org._
https://intoli.com/blog/analyzing-one-million-robots-txt-files/
#googplus