Fri Nov 6 01:34:05 EST 2009
frickin' robots, man
207.241.227.75 - - [05/Nov/2009:16:00:55 -0500] "GET /robots.txt HTTP/1.1" 200 146 "-" "ia_archiver-web.archive.org" 207.241.227.99 - - [05/Nov/2009:16:01:56 -0500] "GET /robots.txt HTTP/1.1" 200 146 "-" "ia_archiver-web.archive.org" 207.241.227.72 - - [05/Nov/2009:16:01:57 -0500] "GET /robots.txt HTTP/1.1" 200 146 "-" "ia_archiver-web.archive.org" 207.241.227.95 - - [05/Nov/2009:16:01:57 -0500] "GET /robots.txt HTTP/1.1" 200 146 "-" "ia_archiver-web.archive.org" 207.241.227.73 - - [05/Nov/2009:16:02:02 -0500] "GET /robots.txt HTTP/1.1" 200 146 "-" "ia_archiver-web.archive.org" 207.241.227.86 - - [05/Nov/2009:16:02:02 -0500] "GET /robots.txt HTTP/1.1" 200 146 "-" "ia_archiver-web.archive.org" 207.241.227.94 - - [05/Nov/2009:16:02:24 -0500] "GET /robots.txt HTTP/1.1" 200 146 "-" "ia_archiver-web.archive.org" 207.241.227.98 - - [05/Nov/2009:16:02:25 -0500] "GET /robots.txt HTTP/1.1" 200 146 "-" "ia_archiver-web.archive.org"
Dear Internet Archive,
My robots.txt doesn't disallow you.
You can stop downloading it.
Love, bbot.
123.125.66.82 - - [05/Nov/2009:14:21:58 -0500] "GET / HTTP/1.1" 200 1558 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)" 220.181.7.64 - - [05/Nov/2009:14:24:56 -0500] "GET / HTTP/1.1" 200 1558 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)" [snip] 123.125.66.126 - - [05/Nov/2009:15:28:36 -0500] "GET / HTTP/1.1" 200 1558 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)" 220.181.7.115 - - [05/Nov/2009:15:29:03 -0500] "GET / HTTP/1.1" 200 1558 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
Dear Baidu,
My robots.txt does disallow you, since all you do is download my index once a hour without any further spidering. I never see any hits from you anyway, so you can fuck right off.
More love, bbot.