« Precompiled ASP.NET web sites in Mono | Main | Gnome-keyring-sharp 1.0.2 released »

msnbot/2.0b and robots.txt

Sunday, Aug 30, 2009 01:00 AM GMT
Looks like the new MSN spider is having some trouble following the rules. While they fix it, this can be blocked in Apache by doing:
 SetEnvIfNoCase user-agent "^msnbot/2.0b" banned_by_now
 ...
 <Directory "/srv/www/htdocs/">
    Order allow,deny
    Allow from all
    Deny from env=banned_by_now
 </Directory>
This is a personal web page. Things said here do not represent the position of my employer.