It is so easy to change user-agent string, so don't even bother. I don't
know how yahoo blocks all. Here is what they did it
Please try to user Perl's useragent even changed the useragent to Mozzila to
get this url
http://news.yahoo.com
If I use IE, it takes me 5 seconds, but with perl, it takes me 15 minutes.
Yahoo is first sending a cookie, but there is more, because using LYNX it is
ok to view the page.
So, I hope I can implement something similar. I am thinking using Javascript
to test it, since all PHP/Perl/Wget will not activate javascripts.
However, when googlebot come, I goto welcome it.
So is there a way not to display content to googlebot and human eyes, but
block all PHP/Perl/Wget even when they id themselves as Mozzila?