Information IP SEO Yahoo 
SoCal Delicious
JANUARY 15, 2010
While handling bots, you should detect search engine crawlers, too: /* lookup your crawler IP database to populate $isCrawler; then, if the IP wasnt identified as search engine crawler: */ if ($isCrawler !== from the IP address range 65.52.0.0 Other rogue bots identify themselves by IP addy, user agent, and/or referrer.
SoCal Delicious
JANUARY 15, 2010
ThingFetcher sometimes requests a (shortened) URI 30 times per second, from different IPs. ThingFetcher update: The owners of ThingFetcher are now aware of the problem, and will try to fix it asap ( more information ). A method to verify the bot All IP addresses used by a bot should resolve to server names having a unique pattern.
Expert insights. Personalized for you.
37 
Let's personalize your content