author greg-ip News Related Topics 
SoCal Delicious
JANUARY 15, 2010
If you need a top-notch nasty-bot-handler please contact the authority in this field: IncrediBill. While handling bots, you should detect search engine crawlers, too: /* lookup your crawler IP database to populate $isCrawler; then, if the IP wasnt identified as search engine crawler: */ if ($isCrawler !== crawl.baidu.com";}.
SoCal Delicious
JANUARY 15, 2010
ThingFetcher sometimes requests a (shortened) URI 30 times per second, from different IPs. A method to verify the bot All IP addresses used by a bot should resolve to server names having a unique pattern. For example Googlebot comes only from servers named "crawl" + "-" + replace($IP, ".", "-") + ".googlebot.com"
Expert insights. Personalized for you.
37 
Let's personalize your content