Design IP Patents SEO 
SoCal Delicious
JANUARY 15, 2010
Please note that my design suggestions ain’t black nor white. While handling bots, you should detect search engine crawlers, too: /* lookup your crawler IP database to populate $isCrawler; then, if the IP wasnt identified as search engine crawler: */ if ($isCrawler !== from the IP address range 65.52.0.0
SoCal Delicious
JANUARY 15, 2010
ThingFetcher sometimes requests a (shortened) URI 30 times per second, from different IPs. A method to verify the bot All IP addresses used by a bot should resolve to server names having a unique pattern. For example Googlebot comes only from servers named "crawl" + "-" + replace($IP, ".", "-") + ".googlebot.com" Sad but true.
Expert insights. Personalized for you.
37 
Let's personalize your content