SoCal Delicious

JANUARY 15, 2010

This article is a practical guide designed to help webmasters who already know what they want to block. As I mentioned toone user who asked me for help, trying to block all undesirable bots from your site is like trying to rid the world of pests.Swat one, and another few will take its place. They look like "127.0.0.1". htaccess ".

IP 38

IP PHP Web RSS 38

SoCal Delicious

JANUARY 15, 2010

While handling bots, you should detect search engine crawlers, too: /* lookup your crawler IP database to populate $isCrawler; then, if the IP wasnt identified as search engine crawler: */ if ($isCrawler !== from the IP address range 65.52.0.0 Other rogue bots identify themselves by IP addy, user agent, and/or referrer.

SEO 37

SEO IP PHP Blogging 37

SoCal Delicious

JANUARY 15, 2010

ThingFetcher sometimes requests a (shortened) URI 30 times per second, from different IPs. A method to verify the bot All IP addresses used by a bot should resolve to server names having a unique pattern. For example Googlebot comes only from servers named "crawl" + "-" + replace($IP, ".", "-") + ".googlebot.com"

Social Media 41

Social Media Develop Development Media 41