Ideas IP Web developers Yahoo 
SoCal Delicious
JANUARY 15, 2010
Actually, handling short URIs in your error handler is a pretty good idea when you suffer from a mainstream CMS. Here are a few ideas how to block rogue (crappy, not behaving, …) Web robots. from the IP address range 65.52.0.0 Other rogue bots identify themselves by IP addy, user agent, and/or referrer.
SoCal Delicious
JANUARY 15, 2010
ThingFetcher sometimes requests a (shortened) URI 30 times per second, from different IPs. A method to verify the bot All IP addresses used by a bot should resolve to server names having a unique pattern. For example Googlebot comes only from servers named "crawl" + "-" + replace($IP, ".", "-") + ".googlebot.com" Sad but true.
Expert insights. Personalized for you.
37 
Let's personalize your content