Remove 2005 Remove 2009 Remove IP Remove RSS
article thumbnail

How to cleverly integrate your own URI shortener

SoCal Delicious

Shortcut: sebastian-x.com Blog About Blogroll Links How to cleverly integrate your own URI shortener Posted on 30 December, 2009 This pamphlet is somewhat geeky. from the IP address range 65.52.0.0 Other rogue bots identify themselves by IP addy, user agent, and/or referrer. crawl.baidu.com";}. to 65.55.255.255 (131.107.0.0

SEO 37
article thumbnail

How brain-amputated developers created the social media plague

SoCal Delicious

ThingFetcher sometimes requests a (shortened) URI 30 times per second, from different IPs. A method to verify the bot All IP addresses used by a bot should resolve to server names having a unique pattern. For example Googlebot comes only from servers named "crawl" + "-" + replace($IP, ".", "-") + ".googlebot.com"