Remove Google Remove IP Remove Patents Remove RSS
article thumbnail

How brain-amputated developers created the social media plague

SoCal Delicious

Topsy , for example, does respect the content producer — so morons using “Python-urllib/1.17″ or “AppEngine-Google; (+[link] appid: mapthislink)&# could obey the Robots Exclusion Protocol (REP), too. ThingFetcher sometimes requests a (shortened) URI 30 times per second, from different IPs. googlebot.com&#.

article thumbnail

How to cleverly integrate your own URI shortener

SoCal Delicious

While handling bots, you should detect search engine crawlers, too: /* lookup your crawler IP database to populate $isCrawler; then, if the IP wasnt identified as search engine crawler: */ if ($isCrawler !== from the IP address range 65.52.0.0 Other rogue bots identify themselves by IP addy, user agent, and/or referrer.

SEO 37