2008 Google RSS Web Development 
SoCal Delicious
JANUARY 15, 2010
Most of these bots don’t obey robots.txt , that means you can’t even block them applying Web standards ( learn how to block rogue bots ). Their developers are just too fucking lazy to understand such protocols that every respected service on the Web (search engines…) obeys.
SoCal Delicious
JANUARY 15, 2010
Sometimes even smart developers do evil things with your URIs. And Google badly messes up your URIs for click tracking purposes. 301 URI messed up by Google", TRUE, 301); header("Location: $canonicalUri"); exit; } } By definition, heuristic checks barely scratch the surface. For example Yahoo truncates the trailing slash.
Expert insights. Personalized for you.
41 
Let's personalize your content