Google has millions of spiders, in datacenters all over the world. Maybe respect... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

sbierwagen on June 30, 2014 | parent | context | favorite | on: Robots.txt Disallow: 20 Years of Mistakes To Avoid

Google has millions of spiders, in datacenters all over the world. Maybe respecting crawl delay added more shared-state overhead than they wanted.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact