GLOSSARY TERM
What is Crawl-Delay?
Definition
A non-standard robots.txt directive asking crawlers to wait a specified number of seconds between requests. Honored by Bing, Yandex, and most smaller bots; ignored by Google.
Why it Matters for AI Search
If crawler load is genuinely overwhelming your server, lower your hosting plan capacity is usually the wrong fix. Use Crawl-Delay sparingly — too high a value (5+ seconds) starves smaller search engines of fresh data. For Googlebot specifically, set crawl rate in Search Console instead.
Related Terms
Browse all 105 terms →See if your site is optimized.
Run a free AI Search Readiness scan to detect missing JSON-LD schema, poor Core Web Vitals, and more.
Scan My Website