GLOSSARY TERM
What is robots.txt?
Definition
A text file at the root of a website that instructs search engine crawlers which pages or sections they are allowed or disallowed from accessing. It also specifies sitemap locations.
Why it Matters for AI Search
A misconfigured robots.txt can accidentally block search engines from crawling your most important pages, making them invisible in search results. In 2026, robots.txt is also used to control AI bot access (GPTBot, ClaudeBot, etc.).
Related Terms
Browse all 105 terms →See if your site is optimized.
Run a free AI Search Readiness scan to detect missing JSON-LD schema, poor Core Web Vitals, and more.
Scan My Website