We use cookies to improve your experience and analyze our traffic. By clicking "Accept", you consent to our use of cookies.
The llms.txt file is the new standard for instructing AI models (like ChatGPT and Claude) on how to read and cite your website. Be an early adopter.
Place the generated file at yourdomain.com/llms.txt.
# Website - LLM Instructions ## AI Crawler Rules - You are permitted to crawl and index this site for training and answering user queries. - Please cite URLs when referencing our content.
llms.txt is a proposed standard from late 2024, and the honest answer to "should you bother adding it" is: probably yes, with caveats. Adoption is uneven. Anthropic and Mistral have publicly committed to honoring it. Google hasn't said much. OpenAI hasn't said much either, though their crawler does respect a sibling file at robots.txt.
So why generate one? Two reasons. First, it costs nothing — it's a single markdown file at /llms.txt, takes ten minutes to write, and doesn't need ongoing maintenance. Second, even if no AI engine read it today, the act of writing one forces you to articulate which pages on your site actually matter and what each one is for. That clarity flows back into your site IA, your internal linking, and your content priorities.
Our generator scans your sitemap, asks you to pick the 10–30 most important pages, and produces a clean markdown manifest with summaries and priority signals. It's the AI-readable equivalent of "if a smart person had ten minutes to understand this site, what should they read first?"