--- title: llms.txt Will Not Rank You in AI Search. Build It for the Agents Reading Your Docs. description: The crawlers you built your llms.txt file for are ignoring it. Here is the audience that is actually reading it, and why it still belongs in your stack. author: LETSGROW Dev Team date: 2026-06-11 category: SEO tags: ["llms.txt", "AI Search", "GEO", "Developer Marketing", "Technical SEO"] url: "https://letsgrow.dev/blog/llms-txt-build-for-agents-not-ai-search" --- Here is the uncomfortable truth about the file every AI SEO consultant told you to ship last year: the crawlers you built it for are not reading it. GPTBot skips it. ClaudeBot skips it. PerplexityBot skips it. Google has said plainly it does not support the format and has no plans to. If you added an llms.txt file to chase AI search rankings, you optimized for an audience that never showed up. That does not mean you wasted your time. It means you built the right file for the wrong reason. The value of llms.txt is real, growing, and sitting in a part of the buyer journey most marketing teams are not even measuring yet. You just have to stop thinking of it as an SEO artifact and start thinking of it as the first interface your product exposes to an agent. ## The SEO promise was always thin The pitch was simple and seductive. Create a markdown file at the root of your domain that lists your most important pages in clean, link-rich prose. AI search engines would read it, understand your site faster, and cite you more often. A whole cottage industry of generators and audits sprang up around it. The adoption numbers look healthy on the surface. A study of 300,000 domains found roughly one in ten now publish an llms.txt file. But adoption by publishers is not the same as consumption by crawlers, and that is where the story falls apart. Server log analysis keeps showing the same thing: the major AI search and answer crawlers overwhelmingly ignore the file and parse your rendered HTML directly, exactly as they always have. No major model provider has committed to using it as a ranking or retrieval signal in production. So if your internal deck still lists llms.txt under "AI search optimization," move it. It is not a GEO tactic. The crawlers that decide whether you show up in ChatGPT or AI Overviews are not asking for it. ::stat-block - **10.13%** of 300,000 audited domains publish an llms.txt file - **Near zero** fetch rate from GPTBot, ClaudeBot, and PerplexityBot in server logs - **0** major LLM providers committing to it as a production search signal - **6+** coding agents actively requesting /llms.txt and /llms-full.txt by default :: ## Where the file actually earns its keep Here is the part nobody put on the sales slide. llms.txt has quietly become the first widely adopted Business-to-Agent standard. The consumers are not search crawlers. They are coding agents. Cursor, Windsurf, Claude Code, GitHub Copilot, Cline, and Aider all look for /llms.txt and /llms-full.txt when a developer points them at a documentation site. When someone is building an integration against your API and their agent can pull a clean, structured map of your docs in one request, your product becomes the path of least resistance. When the agent has to scrape your JavaScript-rendered docs and guess, a competitor with a tidy llms.txt file wins the integration. For any company with a developer audience, an API, an SDK, or technical documentation, this is the real funnel. The agent is the new top of funnel for technical adoption, and llms.txt is the file it reads first. That is a marketing surface, even if it does not look like one, because the outcome is the same as any other content asset: faster comprehension, lower friction, higher conversion to use. This reframes the whole conversation. You are not writing for a ranking algorithm. You are writing a product spec that a machine consumes on behalf of a buyer who is actively trying to use you right now. ## How to build one that an agent can actually use Most published llms.txt files are bad. Auditors looking at files in the wild are already cataloging the same anti-patterns: dumping a sitemap with no descriptions, linking to marketing pages instead of documentation, or stuffing the file with keywords as if a human SEO would ever read it. An agent does not reward any of that. It rewards clarity and structure. The standard is deliberately simple. An H1 with your project name. A short blockquote summary. Then sections of markdown links, each with a one-line description of what the agent will find behind it. The companion llms-full.txt holds the expanded content for agents that want everything in one pull. ::checklist - Write for an agent parsing markdown, not a human skimming a landing page - Lead with documentation, API references, and quickstarts, not your pricing page - Give every link a concrete one-line description of what it contains - Maintain llms-full.txt with the full expanded content for single-request ingestion - Keep it current in your release process so it never drifts from your real docs - Validate it parses cleanly, then watch your logs to see which agents fetch it :: The maintenance point matters more than the initial build. A stale llms.txt file is worse than none, because an agent will confidently hand a developer outdated endpoints and your support queue pays for it. Treat the file as part of your docs pipeline, not a one-time SEO checkbox you tick and forget. ## The honest bottom line llms.txt is a community convention, not a ratified standard. There is no IETF RFC behind it, and the search crawlers most marketers care about are not using it. If you measure success by AI Overview citations, you will be disappointed, and you should stop telling your leadership it moves that number. Build it anyway. Build it because the agents writing code against your product read it today, and because the developer choosing between you and a competitor is increasingly delegating that choice to a tool that rewards whoever made themselves easiest to parse. That is not search optimization. That is making your product legible to the machines that now sit between you and your most valuable users. The teams that understand the difference are shipping a file with the right content for the right reader. Everyone else is generating SEO theater for an audience that closed the tab.