Every time an AI agent fetches one of your pages, it downloads your navigation, your cookie banner, your tag manager, and 400 kilobytes of script before it reaches the three paragraphs it came for. Then it pays to read all of it. If agents are now part of your audience, and your server logs say they are, the page they should get is not the page a browser gets. Serve them Markdown, from the same URL, using a header that has existed since the 1990s.
The Agent Reads Your Nav Bar Too
A marketing page is mostly furniture. Menus, footers, modals, inline scripts, tracking pixels, and class names that mean nothing outside your stylesheet. The actual argument, the pricing table, the feature comparison, is a small slice of the bytes.
Humans skip the furniture without thinking. An agent cannot. It either spends tokens parsing the noise or runs a lossy extraction step that sometimes drops your pricing table and keeps your cookie banner. Either way you lose control of how your product gets summarized, and the summary is what the buyer's assistant repeats.
Markdown fixes that. Headings stay headings, tables stay tables, links stay links, and the furniture disappears. The agent gets the same content in a fraction of the tokens, in a format it was trained to read.
Use Content Negotiation, Not a Parallel Site
The tempting shortcut is a folder of .md files or a separate /agents section. Do not. A parallel site doubles your maintenance, drifts out of date within a month, and invites the exact question you do not want from a search engine: why do these two versions disagree?
Content negotiation avoids all of it. The client sends Accept: text/markdown. Your server returns Markdown at the same URL. A browser sends Accept: text/html and gets the normal page. One canonical URL, two representations, and the content is rendered from the same source.
Two details matter. Always send Vary: Accept so CDNs and caches store both versions separately, otherwise a cached Markdown response will be served to a browser and break your homepage for an afternoon. And return Content-Type: text/markdown; charset=utf-8 so clients know what they received.
| Approach | Maintenance | Drift risk | Cache safety | Verdict |
|---|---|---|---|---|
| Parallel /agents site | High | High | Easy | Avoid |
| Static .md files beside pages | Medium | Medium | Easy | Fine for docs only |
| Accept header at the edge | Low | None | Needs Vary: Accept | The default choice |
| Scraper-side extraction | None for you | n/a | n/a | You lose control of the result |
Build It at the Edge
You do not need to touch your CMS templates. An edge worker or middleware layer sits in front of the origin and does four things.
First, inspect the Accept header. Only switch when text/markdown is explicitly requested and ranks above text/html. A wildcard should still get HTML.
Second, fetch the HTML you were already going to serve, extract the main content region, and convert it with a library such as Turndown or a Cloudflare, Vercel, or Netlify equivalent. Strip nav, footer, scripts, and consent UI before conversion, not after.
Third, add a small front matter block with the title, canonical URL, and last modified date. Agents use this to cite you correctly, and it gives them a freshness signal the HTML often buries.
Fourth, cache the result keyed on URL plus Accept. Conversion is cheap, but you do not want to run it on every agent hit.
Start with the pages agents actually ask for: pricing, product pages, documentation, and comparison pages. You can see them in your logs by filtering for known AI user agents, and by watching which paths get requested with a Markdown Accept header once you enable it.
Do Not Turn This Into Cloaking
Search engines penalize showing crawlers different content than users see. Markdown negotiation is safe only if the substance is identical. Keep these rules and you stay on the right side of it.
Same facts, same numbers, same claims in both versions. Do not stuff the Markdown with keywords or extra promotional copy. Point both versions at the same canonical URL. Never gate the Markdown behind user agent sniffing alone, because the Accept header is the honest signal and user agent strings are easy to spoof.
Then measure it. Log Markdown responses separately, track which agents request them, and compare against your llms.txt and robots.txt policy so the three tell one consistent story.
Markdown
- Pull 30 days of logs and list the AI user agents hitting your site
- Pick the five pages agents request most
- Add Accept header detection at the edge, defaulting to HTML
- Convert only the main content region, never the whole document
- Send Vary: Accept and a text/markdown content type
- Add front matter with title, canonical URL, and last modified
- Confirm HTML and Markdown carry identical facts and prices
- Log Markdown requests by agent and review monthly
Your site already has a human interface. The agent interface is a header and a converter away. Build it before someone else's summary of your pricing becomes the one everybody repeats.
Tags
LETSGROW Dev Team
Marketing Technology Experts
Ready to Apply This Insight?
Schedule a strategy call to map these ideas to your architecture, data, and operating model.
Schedule Strategy Call