In this piece
Agent-Readable Websites: llms.txt, Markdown Mirrors and What Actually Breaks
Make your public content easy to retrieve and understand, then validate any extra formats you publish. Our implementation uses HTML, crawler controls, Markdown and JSON-LD. Keeping these representations consistent is an engineering problem we can test.
We run this stack on the site you are reading: a Markdown mirror of every page, an llms.txt per language, published agent skills, and a verifier that fails the production build when any of it breaks. This post is the list of things that verifier has caught, which is a more useful list than the one you get from reading the specification.
Want to know whether an agent can read your site before you invest in fixing it?
Run the free checkerFour surfaces in our implementation
These surfaces serve different consumers. They are not four universal prerequisites for AI visibility. A useful HTML page can be retrieved and cited without a Markdown mirror or JSON-LD.
| Surface | Question it answers | Failure looks like |
|---|---|---|
| robots.txt | Which crawler policy applies? | A restriction in the relevant group and path |
| Served HTML | What can a consumer read without rendering JavaScript? | A sparse shell for consumers that do not render |
| Markdown mirror (optional) | Can a chosen consumer use a faithful alternate copy? | Broken links or lost content in that copy |
| JSON-LD (optional) | Are public facts expressed consistently? | Invalid or conflicting structured facts |
Google’s guidance says llms.txt, Markdown and special schema are not required for its AI search features. It can process JavaScript under its normal SEO rules. Index and snippet eligibility, plus the effective Search Console generative-AI inclusion setting, matter. Confirm those separately from optional-format tests. Google Search Central.
llms.txt, honestly
llms.txt is a convention, not a web standard. No engine is obliged to read it, and anyone selling you guaranteed pickup is overselling. It is also cheap to generate and it gives an agent a clean map instead of your rendered navigation, which is why we publish one and usually recommend one.
If you publish it, three things decide whether it is useful:
- Portable URLs. The v2 convention accepts Markdown links and does not make absolute URLs mandatory. Relative links work when a consumer retains the llms.txt source URL as its base, but that context is sometimes lost when the file is copied into another tool. We use absolute URLs to keep the map portable.
- The blockquote under the H1. The H1 is the only required section; the convention describes the following summary as optional. Add it anyway when one sentence can identify the site without forcing the agent to infer that context.
- Notes on important links. Notes after each link are optional, but they help an agent with a limited budget decide which resource to open. Add concise descriptions where a title alone is ambiguous.
Markdown mirrors and how to advertise them
A mirror is the same content as the page, minus the chrome, served as Markdown at a predictable path and pointed to from the page head:
<link rel="alternate" type="text/markdown" href="/services/ai-visibility.md">Generating mirrors is straightforward. Keeping them faithful is the part that needs a machine, because a mirror can be subtly wrong in ways nobody notices for months. The generator runs after the site build, walks the rendered HTML, and the verifier then compares the two.
The failure modes worth knowing about
These are real findings from our own build, not hypotheticals. Each one shipped at least once before the gate existed.
| Failure | Why it happens | What an agent sees |
|---|---|---|
| Unresolved template value | A generator emits its no-value sentinel, or a pair of template delimiters survives unrendered, and nobody reads the output | A page that quotes your templating language back at you |
| Unbalanced code fence | An opening fence with no closing one | Every heading after it stops being a heading, so the document loses its structure |
| Unresolved HTML entity | Entities not decoded during conversion | An undecoded ampersand or apostrophe entity, read as its literal characters rather than as the punctuation |
| Collapsed adjacent links | Whitespace between two anchors dropped in conversion | Two link texts glued into one phrase |
| Attribution glued to its link | A missing space before an author link | An author string where the word before the link is fused to the name |
| Table column mismatch | Header row and delimiter row disagree on column count | The table stops parsing as a table, so every number loses its column |
| More than one H1 | Chrome headings leaking into the body | Ambiguity about what the page is even about |
| Empty body | Content injected client-side, so the mirror has nothing to mirror | Front matter and silence |
Our favourite was subtler than any of these. Adding one decorative glyph to a page put a literal double-quote character inside an SVG text node, which stopped the HTML minifier inside foreign content. The rest of that page shipped unminified, and its Markdown mirror silently truncated at two thirds, taking the entire FAQ with it. Nothing looked wrong in a browser. The gate failed the build, named the route, and the fix was one character.
The robots.txt trap: retrieval is not training
This is the single most expensive misunderstanding in the topic, and it is one line of configuration.
GPTBot collects training data; OAI-SearchBot supports search, and ChatGPT-User handles user-initiated visits. Anthropic similarly distinguishes ClaudeBot, Claude-SearchBot and Claude-User. A restriction affects the documented purpose of that agent, not every possible answer about your site:
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /The first group declares a GPTBot training restriction. The wildcard applies when no more specific group matches. Specific groups take precedence; matching path rules use the longest match, with Allow winning a tie. This example restricts compliant search crawlers without their own group. It does not prove that every assistant cannot mention or cite the site.
Keep training preferences separate from search and user-initiated retrieval. OpenAI notes that robots rules may not apply to ChatGPT-User. Check each vendor’s documentation and the actual deployed response before changing policy. OpenAI; Anthropic.
Why a gate, and not a checklist
Every item above is easy to fix once and impossible to keep fixed by intention. The content changes weekly, the templates change monthly, and none of these failures produces a visible symptom. A quarterly audit finds them a quarter late.
Our production verifier checks the mirrors we choose to publish. The agent readability checker ports selected mirror rules into a local browser tool. It checks one pasted file, with explicit coverage limits; it does not run our full-site build or fetch linked pages.
For the neighbouring problems, our Open Knowledge Format guide covers packaging internal knowledge as portable Markdown, and the AI-ready company wiki guide covers serving that knowledge to your own agents. This post is strictly about the public surface: what somebody else's agent can read.
Frequently Asked Questions
What makes a website agent-readable?
Is llms.txt a standard?
Do relative URLs work in llms.txt?
Should we block GPTBot?
Do Markdown mirrors duplicate content for search engines?
How do we stop this decaying after launch?
Final thoughts
Start with readable public content and the right access controls. Publish extra formats when a consumer benefits from them, and put their integrity checks in the build.
Our checker helps you inspect one file locally. Use its evidence to fix the tested issue, then verify deployed access and search eligibility separately.