Agent-Readable Websites: llms.txt, Markdown Mirrors and What Actually Breaks
An agent-readable website serves a clean machine copy of each page, advertises it in the page head, and blocks nothing that needs to fetch it. The hard part is not deciding to do this. It is keeping it true after the fourth deploy, because every failure mode here is invisible in a browser and silent in your logs.
We run this stack on the site you are reading: a Markdown mirror of every page, an llms.txt per language, published agent skills, and a verifier that fails the production build when any of it breaks. This post is the list of things that verifier has caught, which is a more useful list than the one you get from reading the specification.
Want to know whether an agent can read your site before you invest in fixing it?
Run the free checkerThe four surfaces an agent actually uses
Agent readability gets discussed as one topic, but it is four independent surfaces, and a site can pass three and still be invisible.
| Surface | Question it answers | Failure looks like |
|---|---|---|
| robots.txt | Am I allowed to fetch this at all? | Silent absence from answers |
| Served HTML | Is the text here before JavaScript runs? | An empty shell with navigation |
| Markdown mirror | Is there a cheap, unambiguous copy? | Expensive, noisy parsing of rendered chrome |
| JSON-LD | Who published this and what is it? | Facts guessed from prose, or nothing |
The order matters. Fixing structured data on a page that a retrieval fetcher is disallowed from reading is wasted work, and that is the most common way this project gets started backwards.
llms.txt, honestly
llms.txt is a convention, not a standard. No engine is obliged to read it, and anyone selling you guaranteed pickup is overselling. It is also cheap to generate and it gives an agent a clean map instead of your rendered navigation, which is why we publish one and usually recommend one.
If you publish it, three things decide whether it is useful:
- Absolute URLs. This is the mistake we see most, and we made it ourselves. An llms.txt gets fetched and passed around detached from the page it came from, so there is no base URL left to resolve relative links against. A file full of
/services/…paths is a file full of dead ends. - The blockquote under the H1. That one sentence is what an agent is most likely to reuse verbatim when it introduces you. Leave it out and the agent writes that sentence itself, from whatever it inferred.
- Notes on every link. The
: noteshalf of each bullet is how an agent with a limited budget decides which link to open. A bare list of titles makes it guess.
Markdown mirrors and how to advertise them
A mirror is the same content as the page, minus the chrome, served as Markdown at a predictable path and pointed to from the page head:
<link rel="alternate" type="text/markdown" href="/services/ai-visibility.md">Generating mirrors is straightforward. Keeping them faithful is the part that needs a machine, because a mirror can be subtly wrong in ways nobody notices for months. The generator runs after the site build, walks the rendered HTML, and the verifier then compares the two.
The failure modes worth knowing about
These are real findings from our own build, not hypotheticals. Each one shipped at least once before the gate existed.
| Failure | Why it happens | What an agent sees |
|---|---|---|
| Unresolved template value | A generator emits its no-value sentinel, or a pair of template delimiters survives unrendered, and nobody reads the output | A page that quotes your templating language back at you |
| Unbalanced code fence | An opening fence with no closing one | Every heading after it stops being a heading, so the document loses its structure |
| Unresolved HTML entity | Entities not decoded during conversion | An undecoded ampersand or apostrophe entity, read as its literal characters rather than as the punctuation |
| Collapsed adjacent links | Whitespace between two anchors dropped in conversion | Two link texts glued into one phrase |
| Attribution glued to its link | A missing space before an author link | An author string where the word before the link is fused to the name |
| Table column mismatch | Header row and delimiter row disagree on column count | The table stops parsing as a table, so every number loses its column |
| More than one H1 | Chrome headings leaking into the body | Ambiguity about what the page is even about |
| Empty body | Content injected client-side, so the mirror has nothing to mirror | Front matter and silence |
Our favourite was subtler than any of these. Adding one decorative glyph to a page put a literal double-quote character inside an SVG text node, which stopped the HTML minifier inside foreign content. The rest of that page shipped unminified, and its Markdown mirror silently truncated at two thirds, taking the entire FAQ with it. Nothing looked wrong in a browser. The gate failed the build, named the route, and the fix was one character.
The robots.txt trap: retrieval is not training
This is the single most expensive misunderstanding in the topic, and it is one line of configuration.
Some crawlers exist to collect training data. Others fetch a page in order to answer a question and cite it, right now. Blocking the first group is a licensing decision you may well want to make. Blocking the second group removes you from answers entirely, and it is almost always unintentional:
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /The first block is a deliberate training opt-out. The second one takes every answer-engine fetcher down with it, because a crawler with no group of its own inherits the wildcard. The team that wrote this believed they had opted out of training. They had also opted out of being cited.
If you want the position "cite me, but do not train on me", that is coherent and configurable: name the retrieval fetchers explicitly and let them through, and disallow the training crawlers by name.
Why a gate, and not a checklist
Every item above is easy to fix once and impossible to keep fixed by intention. The content changes weekly, the templates change monthly, and none of these failures produces a visible symptom. A quarterly audit finds them a quarter late.
So the checks belong in the build, next to the tests. Ours run after the site is generated and fail the deploy, which means a broken mirror is a red pipeline rather than a slow leak. That is the whole trick, and it is why we hand people the rule catalog instead of a report: the agent readability checker runs the same mirror rules in your browser, and it is the same code path that decides whether this site deploys.
For the neighbouring problems, our Open Knowledge Format guide covers packaging internal knowledge as portable Markdown, and the AI-ready company wiki guide covers serving that knowledge to your own agents. This post is strictly about the public surface: what somebody else's agent can read.
Frequently Asked Questions
What makes a website agent-readable?
Is llms.txt a standard?
Do relative URLs work in llms.txt?
Should we block GPTBot?
Do Markdown mirrors duplicate content for search engines?
How do we stop this decaying after launch?
Final thoughts
Agent readability is not a content project. It is four mechanical surfaces, a short list of conversion bugs, and one configuration line that decides whether any of the rest matters.
Start with robots.txt, because it is the cheapest check and the most expensive mistake. Then serve a clean copy, advertise it, validate your structured data, and put all of it behind a gate so the next deploy has to keep it true.