# Making a Next.js Site Agent-Readable

## Seven files, no middleware, and which half of it the evidence supports

> Serving a markdown twin of every page cuts what an agent has to parse by 92%. Serving an llms.txt may do nothing at all. Here is the code for both, and the evidence for each.

Source: https://mrova.rocks/blog/agent-readable-nextjs-site
Published: 2026-08-27
Author: Sunny Luthra · mRova (https://mrova.rocks)
Tags: AI, Agents, Engineering

---

Two readers now arrive at every URL you publish. One of them lands, scans, clicks something,
and reads. The other fetches a single URL, receives eighteen kilobytes of markup wrapped around
two kilobytes of words, pays for every token of it, and then decides whether you were worth the
trouble.

The second reader is new enough that most sites have never been designed for it, and it is
already making recommendations about you.

Christopher Burns makes the sharpest available case for taking this seriously. He founded
[Inth](https://inth.com) and maintains [c15t](https://c15t.com), an open-source consent-management
library doing over half a million npm downloads a month. Speaking at the AI Engineer World's Fair
in July 2026, he reported that since April, the single largest answer to "how did you hear about
us?" on their onboarding form has been *an LLM told me to install it*
([2:23](https://www.youtube.com/watch?v=V_5bn4q-vAI&t=143)).

That is a version of the old Collison installation — the founder who shows up and installs the
product for you on the spot — except the founder has been replaced by a prompt, and the prompt
only recommends what it can cheaply read.

We rebuilt this site around that idea, measured what it saved, and found that roughly half of the
standard advice on the subject has real evidence behind it and the other half does not. Both
halves are below, with the code.

## What an agent actually does

The instinct is to imagine a crawler moving through your site the way a person does. It does not.
It performs one fetch, and whatever comes back is what it knows about you.

**The asymmetry is the whole argument. One reader explores; the other gets one shot and pays by the token.**

*What a person does*
01. Lands — on any page, from anywhere
02. Scans — headings, images, nav
03. Clicks through — follows what looks relevant
04. Reads — and comes back later

*What an agent does*
01. Fetches one URL — usually the one it was handed
02. Parses markup it did not want — scripts, styles, nav, footer
03. Answers, or gives up — no second visit

There is no click-through. There is no second visit. Everything your site is going to say to that
reader has to survive one request, and every byte of layout in that response is a byte it paid for
and discarded.

## The seven artifacts

Burns' pipeline slide ([5:20](https://www.youtube.com/watch?v=V_5bn4q-vAI&t=320)) fans one markdown
source out into eight machine-readable surfaces. Most sites already ship two of them without
thinking about it.

**Nine surfaces, six of them new here. Items 10 and 11 are deliberate omissions, not oversights.**

*Already shipped, by most sites*
01. robots.txt — who may read, and what
02. sitemap.xml — every URL, with dates
03. feed.xml — what changed, in order

*What this post ships*
04. The markdown twin — every article, without the layout
05. Three doors to it — suffix, header, query param
06. The head hint — so the page announces the twin
07. llms.txt — hand-written positioning
08. llms-full.txt — the whole corpus, one fetch
09. AGENTS.md — for agents inside the repo

*Not built, and why is in the last section*
10. WebMCP — letting an agent query the site
11. Package bundling — docs shipped inside node_modules

Only items 4 through 9 need writing, and one of them is a text file you type by hand.

## Step 1 — the markdown twin

The centrepiece, and on a site whose articles are already markdown on disk, the cheapest thing on
the list. The hard part of Burns' version — extracting a markdown source back out of a rendered
docs site — is a file read here.

What the twin has to add is provenance. A `.md` file fetched in isolation is an anonymous block of
text that cannot be cited back to anyone, so the emitter prepends what the body cannot carry:

```ts
// lib/agent-markdown.ts
export function postToMarkdown(post: Post): string {
  const url = `${SITE.url}/blog/${post.slug}`;

  const header = [
    `# ${post.title}`,
    `## ${post.subtitle}`,
    `> ${post.description}`,
    `Source: ${url}`,
    `Published: ${post.date}`,
    `Author: ${post.author.name} · ${SITE.name} (${SITE.url})`,
    `---`,
  ].join("\n\n");

  return `${header}\n\n${flattenDiagramFigures(post.content.trim())}\n`;
}
```

That `flattenDiagramFigures` call is the one place the twin is not a byte-for-byte copy of the
source. Figures like the two above are raw HTML inside the markdown, and they are the only part of
a post where the source is not already the cheapest representation of itself — the diagram in our
[harness engineering](/blog/harness-engineering) piece spends 2.1KB of markup on about sixty words.
Flattening turns it into an ordered list that keeps every label, name and note.

The rule that matters when you write your own version: **anything that does not match the expected
shape passes through untouched.** A figure that renders a bit oddly for an agent is a much better
outcome than one silently mangled by a regex that almost matched.

Serving it is a route handler. The only structural constraint is that a Next.js directory cannot
hold both `[slug]/page.tsx` and `[slug].md/route.ts`, so the twin lives one segment below the page:

```ts
// app/blog/[slug]/md/route.ts
export function generateStaticParams() {
  return getPostSlugs().map((slug) => ({ slug }));
}

export const dynamic = "force-static";
export const dynamicParams = false;

export async function GET(_request: Request, { params }: Params) {
  const { slug } = await params;
  const post = getPost(slug);
  if (!post) return new Response("Not found\n", { status: 404 });

  return new Response(postToMarkdown(post), { headers: MARKDOWN_HEADERS });
}
```

`force-static` plus `generateStaticParams` means every twin prerenders at build time alongside its
page. There is no request-time work anywhere in this system, which is worth insisting on: the point
is to be cheap to read, and a feature that makes your site slower to serve in order to be cheaper to
parse has traded the wrong way.

## Step 2 — three doors, no middleware

An agent should not have to know that `/blog/<slug>/md` is where the file lives. It will try
whatever seems natural, and there are three natural things to try.

**Three rewrites in next.config.ts. The page keeps its URL and a person never sees any of this.**

*Three ways an agent asks*
01. /blog/slug.md — the suffix — guessable, pasteable
02. Accept: text/markdown — for clients that can set headers
03. ?mode=agent — for the many that cannot

*One place they land*
→. /blog/slug/md — one handler, prerendered, no runtime cost

The conventional advice here is middleware, and middleware is the wrong tool. It runs on the edge
runtime, so it cannot touch the filesystem to read your posts — meaning it can only rewrite anyway
— and it adds a function invocation to every matching request forever. Next's `rewrites()` covers
all three cases with no runtime at all:

```ts
// next.config.ts
async rewrites() {
  const markdownTwin = "/blog/:slug/md";

  return {
    beforeFiles: [
      { source: "/blog/:slug.md", destination: markdownTwin },
      {
        source: "/blog/:slug",
        has: [{ type: "query", key: "mode", value: "agent" }],
        destination: markdownTwin,
      },
      {
        source: "/blog/:slug",
        // Substring, not equality: a real `Accept` is a q-weighted list.
        has: [{ type: "header", key: "accept", value: ".*text/markdown.*" }],
        destination: markdownTwin,
      },
    ],
  };
}
```

Two details are load-bearing, and both cost a build to discover.

**`beforeFiles`, not `afterFiles`.** `/blog/<slug>` is a real page. Rewrites in `afterFiles` are
evaluated only when nothing on the filesystem matched, so the header and query rules would never
fire once. The page always wins.

**The `Accept` value is a substring match.** A real `Accept` header is a q-weighted list —
a browser sends something like `text/html,application/xhtml+xml,*/*;q=0.8`. Matching on equality
means matching almost nobody. Matching on `.*text/markdown.*` catches every client that names the
type anywhere in its list, and never catches a browser, which does not name it at all.

Then the twin sends `Vary: Accept` back, because the page and the file are now two representations
of one resource. Without it, a shared cache will eventually hand an agent the HTML it explicitly
asked not to receive.

## Step 3 — the head hint

The doors only help an agent that already suspects markdown exists. This is the line that tells one
holding the rendered page that a cheaper version is available:

```ts
// app/blog/[slug]/page.tsx
alternates: {
  canonical: url,
  types: { "text/markdown": markdownTwinUrl(post.slug) },
},
```

Which emits `<link rel="alternate" type="text/markdown" href="…/slug.md">`. Four lines, and it is
the only part of the system that is discoverable from the page a person sees.

One thing to get right alongside it: disallow the twin's real path in `robots.txt`. The twin and the
page are the same words at two URLs, and only one of them should be competing for a search result.

## Step 4 — llms.txt, written by hand

`llms.txt` is a markdown file at your root that says what your site is and where to start. The
temptation is to generate it. Burns' rule is the opposite, and it is right:
**write it by hand — forty good lines beat a thousand lines of noise**
([5:52](https://www.youtube.com/watch?v=V_5bn4q-vAI&t=352)).

It is positioning, not a sitemap. You already have a sitemap. The shape c15t uses on its
[live file](https://c15t.com/llms.txt) is worth copying outright:

```md
# Your company
> One sentence. What this is, for whom.

## What it is
## What we have shipped
## Who it is for
## Technical range
## How to reach us
## Machine-readable interfaces
## Best starting points
```

Three of those headings do the work. *What we have shipped* is where the verifiable claims go —
named, with numbers you can defend. *Machine-readable interfaces* points at the markdown twin,
`llms-full.txt` and the feed. *Best starting points* is about seven links, each with one line on
what it is for.

Only the link index at the bottom should be generated, because it is the part that goes stale.
Everything above it is prose you review.

The discipline that matters here is that this file is a machine-readable claim about your business.
An unchecked number in an `llms.txt` is not marketing copy — it is a fabrication with a file
extension, quoted back by something that will not caveat it. Every figure in ours traces to
something we can show.

`llms-full.txt` is the generated counterpart: every article in full, in one file. For a nine-post
site it is 62KB, and it turns nine fetches into one.

## The part nobody wants to write

Here is where most posts on this topic stop, having told you to add an `llms.txt` and implied that
something good will follow. The evidence does not support that, and pretending otherwise is how
this whole area turns into cargo cult.

- Google's John Mueller has stated that no AI system currently uses `llms.txt`, and Gary Illyes
  confirmed at Search Central Live that Google does not support it and has no plans to
  ([Search Engine Roundtable](https://www.seroundtable.com/google-ai-llms-txt-39607.html)).
- Google's own AI-optimisation guidance, updated June 2026, says plainly: *"You don't need to
  create new machine readable files, AI text files, markup, or Markdown to appear in Google
  Search."*
- An Ahrefs study of 137,000 sites reported that **97% of `llms.txt` files received zero traffic**
  in May 2026.

Set against that, every number Burns actually measures is about **markdown an agent fetches or reads
off disk** — not about a discovery manifest. His benchmark
([11:40](https://www.youtube.com/watch?v=V_5bn4q-vAI&t=700)) ran the same package with and without
bundled docs, n=10 per cell with a neutral LLM judge, and found token reductions from 16% to 50%
across five models. His most striking result — a bundle read-rate going from 29% to 90–100% — came
from adding a single pointer line to an `AGENTS.md`.

So the honest split:

| Artifact | Who reads it today | Evidence | Ship it? |
|---|---|---|---|
| The `.md` twin | Coding agents and assistant crawlers, on fetch | −16% to −50% tokens across 5 models (Burns, n=10/cell) | Yes |
| `Accept:` negotiation | Any client that can set a header | Adopted by Vercel, Mintlify, c15t | Yes — it is four lines |
| `AGENTS.md` in a package | Coding agents, off disk in `node_modules` | 29% → 90–100% read rate from one pointer line | If you publish a package |
| `AGENTS.md` in a repo | Coding agents working in your codebase | No public study; obviously useful | Yes |
| `llms.txt` | Perplexity, Claude, some coding agents | 97% of 137k files got zero traffic (Ahrefs, May 2026); Google uses none | Yes — cheap, but do not bill it as the win |
| WebMCP | Almost nobody, yet | Early; c15t ships it | Later |

**Measured: serving markdown an agent can fetch or read locally.** **Speculative: `llms.txt` as a
discovery manifest.** Ship both — the second one costs a single route — but only one of them has
earned a claim.

## What it actually saved

Nine articles, production build, measured on this site. Token counts estimated at four characters
per token. "HTML → text" is the rendered page after stripping scripts, styles, SVG and tags —
roughly what an agent with a decent extractor ends up holding.

| Post | HTML | HTML → text | Markdown | vs HTML | vs text |
|---|---|---|---|---|---|
| agent-verification-receipts | 18,697 | 1,740 | 1,418 | −92% | −19% |
| ai-initiatives-no-benefit | 18,694 | 1,775 | 1,515 | −92% | −15% |
| ai-shift-in-dentistry | 17,907 | 1,232 | 1,024 | −94% | −17% |
| boarding-os-for-india | 19,118 | 2,014 | 1,670 | −91% | −17% |
| context-engineering-vs-rag | 18,848 | 1,788 | 1,476 | −92% | −17% |
| fedramp-2026-rules | 18,289 | 1,533 | 1,208 | −93% | −21% |
| harness-engineering | 24,416 | 3,100 | 2,893 | −88% | −7% |
| hipaa-architecture-startup-budget | 19,045 | 1,960 | 1,618 | −92% | −17% |
| whatsapp-to-operating-system | 19,243 | 2,044 | 1,718 | −91% | −16% |
| **All nine** | **174,257** | **17,186** | **14,540** | **−92%** | **−15%** |

**That is two numbers, and publishing only the first would be dishonest.** An agent that ingests
raw HTML pays roughly twelve times over, and the twin saves it 92%. An agent that runs a good
extractor first has already recovered most of that on its own, and the twin saves it a further 15%.
Which number applies to you depends entirely on which client is reading, and you do not control that.

Most writing on this subject quotes the first number alone. It is technically correct and leaves
the reader with a wrong impression of the size of the win.

The corpus-level number is the more interesting one anyway:

| Route | Fetches | Tokens (est.) |
|---|---|---|
| Nine HTML pages | 9 | 174,257 |
| `/llms-full.txt` | 1 | 15,630 |
| `/llms.txt` | 1 | 1,851 |

One fetch instead of nine, at nine percent of the cost.

And the −7% outlier on `harness-engineering` is the figure-flattening working exactly as intended:
that post carries the most markup per word on the site, so its extracted text was already close to
minimal before we touched it.

## Scoring it

Two public scanners will grade a site on this axis. [ora.ai](https://ora.ai) scores out of 100
across discovery, access, usability and payments, and it does it by spawning real agent sessions
against your URL rather than reading your HTML. Vercel's free "Is Agentic" tool, released in August
2026, runs the same check set.

A caution learned the expensive way: **run the baseline scan before you ship anything.** Ours was
triggered the same afternoon the implementation landed, and because Ora spawns real sessions it
takes minutes rather than seconds — by the time it returned there was nothing left to measure it
against. The before/after is the only proof that any of this moved a number, and it is not
recoverable after the fact.

## If you ship a package, do this first

The strongest technique in Burns' talk is one a marketing site cannot use, and it deserves saying
plainly: coding agents mostly do not visit your documentation site at all. They read the repository
they are working in and the packages already sitting in `node_modules`
([10:09](https://www.youtube.com/watch?v=V_5bn4q-vAI&t=609)).

So if you publish a package, the highest-leverage file you can ship is an `AGENTS.md` and a
`docs/*.md` folder **inside the published tarball**, where an agent finds them on disk without a
network request. That is where his 29% → 90–100% number comes from.

We publish no package, so the adjacent version is a root `AGENTS.md` in this repository — for the
coding agents that work in it and were otherwise guessing at conventions already written down
somewhere else. Different mechanism, same principle: put the instructions where the reader already
is.

## What we did not build

WebMCP — a `/.well-known/mcp.json` and an `/ask` endpoint that would let an agent query the site
rather than fetch from it — is the obvious next step and we have not shipped it. Burns is candid
that it is early ([9:17](https://www.youtube.com/watch?v=V_5bn4q-vAI&t=557)), and c15t shipping it
is close to the whole of the current adoption story.

Early is a reason to sequence it last, not to skip it. Everything above works today for readers
that exist today, which is the correct order to build things in.

His closing line is the right one to end on: the ground here moves weekly, and nothing you build
stays perfect ([13:46](https://www.youtube.com/watch?v=V_5bn4q-vAI&t=826)). Which is an argument for
dating your claims rather than hedging them. Everything above was true, built and measured in
August 2026. The markdown twin will still be cheap to read in a year. Whether anything reads the
`llms.txt` is a different question, and we will tell you when we know.

You can read this article the way an agent would: append `.md` to this URL, or add `?mode=agent`,
or send `Accept: text/markdown`.

And this particular post is the weakest case on the whole site for the technique it is arguing for.
Its HTML is 133KB and its twin is 20KB — a 85% saving against raw markup. But it is mostly prose,
code blocks and tables, with barely any decorative structure, so an agent running a good extractor
gets to 20.4KB on its own. Against that, the twin saves **2%**.

Which is the right number to end on. The win is real and it is largest where your pages are
heaviest, and if you go looking for it on a page that is already almost all words, you will not
find much. Measure your own site rather than ours.

---

## Frequently asked questions

### What is an llms.txt file and does it work?

It is a hand-written markdown manifest at your site root that tells an agent what your site is and where the good pages are. Whether it works depends on the reader: Google states that no part of Search uses it and that it has no plans to, while Perplexity, Claude and several coding agents do retrieve it. It costs one route to ship, so ship it — but do not expect it to be the thing that moves.

### How do I serve markdown instead of HTML in Next.js?

Put a route handler at a path the page does not occupy, have it read your markdown source and return it as text/markdown, then point the guessable URLs at it with rewrites in next.config.ts. You do not need middleware, and avoiding it keeps the whole thing static — every markdown twin prerenders at build time alongside its page.

### Do AI agents actually read my documentation site?

Coding agents mostly do not. As Christopher Burns put it at the AI Engineer World's Fair, they read the repository they are working in and the packages already on disk in node_modules. That is why a file bundled into your published package outperforms a beautiful docs site for that particular reader.

### What is the difference between AGENTS.md and llms.txt?

They address different readers. AGENTS.md is for a coding agent already inside a codebase and needs to state conventions it would otherwise guess. llms.txt is a discovery manifest at a public URL, for a crawler or assistant that has never seen your site. Shipping one does not substitute for the other.