Making a Next.js Site Agent-Readable

Two readers now arrive at every URL you publish. One of them lands, scans, clicks something, and reads. The other fetches a single URL, receives eighteen kilobytes of markup wrapped around two kilobytes of words, pays for every token of it, and then decides whether you were worth the trouble.

The second reader is new enough that most sites have never been designed for it, and it is already making recommendations about you.

Christopher Burns makes the sharpest available case for taking this seriously. He founded Inth and maintains c15t, an open-source consent-management library doing over half a million npm downloads a month. Speaking at the AI Engineer World's Fair in July 2026, he reported that since April, the single largest answer to "how did you hear about us?" on their onboarding form has been an LLM told me to install it (2:23).

That is a version of the old Collison installation — the founder who shows up and installs the product for you on the spot — except the founder has been replaced by a prompt, and the prompt only recommends what it can cheaply read.

We rebuilt this site around that idea, measured what it saved, and found that roughly half of the standard advice on the subject has real evidence behind it and the other half does not. Both halves are below, with the code.

What an agent actually does

The instinct is to imagine a crawler moving through your site the way a person does. It does not. It performs one fetch, and whatever comes back is what it knows about you.

There is no click-through. There is no second visit. Everything your site is going to say to that reader has to survive one request, and every byte of layout in that response is a byte it paid for and discarded.

The seven artifacts

Burns' pipeline slide (5:20) fans one markdown source out into eight machine-readable surfaces. Most sites already ship two of them without thinking about it.

Only items 4 through 9 need writing, and one of them is a text file you type by hand.

Step 1 — the markdown twin

The centrepiece, and on a site whose articles are already markdown on disk, the cheapest thing on the list. The hard part of Burns' version — extracting a markdown source back out of a rendered docs site — is a file read here.

What the twin has to add is provenance. A .md file fetched in isolation is an anonymous block of text that cannot be cited back to anyone, so the emitter prepends what the body cannot carry:

// lib/agent-markdown.ts
export function postToMarkdown(post: Post): string {
  const url = `${SITE.url}/blog/${post.slug}`;

  const header = [
    `# ${post.title}`,
    `## ${post.subtitle}`,
    `> ${post.description}`,
    `Source: ${url}`,
    `Published: ${post.date}`,
    `Author: ${post.author.name} · ${SITE.name} (${SITE.url})`,
    `---`,
  ].join("\n\n");

  return `${header}\n\n${flattenDiagramFigures(post.content.trim())}\n`;
}

That flattenDiagramFigures call is the one place the twin is not a byte-for-byte copy of the source. Figures like the two above are raw HTML inside the markdown, and they are the only part of a post where the source is not already the cheapest representation of itself — the diagram in our harness engineering piece spends 2.1KB of markup on about sixty words. Flattening turns it into an ordered list that keeps every label, name and note.

The rule that matters when you write your own version: anything that does not match the expected shape passes through untouched. A figure that renders a bit oddly for an agent is a much better outcome than one silently mangled by a regex that almost matched.

Serving it is a route handler. The only structural constraint is that a Next.js directory cannot hold both [slug]/page.tsx and [slug].md/route.ts, so the twin lives one segment below the page:

// app/blog/[slug]/md/route.ts
export function generateStaticParams() {
  return getPostSlugs().map((slug) => ({ slug }));
}

export const dynamic = "force-static";
export const dynamicParams = false;

export async function GET(_request: Request, { params }: Params) {
  const { slug } = await params;
  const post = getPost(slug);
  if (!post) return new Response("Not found\n", { status: 404 });

  return new Response(postToMarkdown(post), { headers: MARKDOWN_HEADERS });
}

force-static plus generateStaticParams means every twin prerenders at build time alongside its page. There is no request-time work anywhere in this system, which is worth insisting on: the point is to be cheap to read, and a feature that makes your site slower to serve in order to be cheaper to parse has traded the wrong way.

Step 2 — three doors, no middleware

An agent should not have to know that /blog/<slug>/md is where the file lives. It will try whatever seems natural, and there are three natural things to try.

The conventional advice here is middleware, and middleware is the wrong tool. It runs on the edge runtime, so it cannot touch the filesystem to read your posts — meaning it can only rewrite anyway — and it adds a function invocation to every matching request forever. Next's rewrites() covers all three cases with no runtime at all:

// next.config.ts
async rewrites() {
  const markdownTwin = "/blog/:slug/md";

  return {
    beforeFiles: [
      { source: "/blog/:slug.md", destination: markdownTwin },
      {
        source: "/blog/:slug",
        has: [{ type: "query", key: "mode", value: "agent" }],
        destination: markdownTwin,
      },
      {
        source: "/blog/:slug",
        // Substring, not equality: a real `Accept` is a q-weighted list.
        has: [{ type: "header", key: "accept", value: ".*text/markdown.*" }],
        destination: markdownTwin,
      },
    ],
  };
}

Two details are load-bearing, and both cost a build to discover.

beforeFiles, not afterFiles. /blog/<slug> is a real page. Rewrites in afterFiles are evaluated only when nothing on the filesystem matched, so the header and query rules would never fire once. The page always wins.

The Accept value is a substring match. A real Accept header is a q-weighted list — a browser sends something like text/html,application/xhtml+xml,*/*;q=0.8. Matching on equality means matching almost nobody. Matching on .*text/markdown.* catches every client that names the type anywhere in its list, and never catches a browser, which does not name it at all.

Then the twin sends Vary: Accept back, because the page and the file are now two representations of one resource. Without it, a shared cache will eventually hand an agent the HTML it explicitly asked not to receive.

Step 3 — the head hint

The doors only help an agent that already suspects markdown exists. This is the line that tells one holding the rendered page that a cheaper version is available:

// app/blog/[slug]/page.tsx
alternates: {
  canonical: url,
  types: { "text/markdown": markdownTwinUrl(post.slug) },
},

Which emits <link rel="alternate" type="text/markdown" href="…/slug.md">. Four lines, and it is the only part of the system that is discoverable from the page a person sees.

One thing to get right alongside it: disallow the twin's real path in robots.txt. The twin and the page are the same words at two URLs, and only one of them should be competing for a search result.

Step 4 — llms.txt, written by hand

llms.txt is a markdown file at your root that says what your site is and where to start. The temptation is to generate it. Burns' rule is the opposite, and it is right: write it by hand — forty good lines beat a thousand lines of noise (5:52).

It is positioning, not a sitemap. You already have a sitemap. The shape c15t uses on its live file is worth copying outright:

# Your company
> One sentence. What this is, for whom.

## What it is
## What we have shipped
## Who it is for
## Technical range
## How to reach us
## Machine-readable interfaces
## Best starting points

Three of those headings do the work. What we have shipped is where the verifiable claims go — named, with numbers you can defend. Machine-readable interfaces points at the markdown twin, llms-full.txt and the feed. Best starting points is about seven links, each with one line on what it is for.

Only the link index at the bottom should be generated, because it is the part that goes stale. Everything above it is prose you review.

The discipline that matters here is that this file is a machine-readable claim about your business. An unchecked number in an llms.txt is not marketing copy — it is a fabrication with a file extension, quoted back by something that will not caveat it. Every figure in ours traces to something we can show.

llms-full.txt is the generated counterpart: every article in full, in one file. For a nine-post site it is 62KB, and it turns nine fetches into one.

The part nobody wants to write

Here is where most posts on this topic stop, having told you to add an llms.txt and implied that something good will follow. The evidence does not support that, and pretending otherwise is how this whole area turns into cargo cult.

• Google's John Mueller has stated that no AI system currently uses llms.txt, and Gary Illyes confirmed at Search Central Live that Google does not support it and has no plans to (Search Engine Roundtable).
• Google's own AI-optimisation guidance, updated June 2026, says plainly: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search."
• An Ahrefs study of 137,000 sites reported that 97% of llms.txt files received zero traffic in May 2026.

Set against that, every number Burns actually measures is about markdown an agent fetches or reads off disk — not about a discovery manifest. His benchmark (11:40) ran the same package with and without bundled docs, n=10 per cell with a neutral LLM judge, and found token reductions from 16% to 50% across five models. His most striking result — a bundle read-rate going from 29% to 90–100% — came from adding a single pointer line to an AGENTS.md.

So the honest split:

Measured: serving markdown an agent can fetch or read locally. Speculative: llms.txt as a discovery manifest. Ship both — the second one costs a single route — but only one of them has earned a claim.

What it actually saved

Nine articles, production build, measured on this site. Token counts estimated at four characters per token. "HTML → text" is the rendered page after stripping scripts, styles, SVG and tags — roughly what an agent with a decent extractor ends up holding.

That is two numbers, and publishing only the first would be dishonest. An agent that ingests raw HTML pays roughly twelve times over, and the twin saves it 92%. An agent that runs a good extractor first has already recovered most of that on its own, and the twin saves it a further 15%. Which number applies to you depends entirely on which client is reading, and you do not control that.

Most writing on this subject quotes the first number alone. It is technically correct and leaves the reader with a wrong impression of the size of the win.

The corpus-level number is the more interesting one anyway:

One fetch instead of nine, at nine percent of the cost.

And the −7% outlier on harness-engineering is the figure-flattening working exactly as intended: that post carries the most markup per word on the site, so its extracted text was already close to minimal before we touched it.

Scoring it

Two public scanners will grade a site on this axis. ora.ai scores out of 100 across discovery, access, usability and payments, and it does it by spawning real agent sessions against your URL rather than reading your HTML. Vercel's free "Is Agentic" tool, released in August 2026, runs the same check set.

A caution learned the expensive way: run the baseline scan before you ship anything. Ours was triggered the same afternoon the implementation landed, and because Ora spawns real sessions it takes minutes rather than seconds — by the time it returned there was nothing left to measure it against. The before/after is the only proof that any of this moved a number, and it is not recoverable after the fact.

If you ship a package, do this first

The strongest technique in Burns' talk is one a marketing site cannot use, and it deserves saying plainly: coding agents mostly do not visit your documentation site at all. They read the repository they are working in and the packages already sitting in node_modules (10:09).

So if you publish a package, the highest-leverage file you can ship is an AGENTS.md and a docs/*.md folder inside the published tarball, where an agent finds them on disk without a network request. That is where his 29% → 90–100% number comes from.

We publish no package, so the adjacent version is a root AGENTS.md in this repository — for the coding agents that work in it and were otherwise guessing at conventions already written down somewhere else. Different mechanism, same principle: put the instructions where the reader already is.

What we did not build

WebMCP — a /.well-known/mcp.json and an /ask endpoint that would let an agent query the site rather than fetch from it — is the obvious next step and we have not shipped it. Burns is candid that it is early (9:17), and c15t shipping it is close to the whole of the current adoption story.

Early is a reason to sequence it last, not to skip it. Everything above works today for readers that exist today, which is the correct order to build things in.

His closing line is the right one to end on: the ground here moves weekly, and nothing you build stays perfect (13:46). Which is an argument for dating your claims rather than hedging them. Everything above was true, built and measured in August 2026. The markdown twin will still be cheap to read in a year. Whether anything reads the llms.txt is a different question, and we will tell you when we know.

You can read this article the way an agent would: append .md to this URL, or add ?mode=agent, or send Accept: text/markdown.

And this particular post is the weakest case on the whole site for the technique it is arguing for. Its HTML is 133KB and its twin is 20KB — a 85% saving against raw markup. But it is mostly prose, code blocks and tables, with barely any decorative structure, so an agent running a good extractor gets to 20.4KB on its own. Against that, the twin saves 2%.

Which is the right number to end on. The win is real and it is largest where your pages are heaviest, and if you go looking for it on a page that is already almost all words, you will not find much. Measure your own site rather than ours.

• AI
• Agents
• Engineering