Axortex

The culture of tech, food & beauty

← Tech
✦Tech

Writing for the Machines That Read You: A Field Guide to llms.txt, robots.txt, and JSON-LD

Your audience now includes crawlers and language models. Three low-effort files — an llms.txt index, an AI-welcoming robots.txt, and JSON-LD structured data — make a blog legible to them. Here is the copy-paste stack, with the honest caveats.

A Axortex ·

TL;DR — Three low-effort files make your site easier for AI to read and cite: a /llms.txt index of your best content, a robots.txt that explicitly welcomes AI crawlers and points to your sitemap, and JSON-LD structured data (BlogPosting, Organization, BreadcrumbList) on every page. None of them are magic, but together they remove friction. This very blog ships all three.

For most of the web's life, the readers worth designing for were two kinds: humans, and the search engine deciding which humans to send. That list has quietly grown. ChatGPT, Claude, Perplexity, and Google's own AI answers now fetch your pages, parse them, and — increasingly — quote them back to someone who never visits your site. These readers are fast, literal, and easily annoyed. Bury your writing in cluttered markup with no machine-readable signals and you are not hiding from them so much as making their job harder, which tends to mean getting cited less. The good news is that meeting them halfway costs almost nothing. What follows is the practical stack, caveats included.

A curated map for language models

The first file is the newest idea. llms.txt is a convention proposed by Jeremy Howard of Answer.AI: a single Markdown file at your site root (/llms.txt) that hands a language model a clean, curated index of your most important content, so it does not have to crawl and de-clutter your entire site just to understand what you are about.

The format is short and strict, and it runs in this order:

  1. An # H1 with the site name (the only required line)
  2. A > blockquote summary
  3. Optional prose
  4. ## H2 sections, each a list of links: - [name](url): note

A minimal example:

# Terapep

> A blog about technology and good food — practical notes, honest takes, and recipes worth keeping.

## Posts

- [How Korean Ramyeon Took Over the World](https://terapep.com/blog/korean-ramyeon-global-boom/): The export data behind K-ramen's rise.
- [Does GEO Actually Work?](https://terapep.com/blog/does-geo-actually-work/): What the research really says about AI-search optimization.

There is a heavier sibling, llms-full.txt, which inlines your entire content into one large Markdown file so a model can swallow everything in a single fetch. Watch the size, though — on a big site it can blow a context window.

And here is the caveat to internalize before you over-invest: llms.txt is "mainly useful for inference" — that is, when an LLM is actively helping a user — not training, and no major AI vendor has committed to using it as a ranking input. Google has explicitly said you don't need it for its AI features. Treat it as a low-cost convenience that some tools read, not a guaranteed win.

Telling the crawlers they are welcome

The second file is older and better understood, but the AI era has added nuance. Not all bots want the same thing, and the differences matter:

Class What it does Examples
Training Collects content to train models GPTBot, ClaudeBot, CCBot, Bytespider
Search index Builds the index behind AI answers (sends citations back to you) OAI-SearchBot, PerplexityBot, Claude-SearchBot
Live fetch Reads a page because a user asked right now ChatGPT-User, Perplexity-User

If you want AI visibility — and most blogs do, because the search-index bots are the ones that drive citations and referral traffic — name them on purpose:

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://terapep.com/sitemap.xml

Two things are worth knowing here. Google-Extended and Applebot-Extended aren't crawlers — they are control tokens that toggle whether already-crawled content can be used for AI training, with no effect on Search ranking. And robots.txt is advisory: reputable bots honor it, but if you genuinely need to block one (Bytespider has a reputation for ignoring the rules), enforce it at the edge or WAF, not just here. The same three-class table is also your opt-out lever — Disallow: / the training bots while you Allow: / the search-index ones, and you keep the citations without feeding the training sets.

The structured data that still pays

The third file is not a file at all but markup you embed on every page. JSON-LD tells machines what a page is; you drop it in as <script type="application/ld+json">. For a blog, the high-value types in 2026 are BlogPosting, Organization/WebSite, and BreadcrumbList.

A trimmed BlogPosting:

{
  "@context": "https://schema.org",
  "@type": "BlogPosting",
  "headline": "Make Your Blog AI-Readable",
  "description": "A practical guide to llms.txt, robots.txt, and JSON-LD.",
  "image": "https://terapep.com/og.jpg",
  "datePublished": "2026-06-15T09:00:00+09:00",
  "dateModified": "2026-06-15T09:00:00+09:00",
  "author": { "@type": "Organization", "name": "Terapep" },
  "publisher": {
    "@type": "Organization",
    "name": "Terapep",
    "logo": { "@type": "ImageObject", "url": "https://terapep.com/favicon.svg" }
  },
  "mainEntityOfPage": "https://terapep.com/blog/make-your-blog-ai-readable/"
}

Two 2026 reality checks belong on a sticky note. Always include a timezone offset in your dates (+09:00); leave it off and Google assumes Googlebot's timezone. And don't chase FAQPage rich results — Google is sunsetting FAQ rich results for most sites, so the markup stays valid and may still help AI, but the snippet is gone. (A real FAQ section with question-shaped headings still earns its keep — more on that in the GEO post.) Whatever you ship, validate it first with Google's Rich Results Test.

Why this is pleasant in Astro

If you are building on Astro, the whole stack falls into place, because Astro ships static HTML by default — your content lives in the markup with no hydration required, which is exactly what an AI fetcher wants to see. The three pieces map to three small moves:

  • robots.txt: drop a static file in public/, or generate it from a src/pages/robots.txt.ts endpoint so you can inject your site URL.
  • llms.txt: a src/pages/llms.txt.ts endpoint that iterates your posts and returns text/plain.
  • JSON-LD: a small component fed by frontmatter, rendered once per page — site-wide Organization/WebSite in your base head, per-post BlogPosting + BreadcrumbList in the post layout.

None of it is a silver bullet, and anyone promising you a ranking miracle is selling something. But the friction these files remove is real, and the cost of shipping them is an afternoon. In a web where machines increasingly decide who gets quoted, that is a trade worth making.

FAQ

What is llms.txt and do I need it?

llms.txt is a Markdown index of your site's key content, served at /llms.txt, that helps language models understand your site at inference time. It's a low-cost convenience that some AI tools read, but it is not a ranking signal, and Google says you don't need it.

How do I let AI crawlers read my site?

Add explicit User-agent / Allow: / blocks in robots.txt for the bots you want (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, etc.) and include a Sitemap: line. Remember robots.txt is advisory — enforce hard blocks at the edge.

Does FAQ schema still work in 2026?

FAQ rich results are being retired by Google for most sites, so you won't get the snippet. The JSON-LD is still valid markup, and a real FAQ section (question-shaped headings + self-contained answers) still helps AI comprehension.

Sources: llmstxt.org, Google AI optimization guide, Google: web publisher controls, Schema.org Article docs, Search Engine Land.

Image: Markus Spiske, CC0 / public domain, via Wikimedia Commons.

#llms-txt#seo#schema#ai#astro
← Back to all posts