What Is llms.txt and How Do You Create One? With a Copy-Paste Example
llms.txt is a file that tells AI crawlers, in markdown, what your site is and which pages matter most. It takes about half an hour to produce. Full example and validation commands inside.
llms.txt is a file at your site root that tells AI crawlers, in markdown, what your site is and which pages matter most. Think of it as the AI counterpart to sitemap.xml — with one difference: a sitemap is a machine-readable list of URLs, while llms.txt is a human-readable table of contents ordered by priority.
It takes about half an hour to produce. A complete example and the generation logic are below.
Three files, three different jobs
They get conflated. They should not be.
| File | What it says | Format |
|---|---|---|
robots.txt |
"Where you may crawl" | Directives |
sitemap.xml |
"Which URLs exist" | XML, flat list |
llms.txt |
"What this site is, what matters" | Markdown, prioritised |
ai.txt |
"What you may use it for" | Directives |
llms.txt does not replace the others. Your sitemap stays. llms.txt adds a meaning and priority layer: if you have a thousand pages, it says which thirty actually matter.
Why it works
When an AI crawler arrives at your site it has two problems.
Problem one: working out what you are. Your homepage is written in marketing language, the navigation has fifteen links, the hero section has a slogan. The crawler struggles to extract "what exactly does this company do" from that.
Problem two: working out what is important. The sitemap has 400 URLs; your privacy policy and your primary service page carry identical weight.
llms.txt solves both in one file: one paragraph of plain definition, then links grouped in priority order.
Anatomy of the file
The structure is simple, and it is markdown:
# Company or Site Name
> One or two sentences of plain definition. What you do, for whom,
> in which geography. Definition language, not marketing language.
A free-form paragraph — any special notes, language coverage,
technical details like a markdown twin.
## Most Important Section
- [Page name](https://site.com/page): Short description
- [Another page](https://site.com/page2): Short description
## Second Section
- [Page](https://site.com/x): Description
## Optional
- [Less important page](https://site.com/y)
Four rules:
1. Heading grouping is a priority signal. Higher ## sections read as more important. Order them by business priority — services near the top, legal text at the bottom.
2. ## Optional has special meaning. Everything under this heading reads as "look only if needed." Privacy policy, terms of service, archive pages go here.
3. Add a description to every link. A short description after the colon. The crawler understands the page without visiting it.
4. Canonical URLs only. Redirected, noindexed or draft pages do not belong.
Complete example
The structure of this site's llms.txt:
# Algoritma Agency
> A technology agency providing custom software development, AI-supported
> SEO, web design and AI integration training across sectors. Based in
> Türkiye. Content is published in Turkish and English.
This file is prepared for AI search engines and language models.
A markdown version of every page is available by appending .md to the URL.
## Services
- [Services](https://algoritmaajans.com/en/hizmetler): Software development, SEO, web design, AI consulting
- [AI Training](https://algoritmaajans.com/en/egitim): AI for daily work and Vibe Coding training packages
- [About](https://algoritmaajans.com/en/hakkimizda): Agency, team and working approach
- [Contact](https://algoritmaajans.com/en/iletisim): Quotes and project enquiries
## Products
- [All Products](https://algoritmaajans.com/en/urunler): Ready-to-deploy software solutions
- [AI Autonomous Content CRM](https://algoritmaajans.com/en/urunler/ai-icerik-crm)
- [Smart Appointment System](https://algoritmaajans.com/en/urunler/akilli-randevu-sistemi)
- [Online Education Software](https://algoritmaajans.com/en/urunler/online-egitim-yazilimi)
- [News Portal Software](https://algoritmaajans.com/en/urunler/haber-portali-yazilimi)
## Blog
- [Post title](https://algoritmaajans.com/en/blog/slug): Post summary
## Optional
- [Privacy Policy](https://algoritmaajans.com/en/gizlilik)
- [Terms of Service](https://algoritmaajans.com/en/kullanim-sartlari)
- [Turkish version](https://algoritmaajans.com/tr)
Note the ordering: services and products at the top, blog in the middle, legal text under Optional. That sequence is not accidental — it reflects business priority.
llms-full.txt: the full-text version
There is a second file: llms-full.txt. It serves your entire content as plain text in one file, so the crawler does not have to walk page by page.
The logic: to read your blog post normally, a crawler must download HTML and strip out navigation, footer and scripts. With llms-full.txt it all arrives in a single request as clean text.
The difference we measured on this site is striking — same post:
| Format | Size |
|---|---|
| HTML | 139 KB |
| Markdown | 8.5 KB |
Roughly sixteen times. On the crawler side that is token cost, directly.
When is it worth it? If your content count is manageable — dozens of posts, not thousands — llms-full.txt makes sense. On a site with thousands of pages the file grows unwieldy; there, llms.txt plus per-page markdown twins is the better fit.
Generating it dynamically
A static file can be hand-written, but nobody remembers to update it with every new post. The right approach is to generate it from your content.
In the Next.js App Router, llms.txt is built as a route handler — a file at src/app/llms.txt/route.ts. A dot in the folder name is fine; Next treats it as a literal segment.
The logic runs like this: pull published content from the database, convert it to markdown lines, return it with a Content-Type: text/plain header. Add a cache duration so it does not hit the database on every request.
On this site revalidate is 3600 seconds — refreshed hourly. A newly published post appears in llms.txt within an hour at most.
Three things to watch:
Published and indexable content only. Drafts and noindex pages must not appear.
Truncate descriptions. The first 150–160 characters of the excerpt is plenty. More just inflates the file.
Pick one language or separate them. A single mixed-language file confuses the crawler. Either use your primary language or open a separate section per language.
Five common mistakes
1. Turning your sitemap into markdown. llms.txt is not a list of 400 URLs. Thirty to forty genuinely important pages is enough. The point is to signal priority; list everything and you have signalled nothing.
2. Writing it in marketing language. "As the sector's leading solution partner, with our customer-focused approach" tells a crawler nothing. "A Türkiye-based agency providing custom software and SEO for SMEs" tells it something.
3. Skipping descriptions. Listing only titles and URLs halves the information value.
4. Letting it go stale. A six-month-old file linking to deleted pages damages trust. Dynamic generation eliminates this entirely.
5. Writing llms.txt while robots.txt is still closed. The most frequent contradiction. If the crawler cannot enter the site, it cannot read llms.txt either. Remove the AI crawler blocks first.
Validation
Three checks after publishing:
# 1 — reachable, and with the right content type?
curl -sI https://yoursite.com/llms.txt | grep -iE "^HTTP|content-type"
# Expected: 200 and text/plain
# 2 — is the content right?
curl -s https://yoursite.com/llms.txt | head -30
# 3 — do the links work? (lists anything not returning 200)
curl -s https://yoursite.com/llms.txt \
| grep -oE 'https://[^)]+' \
| while read u; do
code=$(curl -s -o /dev/null -w '%{http_code}' "$u")
[ "$code" != "200" ] && echo "$code $u"
done
The third check matters — an llms.txt full of broken links is worse than none.
On its standard status
To be straight about it: llms.txt is not an official standard yet. No AI provider has documented "we read this file and use it this way."
So why do it? Three reasons.
The cost is very low. Half an hour of work; generated dynamically, maintenance is zero.
There is no downside risk. Nothing breaks; the effect on classic SEO is neutral.
The direction is clear. That crawlers prefer clean, structured content is an observed fact. Markdown twins and llms.txt are parts of the same logic.
This is not a claim that it definitely works. It is the calculation that the cost is low and the potential upside is high, so you do it. Being ready when the standard settles a year from now beats starting then.
Frequently asked questions
sitemap.xml is a machine-readable flat list of URLs that gives every page equal weight. llms.txt is a human-readable, priority-ordered table of contents in markdown: one paragraph stating what the site is, then the most important pages under headings. Neither replaces the other.
Thirty to forty genuinely important pages is enough. llms.txt is not a copy of your sitemap; its purpose is to signal priority. If you list 400 URLs you have signalled nothing. Legal text and archive pages belong under the Optional heading.
No. It makes sense when your content count is in the dozens, because everything arrives as clean text in one request. On a site with thousands of pages the file becomes unmanageable; there, llms.txt plus per-page markdown twins is the better approach.
Generate it. Nobody remembers to update a hand-written file with every new post, and six months later a file linking to deleted pages damages trust. In the Next.js App Router you build a route handler at src/app/llms.txt/route.ts and generate it from your content.