# Why ChatGPT Can't Read Your Site: 7 Causes and a Checklist
> If ChatGPT is not reading your site, the cause is almost always one of seven things. A checklist that eliminates each in order, with copy-paste commands.
- Kanonik URL: https://algoritmaajans.com/en/blog/why-chatgpt-cant-read-your-site-checklist
- Dil: en
- Kategori: Artificial Intelligence
- Yayın: 2026-08-17
- Güncelleme: 2026-08-27
- Yayıncı: Algoritma Ajans
---
**If ChatGPT is not reading your site, the cause is almost always one of seven things: `robots.txt` blocks the AI crawler, content arrives via JavaScript, the same content is published at multiple URLs, a canonical or noindex error, no discovery layer (`llms.txt`), missing or broken structured data, or a server that is simply too slow.** The checklist below eliminates each in order.
Order matters. Working on item six before fixing item one is wasted time — if the crawler cannot enter the site at all, perfect schema changes nothing.
## 1. robots.txt is blocking AI crawlers
The most common cause. And usually nobody did it on purpose — a plugin, a theme, or a "security" recommendation added it.
**Check:** open `yoursite.com/robots.txt` in a browser. Look for lines like:
```
User-agent: GPTBot
Disallow: /
```
If any of `GPTBot`, `OAI-SearchBot`, `ChatGPT-User`, `Google-Extended`, `ClaudeBot`, `PerplexityBot` or `CCBot` is matched with `Disallow: /`, you have found the problem.
This is exactly what was happening on this site. `robots.txt` blocked `GPTBot`, `Google-Extended` and `CCBot` wholesale. An agency selling AI integration had a site closed to AI.
**Fix:** allow the crawlers, keep private areas closed:
```
User-agent: GPTBot
Allow: /
Disallow: /api/
Disallow: /admin/
```
**Do not confuse two things:** crawl permission and training-data permission are separate. If you do not want your content used for model training, declare it in `ai.txt`. Blocking crawl in `robots.txt` does not remove you from training sets — it only removes you from being cited in answers. You are the one who loses.
## 2. Content arrives via JavaScript
If the site was built as a single-page application, the HTML the server returns probably looks like this:
```
```
The browser runs JavaScript and fills in the content. But most AI crawlers do not execute JavaScript. What they see is an empty box.
**Check:** fetch the raw HTML and look for a sentence from your content:
```bash
curl -s https://yoursite.com/blog/some-post | grep -c "a sentence from your post"
```
If the result is `0`, content is not server-rendered.
**Fix:** server-side rendering (SSR) or static generation (SSG). Frameworks like Next.js, Astro and Nuxt do this by default. Migrating an existing SPA is significant work; at minimum, prerender your highest-traffic pages.
## 3. The same content is published at multiple URLs
If a crawler finds the same text at three addresses, it cannot tell which to cite. Usually it cites none.
Multilingual sites are the most common source. On this site every blog post returned 200 **under both locales**. Because Turkish and English posts have different slugs, `/en/blog/turkish-slug` served Turkish content under the English locale — and gave itself a self-referencing canonical on top.
**Check:** take a post URL, swap the language code, open it. Does the same content load? Then you have the problem.
Also try random paths containing a dot: `yoursite.com/test.xyz`. If the homepage returns 200, something is badly wrong. That is precisely what happened here — every path containing a dot returned the homepage with a self-referencing canonical. Infinite duplicate URLs.
**Fix:** each piece of content lives at exactly one canonical URL. Add the language filter to the query, and return 404 for invalid language codes.
## 4. Canonical or noindex error
The silent killer. The page loads, looks fine, but carries a noindex in a header or meta tag.
**Check:**
```bash
curl -sI https://yoursite.com/page | grep -i "x-robots-tag"
curl -s https://yoursite.com/page | grep -o ']*>'
curl -s https://yoursite.com/page | grep -o ']*>'
```
Three errors to look for: `noindex` sitting on a page that should be indexed; a canonical pointing at a different page; no canonical at all.
**Fix:** every page should carry a self-referencing canonical, and noindex tags copied over from a staging environment must be cleaned out.
## 5. No discovery layer
`sitemap.xml` serves classic search engines. AI crawlers get two additional files: `llms.txt` and `ai.txt`.
`llms.txt` is a markdown file that states what your site is in one paragraph, then lists your most important pages under headings. The heading grouping carries a priority signal.
**Check:** does `yoursite.com/llms.txt` return 404?
**Fix:** create it. Even a static file helps, but generating it from your content is better — new posts then enter automatically.
## 6. Structured data missing or broken
AI engines derive meaning from entity relationships. If `Article` schema exists but has no author, that content stays "text of unclear origin."
Broken schema is as bad as missing schema, and more insidious — because it looks present.
An example we caught on this site: FAQ records were stored with `{q, a}` fields, but the schema-generating code read `{question, answer}`. Eight posts published `FAQPage` output full of `undefined` values.
**Check:** run the page URL through Google Rich Results Test. Errors and warnings should be zero. Also inspect the raw output:
```bash
curl -s https://yoursite.com/page | grep -c "undefined"
```
**Fix:** build connected entities inside a single `@graph`: `Article` → `author` → `publisher` → `image` → `breadcrumb`.
## 7. The server is too slow
A crawler's patience is finite. If your page takes ten seconds to start responding, it gives up.
A concrete measurement from this site: the blog listing page took **9–15 seconds** server-side. The cause was nine PNG files at 1024×1024 and roughly 1 MB each, all being converted to AVIF simultaneously. AVIF encoding costs ten to twenty times the CPU of WebP.
Disabling AVIF and keeping WebP brought the same page down to **1.1–2 seconds**. About a tenfold improvement.
**Check:**
```bash
curl -s -o /dev/null -w "%{time_total}s\n" https://yoursite.com/blog
```
Above two seconds deserves investigation; above five is urgent.
**Fix:** downscale images at source (1600px width is more than enough for most layouts), review expensive image formats, simplify database queries.
The same audit turned up something else: the listing page was shipping full markdown bodies to the client that the cards never used — roughly 55 KB of dead weight for nine posts. Selecting only the needed fields dropped the payload to 22 KB.
## Copy-paste checklist
Run these in order, stop and fix at the first failure:
```bash
SITE="https://yoursite.com"
# 1 — are AI crawlers blocked?
curl -s $SITE/robots.txt | grep -A2 -iE "gptbot|google-extended|claudebot|perplexity|ccbot"
# 2 — is content server-rendered? (0 means no)
curl -s $SITE/blog | grep -c "]*>'
# 5 — discovery layer
for f in /sitemap.xml /robots.txt /llms.txt /ai.txt; do
echo "$(curl -s -o /dev/null -w '%{http_code}' $SITE$f) $f"
done
# 6 — is schema broken?
curl -s $SITE/blog/some-post | grep -c '"@type"'
curl -s $SITE/blog/some-post | grep -c 'undefined'
# 7 — response time
curl -s -o /dev/null -w "%{time_total}s\n" $SITE/blog
```
## Priority order
If you cannot do all of it in one day, this is the sequence:
1. **robots.txt** — ten minutes, biggest impact
2. **Response time** — if the crawler gives up, nothing else matters
3. **Duplicate URLs** — it must be clear which page is the source
4. **llms.txt** — half an hour, lasting benefit
5. **Schema graph** — medium term, improves citation quality
6. **Markdown twin** — advanced, start with your highest-traffic pages
The first three are a single day's work on most sites, and they account for most of the difference.