Why ChatGPT Can't Read Your Site: 7 Causes and a Checklist

If ChatGPT is not reading your site, the cause is almost always one of seven things: robots.txt blocks the AI crawler, content arrives via JavaScript, the same content is published at multiple URLs, a canonical or noindex error, no discovery layer (llms.txt), missing or broken structured data, or a server that is simply too slow. The checklist below eliminates each in order.

Order matters. Working on item six before fixing item one is wasted time — if the crawler cannot enter the site at all, perfect schema changes nothing.

1. robots.txt is blocking AI crawlers

The most common cause. And usually nobody did it on purpose — a plugin, a theme, or a "security" recommendation added it.

Check: open yoursite.com/robots.txt in a browser. Look for lines like:

User-agent: GPTBot
Disallow: /

If any of GPTBot, OAI-SearchBot, ChatGPT-User, Google-Extended, ClaudeBot, PerplexityBot or CCBot is matched with Disallow: /, you have found the problem.

This is exactly what was happening on this site. robots.txt blocked GPTBot, Google-Extended and CCBot wholesale. An agency selling AI integration had a site closed to AI.

Fix: allow the crawlers, keep private areas closed:

User-agent: GPTBot
Allow: /
Disallow: /api/
Disallow: /admin/

Do not confuse two things: crawl permission and training-data permission are separate. If you do not want your content used for model training, declare it in ai.txt. Blocking crawl in robots.txt does not remove you from training sets — it only removes you from being cited in answers. You are the one who loses.

2. Content arrives via JavaScript

If the site was built as a single-page application, the HTML the server returns probably looks like this:

<div id="root"></div>

The browser runs JavaScript and fills in the content. But most AI crawlers do not execute JavaScript. What they see is an empty box.

Check: fetch the raw HTML and look for a sentence from your content:

curl -s https://yoursite.com/blog/some-post | grep -c "a sentence from your post"

If the result is 0, content is not server-rendered.

Fix: server-side rendering (SSR) or static generation (SSG). Frameworks like Next.js, Astro and Nuxt do this by default. Migrating an existing SPA is significant work; at minimum, prerender your highest-traffic pages.

3. The same content is published at multiple URLs

If a crawler finds the same text at three addresses, it cannot tell which to cite. Usually it cites none.

Multilingual sites are the most common source. On this site every blog post returned 200 under both locales. Because Turkish and English posts have different slugs, /en/blog/turkish-slug served Turkish content under the English locale — and gave itself a self-referencing canonical on top.

Check: take a post URL, swap the language code, open it. Does the same content load? Then you have the problem.

Also try random paths containing a dot: yoursite.com/test.xyz. If the homepage returns 200, something is badly wrong. That is precisely what happened here — every path containing a dot returned the homepage with a self-referencing canonical. Infinite duplicate URLs.

Fix: each piece of content lives at exactly one canonical URL. Add the language filter to the query, and return 404 for invalid language codes.

4. Canonical or noindex error

The silent killer. The page loads, looks fine, but carries a noindex in a header or meta tag.

Check:

curl -sI https://yoursite.com/page | grep -i "x-robots-tag"
curl -s  https://yoursite.com/page | grep -o '<meta name="robots"[^>]*>'
curl -s  https://yoursite.com/page | grep -o '<link rel="canonical"[^>]*>'

Three errors to look for: noindex sitting on a page that should be indexed; a canonical pointing at a different page; no canonical at all.

Fix: every page should carry a self-referencing canonical, and noindex tags copied over from a staging environment must be cleaned out.

5. No discovery layer

sitemap.xml serves classic search engines. AI crawlers get two additional files: llms.txt and ai.txt.

llms.txt is a markdown file that states what your site is in one paragraph, then lists your most important pages under headings. The heading grouping carries a priority signal.

Check: does yoursite.com/llms.txt return 404?

Fix: create it. Even a static file helps, but generating it from your content is better — new posts then enter automatically.

6. Structured data missing or broken

AI engines derive meaning from entity relationships. If Article schema exists but has no author, that content stays "text of unclear origin."

Broken schema is as bad as missing schema, and more insidious — because it looks present.

An example we caught on this site: FAQ records were stored with {q, a} fields, but the schema-generating code read {question, answer}. Eight posts published FAQPage output full of undefined values.

Check: run the page URL through Google Rich Results Test. Errors and warnings should be zero. Also inspect the raw output:

curl -s https://yoursite.com/page | grep -c "undefined"

Fix: build connected entities inside a single @graph: Article → author → publisher → image → breadcrumb.

7. The server is too slow

A crawler's patience is finite. If your page takes ten seconds to start responding, it gives up.

A concrete measurement from this site: the blog listing page took 9–15 seconds server-side. The cause was nine PNG files at 1024×1024 and roughly 1 MB each, all being converted to AVIF simultaneously. AVIF encoding costs ten to twenty times the CPU of WebP.

Disabling AVIF and keeping WebP brought the same page down to 1.1–2 seconds. About a tenfold improvement.

Check:

curl -s -o /dev/null -w "%{time_total}s\n" https://yoursite.com/blog

Above two seconds deserves investigation; above five is urgent.

Fix: downscale images at source (1600px width is more than enough for most layouts), review expensive image formats, simplify database queries.

The same audit turned up something else: the listing page was shipping full markdown bodies to the client that the cards never used — roughly 55 KB of dead weight for nine posts. Selecting only the needed fields dropped the payload to 22 KB.

Copy-paste checklist

Run these in order, stop and fix at the first failure:

SITE="https://yoursite.com"

# 1 — are AI crawlers blocked?
curl -s $SITE/robots.txt | grep -A2 -iE "gptbot|google-extended|claudebot|perplexity|ccbot"

# 2 — is content server-rendered? (0 means no)
curl -s $SITE/blog | grep -c "<article"

# 3 — does an invalid path return 404? (200 means trouble)
curl -s -o /dev/null -w "%{http_code}\n" $SITE/test.xyz

# 4 — noindex / canonical
curl -sI $SITE | grep -i x-robots-tag
curl -s  $SITE | grep -o '<link rel="canonical"[^>]*>'

# 5 — discovery layer
for f in /sitemap.xml /robots.txt /llms.txt /ai.txt; do
  echo "$(curl -s -o /dev/null -w '%{http_code}' $SITE$f)  $f"
done

# 6 — is schema broken?
curl -s $SITE/blog/some-post | grep -c '"@type"'
curl -s $SITE/blog/some-post | grep -c 'undefined'

# 7 — response time
curl -s -o /dev/null -w "%{time_total}s\n" $SITE/blog

Priority order

If you cannot do all of it in one day, this is the sequence:

1. robots.txt — ten minutes, biggest impact
2. Response time — if the crawler gives up, nothing else matters
3. Duplicate URLs — it must be clear which page is the source
4. llms.txt — half an hour, lasting benefit
5. Schema graph — medium term, improves citation quality
6. Markdown twin — advanced, start with your highest-traffic pages

The first three are a single day's work on most sites, and they account for most of the difference.