← All checkpoints
AI Crawler & Machine Accessibility · 5 points

robots.txt allows major AI crawlers

Why it matters

If robots.txt disallows GPTBot, Google-Extended, PerplexityBot, ClaudeBot, or similar, those engines cannot crawl the page at all — this is the single highest-impact fix on the entire list, since everything else is irrelevant if the page can't be fetched.

How to fix it

Review robots.txt and remove any blanket Disallow rules targeting these user-agents, unless you deliberately want to opt out of AI training/crawling.

Example

User-agent: GPTBot
Allow: /

User-agent: Google-Extended
Allow: /