AI assistants are now a major discovery channel. They cite, summarize, and send clicks. My take after watching Sam Oh from Ahrefs: too many sites are invisible by accident. That’s a missed opportunity you can fix with a handful of checks.
My opinion is simple: if you want AI to recommend your content, you must make it easy for crawlers to reach, read, and parse your pages. This is table stakes. It’s not code surgery. It’s housekeeping.
The Core Argument
Ahrefs’ data hit me like a splash of cold water. Millions of sites are blocking OpenAI’s GPTBot without knowing it. As Sam put it:
According to our data, around 5.9% of 140 million websites are blocking GPTBot… That’s millions of sites that are invisible to ChatGPT.
If AI can’t crawl you, it can’t credit you. In a world where assistants answer before users click, access is the new ranking. And a handful of default settings, theme leftovers, or CDN toggles can shut the door on you.
What Ahrefs Gets Right—and Why It Matters
First, the robots.txt reality check. It’s not just Google anymore. We’re dealing with GPTBot and OAI SearchBot from OpenAI, ClaudeBot from Anthropic, and Google-Extended. Any “Disallow” line against those bots is a self-inflicted blackout. Sam even flagged a sneaky culprit:
Cloudflare has a feature called “instruct AI bot traffic with robots.txt” that’s now enabled by default… you could be blocking AI crawlers without even realizing it.
That one setting can erase you from assistant answers. As someone who’s built and marketed sites since dial-up days, I’ve seen small toggles tank traffic. This is one of those toggles.
Second, JavaScript. Some assistants render JS. Some don’t. Gemini and Copilot can. ChatGPT’s crawler cannot. If your content depends on JS, it may be invisible. The fix is server-side rendering. Or at least confirm your content loads with JS off. It’s a five-minute test that can save your reach.
Third, speed. I’ve preached performance for years because conversions crater on slow pages. Sam adds a sharper point:
If your page takes too long to load, it can get dropped before it’s even scored.
Slow pages don’t just frustrate users. They vanish from real-time retrieval. Clean HTML and fast loads are no longer “nice-to-haves.” They’re eligibility checks.
Structure, Schema, and Sanity
Clear headings and focused sections help AI chunk content. That’s not theory; that’s how these systems extract facts. Use one H1, sensible H2s, and tight paragraphs. This isn’t about fancy prose. It’s about making each section stand on its own.
Schema? The evidence is mixed for assistant citations. I agree with Sam’s stance: add it if you’re already using it for search. Don’t burn hours expecting it to move the needle for AI. It’s hygiene, not a growth hack.
As for LLM.txt, it’s a neat idea with little support. OpenAI doesn’t use it. Google hasn’t adopted it. Keep your focus on the files that matter now—robots.txt and clean HTML.
Don’t Waste “Hallucinated” Clicks
This part made me sit up. Assistants sometimes make up URLs. Users click. They hit 404s. Ahrefs found assistants send visitors to 404s 2.87 times more often than Google, and about 1% of ChatGPT’s clicked URLs end up dead.
That’s traffic you can rescue. Track AI referrers to 404s. If a fake URL gets steady hits, redirect it to the most relevant real page. It’s triage that turns “oops” into opportunity.
A Quick Action List
Run this today and you’ll stop hiding from AI:
- Check robots.txt for GPTBot, OAI SearchBot, ClaudeBot, and Google-Extended. Remove unintended blocks.
- Review CDN or security defaults, including Cloudflare AI bot settings.
- Disable JavaScript and load key pages. If content disappears, prioritize server-side rendering.
- Speed up: optimize images, caching, and HTML. Keep it lean.
- Use clear H1/H2/H3 structure with focused sections.
- Add reasonable schema; skip overkill.
- Monitor 404s from AI referrers and set smart redirects.
The Bottom Line
We don’t win assistant citations with clever tricks. We win by being crawlable, legible, and fast. Ahrefs’ guidance is practical and right on time. Open the door for AI, and it will open doors for you.
My call to action: audit your site this week. Fix the blocks. Render your content server-side. Trim your pages. Then capture stray clicks with redirects. The AI feed is already shaping attention. Make sure your work gets a seat at the table.
