Most advice on this topic is either "block all the AI bots" or "don't worry about it." Both are wrong for most sites, and both skip the part that matters: the different bots want different things, and blocking them has different consequences.
Here's the version I wish I'd had.
There are four kinds of visitor, not one
Lumping them together as "AI bots" is why the advice is bad.
Training crawlers
GPTBot, ClaudeBot, Google-Extended, Bytespider, CCBot, Applebot-Extended
They collect pages that may end up in the training data for future models. This is the group most writers actually care about — your words becoming someone else's model.
Blocking costs you nothing in search or referrals. They're unrelated to indexing. If you don't want your work used this way, this is the group to disallow, and the well-behaved ones honour it.
Answer engines
OAI-SearchBot, PerplexityBot, Claude-SearchBot, Google-CloudVertexBot
They index your pages so an AI can cite them when someone asks a question.
Blocking these is like blocking Google. You lose the citations and the traffic that comes with them. For most sites that's a bad trade — this is the channel replacing some of what search used to send.
Live agents
ChatGPT-User, Claude-User, Perplexity-User, Google-Agent
These fetch a page right now because a person asked. Someone said "read this page and summarise it" or "book me a table here."
Blocking these turns away an actual customer, mediated by an assistant. Notably, this is the fastest-growing category, and the one most likely to be caught by rules aimed at the others.
Everything else
Vulnerability scanners, scrapers, monitoring services, and the large category that sends an ordinary browser user-agent and no identification at all.
This group ignores whatever you write in robots.txt, because it was never reading it.
The uncomfortable bit: names aren't evidence
Every one of those visitors announces itself with a line of text:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2That's just text in the request. Anyone can type it. There's nothing behind it to check.
Which means a rule that blocks GPTBot blocks the real GPTBot and a scraper that borrowed the name — and the scraper changes one word and comes back.
I tested this on a site I run. The host had a rule blocking ClaudeBot and Bytespider, which I hadn't configured and didn't know about. Seven other crawler names, including GPTBot, walked straight through. The rule blocked the honest ones and inconvenienced nobody.
What actually verifies identity, in increasing order of strength:
Published IP ranges. OpenAI publishes openai.com/gptbot.json; Google publishes per-crawler files under developers.google.com/static/crawling/ipranges/. Match the claimed name against the operator's own list. An address is hard to fake. Anthropic publishes nothing, so ClaudeBot can't be checked this way.
Cryptographic signatures. Web Bot Auth has agents sign each request with a key whose public half is published. A forgery can't produce a valid signature. Adoption is early — Google signs some agent traffic, most operators don't sign at all — but it's the only method that doesn't degrade into guessing.
What to actually do
1. Write a robots.txt
Most sites don't have one. On the site I was investigating, OAI-SearchBot asked for robots.txt and got a 404 — it asked what the rules were and the site had no answer.
This doesn't stop bad actors and was never meant to. What it does is tell the well-behaved majority what you want. Right now most sites say nothing and are then surprised by what happens.
A reasonable starting point for a site that wants search and AI citations but not training use:
User-agent: *
Allow: /
# Training crawlers — collect pages for model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# Answer engines and live agents are left allowed on purpose:
# blocking them costs citations and turns away real visitors.If you want the opposite — everything welcome — say so explicitly. An empty Allow: / is a clearer signal than silence.
2. Look before you block
You can't make good decisions about traffic you can't see, and blocking blind tends to catch the wrong things — like the answer engine that was about to send you a customer.
Your analytics won't help here. Google Analytics runs in the browser, and crawlers don't execute JavaScript, so the whole category is invisible to it. Absence of AI traffic in your analytics is evidence of nothing at all.
The data is in your server logs. If you have cPanel, it's under Metrics → Raw Access. Sort by user-agent and you'll learn more in ten minutes than from any article, including this one.
3. Understand what the answer engines want
If you'd like to be cited rather than merely crawled: clear headings, factual statements near the top, structured data, and pages that answer a question directly. The same things that made content findable in search, minus the keyword games.
Blocking OAI-SearchBot and PerplexityBot while hoping to appear in AI answers is a contradiction worth noticing.
4. Don't expect to catch the hidden ones
The largest category by volume doesn't claim to be a bot at all. On one day of logs from a small site I found 474 requests carrying an identical iPhone user-agent — iOS 13.2.3, from 2019 — spread across dozens of Tencent Cloud addresses. No bot name. No signature. Invisible to every name-based rule.
Behavioural signals catch the crude versions: datacenter origin, missing browser headers, one fingerprint across many addresses, pages fetched without their images.
They don't catch the careful ones. I tested a real headless fetch through a residential proxy against my own scoring and it came out at 18 out of 100, where 45 means "probably automated." At the HTTP level a well-configured headless browser is genuinely indistinguishable from a person.
Anyone promising complete bot detection at this layer is promising something they don't have.
Questions people actually ask
Will blocking AI bots hurt my search ranking? Blocking answer engines costs citations and referral traffic, the way blocking Google would. Blocking pure training crawlers has no effect on search — they're unrelated to indexing.
Is this a security problem? For most sites, no. A misbehaving crawler is a nuisance — server load, or your content used in ways you didn't choose — not a breach. The vulnerability scanners in the same logs are the actual security story, and they've been there for years.
I'm on shared hosting with no CDN. Can I do anything? Yes, and this is the gap. The signature checking Cloudflare and Vercel bake in for their customers doesn't reach WordPress on cheap hosting. A small server-side plugin gets you the same visibility without changing hosts.
Should I use a CAPTCHA? Not for this. It punishes people to inconvenience software that mostly doesn't solve CAPTCHAs anyway, and the ones that matter can pay a dollar per thousand to have them solved.
The short version
Know the four groups. Write a robots.txt that says what you actually want. Look at your server logs before blocking anything. Expect to see the honest visitors and miss the careful ones — and be suspicious of anyone who tells you otherwise.
I build tooling for this — including a free scanner that shows whether your site can tell a real agent from one wearing its name, at crawlprint.com. No account needed.