Field guide · updated 2026
AI agents are reading your site.
Here's what that actually means.
No fear-mongering, no jargon. Just what's happening, why it's different from the crawlers you already know, and what you can realistically do — whether you run a WordPress blog or a hand-built site.
Three kinds of thing are visiting
“AI bot” gets used for three very different visitors, and lumping them together is why people feel confused about what to do.
- Training crawlers
- Bots like
GPTBot,ClaudeBotandGoogle-Extendedcollect pages that may feed future AI models. This is the one most creators worry about — your words becoming training data. You can ask them not to via robots.txt, and the major ones honor it. - Search & answer bots
OAI-SearchBot,PerplexityBotand similar index your pages so an AI can cite them in an answer. Blocking these is like blocking Google — you lose the referral traffic when an AI recommends you.- Live agents
- The newest and fastest-growing kind. When someone asks ChatGPT or a Google assistant to “book this” or “read that page and summarize it,” an agent visits your site in real time, acting for a specific person. This isn't scraping — it's a customer, mediated by an AI.
The problem: a name is not proof
Every visitor announces itself with a User-Agent — a line of text like Mozilla/5.0 … GPTBot/1.2. For decades that's been the honor system, and it works right up until someone lies. Any scraper can put GPTBot in that line to look official, or borrow Googlebot to slip past filters. There's nothing behind the name to check.
That's exactly what our free scanner demonstrates: it sends requests wearing these famous names and shows you that your site — like most — can't tell them from the real thing.
What changed in 2026: signatures
A standard called Web Bot Auth fixes the honor system. Instead of just claiming a name, an agent signs each request with a cryptographic key and publishes the matching public key at a known address. Now the claim can be verified — a forged GPTBot simply can't produce a valid signature.
The catch: adoption is early. As of 2026, Google signs some of its agent traffic and a few others have started, but most crawlers don't sign yet. So “unsigned” doesn't mean “fake” — it usually just means “hasn't caught up.” Any tool that screams “impostor!” at every unsigned bot is lying to you. The useful move today is visibility: know who signs, who doesn't, and who's faking a name that should be signed.
What you can actually do
Decide what you want
Keep training bots out but let answer bots in? Welcome everything and just watch? There's no universal right answer — it depends on whether AI referral traffic helps you or your words being training data bothers you more.
Set your robots.txt honestly
This is the front door for the well-behaved bots. Allow the search and answer bots if you want the traffic; disallow the training crawlers if you don't. It won't stop bad actors — but it clearly signals intent to the good ones, which is most of them.
Get visibility before you block
You can't make good decisions about traffic you can't see. Knowing which agents actually visit, how often, and what they read comes first. Blocking blind tends to catch the wrong things — like the answer bot that was about to send you a customer.
Verify where it counts
For the traffic that matters — checkout, gated content, anything you'd protect — checking signatures lets you trust the real agents and flag the fakes. That's the part Crawlprint is being built to handle for sites that aren't behind a big CDN.
Common questions
Will blocking AI bots hurt my search ranking?
Blocking the AI answer bots (like PerplexityBot or OAI-SearchBot) can cost you citations and referral traffic, the same way blocking Google would. Blocking pure training crawlers (GPTBot, Google-Extended) has no effect on search — they're unrelated to indexing.
Is this a security threat I need to panic about?
For most sites, no. A misbehaving crawler is a nuisance — extra server load, or your content used in ways you didn't choose — not a breach. The point of getting visibility is to make calm, informed choices, not to react out of fear.
Can I use this on Shopify?
Only the scanner. Shopify renders themes on its own servers and runs apps outside the request, so there's no point where anything of ours could watch a visit. A JavaScript tag wouldn't help — crawlers don't run scripts, so it would miss precisely the traffic you want to see. Putting Cloudflare in front of the store is the one workaround, and it's more setup than most people will want.
I'm on shared hosting with no CDN. Can I even do anything?
Yes — that's exactly the gap Crawlprint targets. The signature checking that Cloudflare and Vercel bake in for their customers doesn't reach WordPress on cheap hosting. A lightweight plugin can bring the same visibility without changing hosts.
Does the scanner change anything on my site?
No. It sends read-only GET requests to your homepage — the same kind your normal visitors send — and compares the responses. It writes nothing and logs nothing on your end.