Building in public: what I got wrong, what the bugs taught me, and where it landed.
I spent a while planning a product before writing any of it, which I'd recommend. Then an hour of checking the current state of the market showed the plan was about nine months out of date, which I'd recommend even more.
Here's the whole arc, including the parts that make me look worse.
The plan
AI agents visit websites now. A standard exists — Web Bot Auth — for an agent to prove its identity by signing requests with a cryptographic key. Most sites can't check those signatures. Enterprise bot-management vendors sell this to companies with security teams, at prices those companies pay and small sites don't.
So: verification as a service, for the long tail. An npm package, a WordPress plugin, a Cloudflare Worker template, a dashboard.
It's a real problem. The reasoning was sound. The plan was dead on arrival.
What an hour of searching found
Cloudflare had activated Web Bot Auth verification at its edge, meaning anyone behind Cloudflare — including the free plan — could turn on signature checking as a dashboard toggle.
Vercel's BotID already returned isVerifiedBot and verifiedBotName, so any Next.js app on Vercel got the feature as a function call.
AWS WAF, Shopify and Akamai had shipped support. Shopify handles it at the platform level, so a merchant can't insert anything at that layer even if they want to.
Cloudflare Research had open-sourced the verification logic — a TypeScript package and a Rust crate, by the people writing the specification.
So the core of my product was: npm install web-bot-auth, plus a cache.
The platforms had absorbed the feature. Not because they'd noticed me — I hadn't started — but because it's infrastructure, and infrastructure companies absorb infrastructure features. That's a structural fact about building anything that sits between a request and a server, and I should have known it before writing a plan.
What survived
Three things, once I stopped defending the original idea.
The sites the platforms don't reach. WordPress on shared hosting. Django on a VPS. Small shops on older stacks. They have the same problem and none of the solution, and the platforms have no reason to serve them — that's the definition of a long tail.
The interpretation. The protocol landscape is genuinely confusing. Which crawler does what, what "unsigned" means, whether blocking one costs you traffic. Nobody was translating it for people without security teams.
The moving data. Which operators sign, which publish IP ranges, what changed this month. A big company won't maintain that as content. A solo developer can.
The product became visibility for the long tail, with verification as a feature inside it rather than the thing being sold.
Four bugs that taught me something
Building it surfaced a category of failure I'd underrated: code that runs perfectly and produces wrong answers.
The honeypot that scored nothing
I added a honeypot — a link hidden from people, on a path robots.txt disallows. Anything reaching it has crawled links programmatically and ignored the one instruction the site left. It's the strongest signal available without TLS access.
It worked in tests. In production it did nothing.
The scoring logic lives in a shared module, but the Supabase Edge Function can't import local files, so it's inlined at build time. I'd added the honeypot signal to the source and never regenerated the inlined copy. The function passed hitHoneypot: true to a version of the scorer that had never heard of it, and the parameter was silently ignored.
No error. No warning. The score just didn't rise.
Fix: a generator script that assembles the function from the source, plus assertions that the output contains the expected markers. Duplication that can't be avoided should at least be unable to drift quietly.
The accusation
A dashboard showed DuckDuckBot with the status "Forged" — meaning a signature arrived and was invalid. DuckDuckGo doesn't sign. So either something strange was happening, or I was wrong.
I was wrong, twice.
The verification path treated a missing Signature-Agent header as forgery. But that header is how you locate the key directory — without it there's no directory to fetch, which makes the signature unjudgeable, not fake. Calling it forged accuses a visitor of something you can't show.
Worse: the URL was reconstructed as https:// + host + path, and the client sent the path without the query string. RFC 9421 signs the whole target URI. So any signed request carrying parameters would fail verification and be reported as a forgery — a false accusation against a genuine agent, generated automatically, at scale.
That one bothers me most. A product about trust that produces confident false accusations is worse than no product.
The flood
The Django middleware forwarded browser traffic so the behavioural signals had something to examine. On a test site, fine. On a municipal site with real visitors, it produced hundreds of rows within minutes — mostly the site's own /api/weather/ polling and cookie-banner checks.
Two failures at once: the dashboard became unreadable, and a free 500 MB database would have filled in days.
The fix was a filter — skip requests carrying the full set of markers a real browser sends, since those score zero anyway. But the useful part was noticing that I'd fixed it in one integration while five others carried the same trap. So the rule moved into a shared module, and the PHP and Python versions are now generated from it and tested against the same cases.
The endpoint that moved
Google relocated its published IP range files, and renamed googlebot.json to common-crawlers.json.
The old URLs neither redirected nor errored. They returned 200 with an unrelated payload. Anything parsing loosely got a successful response, found no prefixes, and continued with an empty allowlist — verifying nothing, forever, with nothing to notice.
Now "parsed successfully but zero prefixes" is treated as a failure. And when a list can't be loaded, the verdict is unknown, never mismatch — an outage on the operator's side must not turn genuine crawlers into accusations.
The finding that changed the pitch
I set up a canary — an unguessable URL containing a phrase nothing could infer — and asked an AI assistant to read it. It quoted the phrase back, so something had fetched the page.
The log showed a request with a complete, ordinary browser header set. No bot name. No signature. A language preference that didn't match the site, and an address in an unrelated region.
Then I ran it through my own behavioural scoring, which is the thing I'd built for exactly this case.
18 out of 100. The threshold is 45. My tool waved it through.
That's not a bug. At the HTTP layer, a well-configured headless browser through a residential proxy is genuinely indistinguishable from a person. There is nothing to catch.
I could have quietly raised the sensitivity until it caught that request. It would have caught a lot of real visitors too, and I'd have had a product that was confidently wrong instead of honestly limited.
Instead the dashboard now says it: "Crude automation stands out clearly. A well-configured headless browser coming through a residential proxy does not, and will sit here undetected."
That's a worse sales pitch and a better product. It's also the only version that survives the first customer checking.
What I'd tell someone starting
Check the market before writing the plan, not after. An hour of searching would have saved a week of planning the wrong thing. I did it in the wrong order.
If your product sits between a request and a server, assume the platforms will absorb it. Build for whoever the platforms don't reach.
The dangerous bugs run fine. Everything above compiled, deployed, and returned 200. Wrong answers don't announce themselves. Test against real captured data, not fixtures you wrote to match your assumptions.
Say what you can't do. The temptation to round "mostly works" up to "works" is constant, and every honest limitation I've written down has made the product easier to defend, not harder to sell.
The product is crawlprint.com — a free scanner that shows whether a site can tell a real agent from one wearing its name, and a log of what actually visits. Happy to talk about any of the above, particularly the parts that went wrong.