Spoofed AI Bots and the Agentic Scraping Wave: Protecting Your Site

AI bot traffic is booming — and so are the fakes. In August 2026, researchers reported large-scale vulnerability scanning campaigns that spoof well-known AI crawlers like ClaudeBot, hiding hostile scans behind trusted-looking user agents. At the same time, legitimate AI agents are crawling sites far more aggressively than traditional search engines ever did.

For anyone running a public server, this changes the bot-defense playbook. You can no longer assume a user agent that says "ClaudeBot" is actually Claude.

Why AI bots changed the threat model

Classic crawlers respected robots.txt and visited politely. Modern AI agents behave differently:

• They execute JavaScript, fill forms, and trigger endpoints that traditional crawlers never touched.
• They can be repurposed — or impersonated — to run mass vulnerability scans.
• Their traffic patterns (high request rates, many unique IPs) look hostile even when the intent is benign.

The net effect: naive allowlists of known agent user agents are now dangerous, because spoofing a user agent is trivial.

Practical defenses

If you run a Next.js site or any public API, these steps cut the risk:

1. Verify agent identity beyond the user agent. Reverse-DNS checks, ASN validation, and published crawler IP lists are far stronger than trusting the UA string. If you allowlist agents like ClaudeBot or GPTBot, verify the IP, not the header.
2. Rate-limit aggressively per IP and per path. Our own Next.js security hardening (see our earlier post on defending against bots) starts with rate limiting at the edge. AI scans are high-volume; throttling neutralizes most of them before they reach your app.
3. Honeypot the unknown. Entice scrapers and bots with hidden links or endpoints, then block everything that follows them. Legitimate agents that obey robots.txt will not bite; scanners often will.
4. Watch your access logs for anomalies. Unexpected POST floods, repeated scanning paths (.env, wp-admin, /.git), and single-IP bursts all deserve attention regardless of the user agent attached.
5. Treat robots.txt as a courtesy, not a security control. It stops polite crawlers and absolutely nothing else.

The takeaway

The agentic web is generating enormous traffic — and the window for spoofing AI bots is wide open. Trust nothing by user agent alone. Combine IP verification, rate limiting, honeypots, and log review, and treat every agent request as potentially hostile until proven otherwise.

Your logs are the ground truth. If you have spotted spoofed AI bot traffic on your own server, tell us what you found — it helps everyone harden their defenses.