AI Crawlers and Scrapers: How to Identify, Verify and Control Them

"AI crawler" covers bots that do very different things, and the distinction matters because you may want to treat them differently:

Google handles AI training differently: there is no separate Google AI crawler. Instead, the Google-Extended robots.txt token controls whether content Google has crawled may be used to train Gemini models and for grounding in Gemini and Vertex AI. Google states that blocking Google-Extended doesn't affect a site's inclusion or ranking in Google Search.

Blocking a training crawler keeps your content out of future training data. Blocking a search crawler keeps your site out of that assistant's cited answers — and out of the traffic those citations send.

Cloudflare, which sits in front of a large share of websites, has published the most detailed data on AI crawling.

Training dominates. In its August 2025 analysis, 80% of AI crawling over the previous twelve months was for training, 18% for search and 2% for user actions. By June 2026, Cloudflare reported that 52% of *all* crawler requests on its network — not just AI crawlers — were for AI training, up from 22% in spring 2025. It also reported that more than half of internet traffic is now non-human.

Crawl-to-refer ratios compare how many pages a platform's bots fetch with how many visits that platform sends back. Cloudflare's figures for July 2025:

Two caveats Cloudflare itself notes: native apps often send no referrer, so the ratios overstate the imbalance somewhat; and the numbers move quickly — Anthropic's ratio fell from about 287,000 in January 2025 to about 38,000 in July 2025. The direction is clear, though: training crawlers take far more than they send back, while search engines remain the main source of referral traffic.

The major AI operators say their crawlers obey robots.txt, the standard formalised as RFC 9309. Each bot has its own user-agent token, so you can be precise.

Allow AI search, block AI training:

# Training — opt out
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# AI search — allow, so assistants can cite and link to you
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

A few details from the operators' documentation:

• OpenAI says robots.txt changes for OAI-SearchBot can take about 24 hours to take effect, and that because ChatGPT-User acts on a user's request, robots.txt rules may not apply to it.
• Anthropic supports the non-standard Crawl-delay directive to slow crawling, and recommends robots.txt rather than IP blocking — blocking its IPs also stops it from reading your robots.txt, so it can't see your opt-out.
• Google-Extended is a control token only; it never appears as a user agent in your logs.

robots.txt is a request, not an enforcement mechanism. Well-behaved crawlers follow it; scrapers that ignore it need to be stopped at the network level.

For what it's worth, this site currently allows all of these bots: our tools and guides are meant to be found and cited, and AI assistants are increasingly how people find them.

Anyone can send a request with User-Agent: GPTBot. Scrapers routinely impersonate well-known crawlers precisely because sites let them through. Never allow or block on the user-agent string alone.

Check the IP against the operator's published ranges. OpenAI, Anthropic and Google all publish JSON files listing the IP ranges their crawlers use — for example https://openai.com/gptbot.json, https://claude.com/crawling/bots.json and Google's common-crawlers.json. They share the same format, so one script checks any of them:

$ curl -s https://openai.com/gptbot.json \
    | jq -r '.prefixes[] | .ipv4Prefix // .ipv6Prefix' > gptbot-ranges.txt

$ python3 -c "import ipaddress,sys; ip=ipaddress.ip_address(sys.argv[1]); \
print(any(ip in ipaddress.ip_network(n.strip()) for n in open('gptbot-ranges.txt')))" 20.171.207.10
True

Fetch the lists regularly — they change.

Or use forward-confirmed reverse DNS where the operator documents it. Google's crawlers reverse-resolve to hostnames under googlebot.com or google.com, and that hostname must resolve back to the same IP. The Reverse DNS Lookup runs both lookups at once; the reverse DNS guide explains why both are needed.

Cryptographic signatures are coming. An IETF draft, HTTP Message Signatures for Bots (often called Web Bot Auth), lets a bot sign its requests so a site can verify the operator with a public key instead of an IP list. It's still a draft, and support varies by operator, so treat it as an addition to IP checks for now.

Start by seeing who actually visits. On a standard nginx or Apache access log, this lists the most active user agents:

$ awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -20

And this lists the IPs making the most requests:

$ awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20

Signs of a scraper rather than a real visitor or legitimate bot:

• A bot user agent from an IP that isn't in that operator's published ranges.
• Hundreds of requests a minute from one IP or a small block, with no CSS, JavaScript or image requests.
• A browser user agent from a hosting provider's network. Real people browse from residential and mobile networks; the IP Reputation check shows an IP's usage type and abuse history.
• Systematic traversal: /page/1, /page/2, /page/3 … at machine speed.

Stopping them: rate-limit per IP at your web server or CDN, block abusive IP ranges or networks at the firewall, and use your CDN's bot management if it has one. Don't block an entire cloud provider's address space unless you're sure no legitimate users or services — including your own monitoring and payment webhooks — come from it.