Open any access log today and you'll see GPTBot, ClaudeBot, PerplexityBot and friends. The problem: a user agent is just a string. Anyone can send User-Agent: GPTBot from a $5 VPS, and scrapers do it on purpose, because many sites whitelist "good bots".

So before you celebrate ("ChatGPT reads my site!") or panic ("AI bots are hammering my server!"), check whether the traffic is real. Here's how, with a few lines of Node.

Why it matters

Fake AI bots cause three kinds of wrong decisions:

  • Wrong stats. You think AI search is crawling you a lot, but half of it is a scraper with a borrowed name.
  • Wrong blocks. You see "GPTBot" hammering a slow endpoint and block it in robots.txt. The real GPTBot obeys and leaves. The fake one never read robots.txt anyway.
  • Wrong allow rules. A WAF rule like "allow anything containing ClaudeBot" is an open door for whoever reads your rule.

Method 1: compare the IP with the vendor's published ranges

Several AI companies publish the IP ranges their crawlers use, usually as a JSON file linked from their bot documentation. OpenAI, for example, publishes separate lists for GPTBot, OAI-SearchBot and ChatGPT-User, and Google publishes a list for Googlebot. The format is typically a list of CIDR prefixes:

{ "prefixes": [ { "ipv4Prefix": "20.171.206.0/24" }, { "ipv6Prefix": "2a03:2880:f800::/48" } ] }

(The prefixes above are just placeholders, always load the current file from the vendor's docs.)

Checking an IPv4 address against a CIDR range takes a few lines:

function ipToInt(ip) { return ip.split('.').reduce((acc, octet) => (acc << 8) + Number(octet), 0) >>> 0; } function inCidr(ip, cidr) { const [range, bits] = cidr.split('/'); const mask = bits === '0' ? 0 : (~0 << (32 - Number(bits))) >>> 0; return (ipToInt(ip) & mask) === (ipToInt(range) & mask); } async function isRealBot(ip, rangesUrl) { const res = await fetch(rangesUrl); const { prefixes } = await res.json(); return prefixes .map((p) => p.ipv4Prefix) .filter(Boolean) .some((cidr) => inCidr(ip, cidr)); }

For IPv6, use a library such as ipaddr.js instead of rolling your own. Cache the JSON for a few hours; the lists change, but not every minute.

Method 2: reverse DNS, then forward DNS

Some crawlers (Googlebot and Bingbot are the classic examples) are verified by DNS instead of, or in addition to, a published list. The check has two steps, and you need both:

  1. Reverse lookup: the IP should resolve to a hostname on the vendor's domain.
  2. Forward lookup: that hostname should resolve back to the same IP.

Step 2 is the important one. Whoever controls an IP block can set any reverse DNS name they like, including crawl-1-2-3-4.googlebot.com. They can't make Google's DNS point that name back at their IP.

import { promises as dns } from 'node:dns'; async function verifyByDns(ip, allowedSuffixes) { try { const hostnames = await dns.reverse(ip); for (const host of hostnames) { if (!allowedSuffixes.some((s) => host.endsWith(s))) continue; const addrs = await dns.resolve(host); if (addrs.includes(ip)) return { ok: true, host }; } } catch {} return { ok: false }; } await verifyByDns('66.249.66.1', ['.googlebot.com', '.google.com']);

Which method applies to which bot is documented by each vendor, and it does change. Before you hard-code anything, check the bot's official page.

Putting it together on a log file

A quick script to see how much of your "AI traffic" is genuine:

import { readFileSync } from 'node:fs'; const BOTS = /gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|perplexitybot|perplexity-user/i; const lines = readFileSync('access.log', 'utf8').split('\n'); const hits = {}; for (const line of lines) { const ua = line.match(BOTS); if (!ua) continue; const ip = line.split(' ')[0]; const key = ua[0].toLowerCase(); hits[key] ??= new Set(); hits[key].add(ip); } for (const [bot, ips] of Object.entries(hits)) { console.log(bot, ips.size, 'unique IPs'); // run each IP through isRealBot / verifyByDns here }

Deduplicate IPs before you verify them: a busy log can have thousands of hits from a handful of addresses, and you don't want thousands of DNS lookups.

What you'll usually find

From what I've seen on small and mid-sized sites:

  • Real AI crawlers tend to come from a small, stable set of ranges and respect robots.txt.
  • Fakes rotate through residential and cloud IPs, ignore robots.txt and often hit login, search and API URLs that real crawlers have no reason to touch.
  • User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) appear in bursts that line up with someone asking an assistant about your page.

What to do with the result

  • Verified bot, unwanted: robots.txt is enough. The real bot will follow it.
  • Fake bot: robots.txt is useless. Rate-limit or block at the firewall or CDN by IP or behaviour, not by user agent.
  • WAF allow rules: never allow by user agent alone. Pair it with an IP or DNS check, or use your CDN's "verified bots" feature if it has one.

Shortcut

If you just want an answer for one address, the AI Bot IP Checker on BotReady checks an IP against official AI crawler ranges. For a whole log file, the AI Bot Log Analyzer shows which AI bots visited, what status codes they got, and which hits look spoofed.

Have you caught a fake crawler in your logs? I'm curious which user agent it was pretending to be.