Names no specific company or product -- this compares against the manual habit
most site owners already have: open each AI-crawler operator's own docs page
one at a time (OpenAI's, Anthropic's, Google's, Apple's, Perplexity's, and five
more), note the user-agent token it names, and cross-reference that token
against your own robots.txt by eye.
Reading each operator's own documentation is free and needs no tool, and it is
the authoritative source -- this tool's own token list is sourced from exactly
those pages. That method has failure modes specific to this domain that this
tool exists to close. First, there are twelve separate operators publishing
twenty tokens between them, in twelve different documentation formats, and a
manual pass that skips one operator silently skips every token that operator
controls -- Diffbot, Amazonbot, Meta-ExternalAgent, CCBot and the
training-specific Applebot-Extended are each easy to miss next to the
well-known GPTBot. Second, the actual effect of a robots.txt file
is not "the first matching rule wins", which is how a list of rules reads to
most people skimming top to bottom -- it is a longest-matching-path-wins
algorithm, ties going to Allow, the same one Google documents for its own
crawlers. Reproducing that correctly by eye, across every group in a file, is
exactly the kind of mechanical step a person skims past. Third, several
operators split one company's crawling into two or three tokens with
different purposes -- training, user-initiated browsing, and search-index
retrieval -- and blocking the training token does nothing to the other two, a
routine manual mistake when only one of an operator's docs pages gets read.
Fourth, a by-hand check leaves nothing behind afterward: no file, no per-token
verdict, no audit trail, only a recollection that someone looked.
| What you are trying to do | Where to do it |
|---|---|
| Confirm exactly what one operator states its crawler does, in its own words, as the authoritative source. | That operator's own docs page -- this tool's token list is sourced from exactly those pages and is not a substitute for reading them directly. |
Check your existing robots.txt against all 20 named tokens across all twelve operators at once, without opening twelve separate docs pages. |
This tool -- paste your file and get every token's current status in one pass. |
| Apply the longest-match-wins, ties-to-Allow algorithm correctly, instead of reading rules top to bottom as if the first match wins. | This tool -- it implements the documented algorithm mechanically, the same way for every group in the file. |
| See whether blocking one of an operator's tokens (say, GPTBot) leaves that operator's other tokens (ChatGPT-User, OAI-SearchBot) allowed. | This tool -- each token is reported separately, with its purpose, so a training block and a user-browsing allow do not get conflated. |
| Add a CI gate that fails a build the moment a specific crawler becomes allowed again after someone edits the file. | This tool's CLI -- node cli.mjs example.com --fail-if-allowed GPTBot, no account needed. |
| Keep a record you can paste into a PR description or an audit trail: every token, and its current allowed/blocked/partial status. | This tool -- --json gives a machine-readable version of the same report. |
Checking the docs by hand is not wrong so much as unverifiable and easy to apply inconsistently: someone who reads all twelve operators' pages, remembers every token, and correctly reproduces the longest-match algorithm by eye would eventually reach the same answer this tool gives. What that method structurally cannot do is guarantee the same twelve pages get re-checked every time the file changes, or leave behind anything but a memory that someone looked -- which is what this tool's fixed token list and deterministic report exist to replace, for free.
It catches whatever tokens you already knew to look for and remembered to check. GPTBot is well known; Diffbot, Amazonbot, Meta-ExternalAgent, CCBot and Applebot-Extended (a separate token from the regular Applebot that has crawled the web for search since long before AI training was a question) are each documented on a different operator's own site, in a different format, and a manual pass that skips one of the twelve operators silently skips every token that operator publishes.
Because the actual effect of a robots.txt file is not "the first matching rule wins", which is how most people read a list of rules top to bottom. Google documents, and most major crawlers follow, a longest-matching-path-wins algorithm where ties go to Allow -- so a broad wildcard Disallow block followed later by a narrower Allow for one token does not mean the wildcard rule applies to that token first. Reproducing that algorithm correctly by eye, across every group in a file, is exactly the kind of mechanical step a person skims past.
No, and this is a common manual mistake. OpenAI alone publishes three separate tokens with three different purposes -- GPTBot for model training, ChatGPT-User for fetching a page because a ChatGPT user's session asked it to, and OAI-SearchBot for building ChatGPT's search results -- and Anthropic and Perplexity each split the same way. Blocking GPTBot to opt out of training does nothing to ChatGPT-User or OAI-SearchBot; a robots.txt written by someone who only checked one of an operator's docs pages routinely blocks or allows the wrong one of the three.
Only as a note that someone looked and did not personally notice a problem -- it names no specific token, no specific line of the file, and no verdict, so there is nothing to attach to an audit trail besides recollection. A structured report naming all 20 tokens and their current allowed/blocked/partial status is stronger evidence than "I checked the docs and it looked right."
← Back to AI Crawler Access Check · exact robots.txt lines for each crawler