Known Agents
(renamed from Dark Visitors in early 2026 -- the old
darkvisitors.com domain now redirects to it) is a hosted, account-based service
built around a continuously updated directory of AI agents, crawlers and bots. Per its own
site and its WordPress plugin listing, it generates and keeps a site's robots.txt
current against that directory as new agents appear, can block scrapers that ignore
robots.txt, and reports analytics on bot traffic and AI-referral traffic --
visits arriving from ChatGPT, Perplexity and similar -- over time. It ships as a WordPress
plugin and an install for other stacks, and needs an account.
This tool is a different shape entirely: a free, static page -- and a zero-dependency Node
CLI -- that reads the robots.txt and llms.txt text you already
have, pasted or fetched, against a fixed, publicly documented list of 20 named AI-crawler
tokens (GPTBot, ClaudeBot, Google-Extended, PerplexityBot and 11 more), and reports right
now, deterministically, which are allowed, blocked, or partially blocked. It writes nothing,
maintains nothing, tracks no traffic, and needs no account or install.
| What you are trying to do | Where to do it |
|---|---|
Keep a robots.txt current against a continuously updated directory of AI agents, without editing it by hand as new ones appear. |
Known Agents -- an auto-updating, managed robots.txt is its core feature. |
| See an analytics dashboard of bot traffic and AI-referral visits (ChatGPT, Perplexity) hitting your site over time. | Known Agents -- this tool reports nothing about traffic, only what a file currently says. |
Check, right now, exactly what your existing robots.txt allows or blocks for 20 named AI crawlers, with no account and nothing installed. |
This tool -- paste the file and get a deterministic verdict in your browser. |
| Add a CI gate that fails a build the moment GPTBot (or any named crawler) becomes allowed again. | This tool's CLI -- node cli.mjs example.com --fail-if-allowed GPTBot, no account needed. |
Actively block scrapers that ignore robots.txt entirely, not just report what the file says. |
Known Agents' blocking feature, per its own listing -- this tool never blocks anything; it only reads and reports. |
Audit a robots.txt you inherited or just wrote, once, with nothing left running afterward. |
This tool -- it is a check, not a service you install and maintain. |
These are complementary rather than competing: a site running Known Agents for an
auto-updating file and traffic analytics can still use this tool to independently verify,
with no account and no trust in a third-party dashboard, exactly what the resulting
robots.txt allows right now -- and a site that wants no ongoing service at all
can use this tool alone as a one-time or CI-gated check.
They are different shapes of tool and plenty of sites end up using both. Known Agents (formerly Dark Visitors, at knownagents.com) is a hosted, account-based service built around a continuously updated directory of AI agents, crawlers and bots; per its own site and WordPress plugin listing it can generate and keep a site's robots.txt current against that directory, block scrapers that ignore robots.txt, and report analytics on bot traffic and AI-referral traffic (from ChatGPT, Perplexity and similar) over time. This tool does none of that. It is a free, static page (and a zero-dependency Node CLI) that reads the robots.txt and llms.txt text you already have -- pasted, fetched, or read from disk -- against a fixed, publicly documented list of 20 named AI-crawler tokens, and reports right now, deterministically, which are allowed, blocked, or partially blocked. Reach for Known Agents if you want an ongoing, managed directory and a traffic dashboard. Reach for this tool for an immediate, no-account read of the file you already have, or to wire a CI gate around it.
No, and it is not trying to. Known Agents' core feature, per its own documentation, is generating and maintaining a robots.txt against its own continuously updated agent directory, so a site's file stays current as new crawlers appear without anyone editing it by hand. This tool never writes or maintains a file -- it only reads whatever robots.txt you give it, against a fixed snapshot of 20 named tokens, and tells you what that file currently does. If you want a robots.txt that updates itself as new AI agents launch, that is Known Agents' job, not this tool's. If you want to check, right now, exactly what an existing file allows or blocks -- your own, or one you inherited -- that is this tool's job.
Not by this tool. The browser page parses the robots.txt/llms.txt text you paste entirely in your own browser and never sends it anywhere; the CLI only ever contacts the one domain you name, to fetch its robots.txt and llms.txt, and makes no other network call. It keeps no traffic log and has no dashboard -- it reports the current file's rules and nothing about who visits your site. Known Agents' analytics dashboard is a different kind of product: reporting ongoing bot and AI-referral traffic necessarily means it observes requests hitting your site over time, which this tool never does.
No. This site is not affiliated with, endorsed by, or sponsored by Known Agents (formerly Dark Visitors) or its operators. Known Agents and Dark Visitors are named here only to describe what this tool is and is not, based on Known Agents' own public site and WordPress plugin listing; check those for what the service currently offers and how.