Everything below follows from one sentence in the standard, and the consequence is worth more attention than the rule, because it is invisible from inside the site it affects.
At the origin root, and nowhere else. RFC 9309 §2.3 fixes the location as scheme://host:port/robots.txt. The file is scoped to the origin, so one file governs every path on that host, and no per-directory file is ever consulted.
That is a deliberate design choice, not an oversight. A crawler would otherwise have to fetch and merge a file for every directory it descends into, and the rules could contradict each other at different depths.
The practical result is that a site published under a path prefix inherits a policy file it did not write and cannot edit:
https://acme.github.io/widget/ <- your site https://acme.github.io/robots.txt <- the file that governs it https://acme.github.io/widget/robots.txt <- fetched by nothing
Sitemap auto-discovery, which is the mechanism that enumerates pages. A crawler finds a sitemap by reading the Sitemap: line out of the governing robots.txt. A sitemap advertised only from a subpath is therefore never fetched, and the URLs listed in it are never discovered that way.
There is a second cost that runs the other direction, and people notice it even less: a Disallow: written in a subpath robots.txt is equally inert. A draft section you thought you had excluded from search is not excluded from anything.
Crawling by following links, which is entirely unaffected. Any page linked from a page the crawler already knows is still reached and can still be indexed.
That is precisely why this failure is hard to spot. Some pages do get indexed. Nothing returns an error. Every tool that checks the origin root finds a perfectly healthy robots.txt — it just belongs to someone else. The gap only shows up as a slow, unexplained shortfall between how many pages you published and how many were ever found.
The portfolio that publishes this tool has the same defect, on the same day this page was written, and says so in its own repository rather than quietly. Its catalogue of several hundred free tools is served from a GitHub Pages project subpath; the Sitemap: line it publishes is at an address no crawler requests; and its own deployment check reports the condition as subpath-robots every time it runs.
That is the honest version of a case study: not a claim that the tool found a customer's problem, but a statement that the problem is real, current, and cost the people writing this the same thing it may be costing you.
It is the only one that restores auto-discovery itself, because it moves the site to an origin root where its own robots.txt governs. The alternatives are real and worth doing, but they are per-search-engine pushes rather than a repair to the discovery mechanism:
| Option | What it fixes | What it does not |
|---|---|---|
| Custom domain | Everything here. Your robots.txt governs, auto-discovery works, your Disallow: rules apply. | Costs a domain registration and a DNS record. |
| Direct sitemap submission | Enumeration, for the search engines you submit to. | Needs a verified account per engine; covers only those engines. |
| IndexNow | Fast notification of new and changed URLs, no account needed. | Google does not consume it. Bing, Yandex and Seznam do. |
| A complete internal link graph | Discovery, via the route that still works. | Slower than a sitemap, and needs at least one inbound link from an already-crawled page. |
meta name="robots" | Exclusion, reliably and per page. | Nothing about discovery. It is the answer to the other half of the problem. |
Stated plainly, because a checker that implies more than it measures is worse than none:
--fetch, and only the origin-root file.