Build a Robots.txt Path Matrix Before Publishing
AIGClub TeamShare
A robots.txt review is more reliable when it starts with URLs and expected decisions, not with a visual scan of directives. Freeze the proposed text, list representative paths for each crawler product token, record the winning rule, and rerun the same matrix after every edit.
Direct answer
Before publishing, test an exact named product token, the wildcard fallback, a public path, an intentionally blocked path, a longer Allow exception, and a query-sensitive pattern. Treat Sitemap and crawler-specific directives as separate checks. After deployment, fetch the live file and compare its HTTP response and content with the reviewed draft; a local Allowed result is not index evidence.
Freeze one proposed file and its assumptions
Save the exact draft that will be deployed. For a small fixture, use a wildcard rule such as `Disallow: /staging/`, then place `Disallow: /reports/` and `Allow: /reports/public/` in two separate AuditBot groups. The repeated named groups should be evaluated together.
Write down what the checker supports: case-insensitive exact product-token selection, wildcard fallback, Allow and Disallow, `*`, a final `$`, and longest-match selection with Allow winning an equal-length tie. Comments, Sitemap, Crawl-delay, and private crawler dialects do not change its decision.
Run a matrix that can fail
AuditBot with `/reports/private.csv` should be Disallowed by `/reports/`; AuditBot with `/reports/public/q2.pdf` should be Allowed by the longer exception. NotAuditBot must not match AuditBot and should use the wildcard group. `/staging/demo` should therefore be Disallowed for that fallback token.
Add a complete URL such as `https://example.com/docs?preview=1#notes` when query matching matters. Confirm that the reported test path is `/docs?preview=1` and that the fragment is absent. Also include a malformed complete URL and a file without a usable User-agent group so error handling is visible before release.
Keep access rules, sitemap inventory, and indexing separate
A Sitemap line is a discovery hint, not an Allow rule, so verify its protocol, host, and path independently. If you have the XML, inventory its loc values and compare them with the canonical public URLs you intend to expose.
An Allowed decision says only that the selected pasted group does not block that sample path. It says nothing by itself about link discovery, HTTP status, noindex, canonical selection, rendering, content quality, crawl scheduling, or an index decision.
Repeat the matrix against deployment evidence
After release, request the canonical `/robots.txt` URL without credentials. Record the final URL after redirects, HTTP status, content type, cache behavior, and exact response body. Compare that body with the approved fixture rather than assuming the deployment used the latest edit.
Then use server logs or the target crawler's official tools for live behavior. If evidence differs from the local matrix, preserve the input, product token, path, matched line, timestamp, and deployed response before changing another rule.
Frequently asked questions
- Why test NotGooglebot as well as Googlebot?
- It exposes substring-selection bugs. A different token that merely contains the text Googlebot should not inherit Googlebot's named group; without an exact group, it should fall back to `*`.
- Should two Googlebot groups be combined?
- Yes for this RFC 9309-oriented check. Put rules from every group carrying the exact product token into the same decision set, then choose the most specific matching rule.
- What is the minimum record to keep for a failed case?
- Keep the reviewed robots.txt body, entered product token, normalized path, expected decision, actual winning line, tool version or commit, and—after release—the live response timestamp and status.