seo-marketing
Robots.txt Tester
Check pasted Allow and Disallow records for one exact product token and path before publishing a robots.txt change.
Tool workspace
The interactive tool requires JavaScript.
Instructions
Enter one exact product token and a path. A complete HTTP URL is reduced to its path and query before matching.
Inspect the winning line, then repeat with a public path, a blocked path, and each crawler token you plan to support.
Robots.txt Tester evaluates pasted User-agent, Allow, and Disallow records with a small browser-side checker. It combines groups for the same product token, falls back to wildcard groups when no exact token exists, and reports the winning line for the path you enter.
Direct answer
Paste the proposed file, enter an exact crawler product token such as Googlebot, and test a path or complete HTTP URL. The result covers the pasted access rules only: it does not fetch the live file, interpret Sitemap or Crawl-delay, reproduce every crawler extension, or prove crawling or indexing.
How to use this tool
- Paste the proposed robots.txt text, including the User-agent, Allow, and Disallow records that govern the path.
- Enter one exact product token and a path. A complete HTTP URL is reduced to its path and query before matching.
- Inspect the winning line, then repeat with a public path, a blocked path, and each crawler token you plan to support.
How product-token groups are selected
The checker compares the entered product token case-insensitively and exactly. If that token appears in more than one group, their Allow and Disallow records are combined. Wildcard groups are used only when no exact group exists, so NotGooglebot does not silently select a Googlebot group.
A reproducible rule example
With `Disallow: /reports/` and `Allow: /reports/public/` in matching AuditBot groups, `/reports/public/q2.pdf` is allowed because the longer matching rule wins. Equal-length Allow and Disallow matches resolve to Allow. An empty Disallow has no effect; `*` and a final `$` are recognized in path patterns.
What is processed and what is ignored
The tool handler uses the values in this browser and does not fetch a site, submit a URL, or call a crawler API. The page can still load ordinary site resources or configured telemetry, so this is not a whole-page no-network guarantee. Comments and unsupported fields such as Sitemap and Crawl-delay do not affect the decision.
Failure and verification boundaries
Missing User-agent groups and invalid complete URLs produce an error. A complete URL contributes pathname and query but not its fragment. After a local matrix passes, fetch the deployed robots.txt separately, compare exact content and HTTP status, inspect server logs or official crawler tools, and keep indexing questions separate from access-rule matching.
Frequently asked questions
What does browser-side processing mean here?
The checker function reads the three fields and computes the decision in this browser; it has no live robots.txt fetch or search-engine submission step. That statement does not mean every resource on the surrounding web page is offline, so do not paste secrets.
Why did a wildcard group lose to a named group?
An exact product-token match takes precedence. All groups carrying that exact token are combined; wildcard groups are the fallback only when the exact token is absent. Test each product token separately.
Does an Allowed decision mean the URL will be indexed?
No. It means no selected pasted rule blocks this path under this check. Discovery, HTTP response, page directives, canonical handling, content quality, crawler policy, and index selection are separate evidence.
Last updated: