robots.txt Tester
Test a URL against your rules. See the group and rule behind the result.
robots.txt testing workspace
Paste a policy, choose a crawler, and see which rule decides a URL.
robots.txt rules
Test a URL
Use a token such as Googlebot or Bingbot, without a version or full HTTP User-Agent string. * tests wildcard groups.
Paste robots.txt to begin
Paste robots.txt rules to test a URL. Comments alone do not define a policy.
Scope: exact product-token groups, merged duplicates, wildcard fallback, case-sensitive paths, * and $, and percent-encoding normalization. Other directives are ignored with warnings. A result describes the pasted rules; it does not check a live site, indexing status, or whether a crawler obeys them. Limit: 500 KiB of rules and an 8,192-character path/query.
Questions
Does Allowed mean Google will index my page?
No. This only tests crawl rules in pasted robots.txt. Indexing also depends on other signals and directives; the tester does not fetch the page.
Why does Allow win over Disallow?
The most specific matching path rule wins. If matching Allow and Disallow rules have equal specificity, Allow wins. A more specific Disallow can still block a URL.
Can I test a live site URL?
Enter the URL to test its path, but paste the robots.txt separately. The tester makes no network request to that site.