Updated September 15, 2026

robots.txt matching reference

Use this sheet to interpret the local robots.txt Tester. It explains the engine's exact product-token groups and normalized path matching, with example inputs you can reproduce.

Decision order in this tester

Step	Behavior
1. Parse groups	Adjacent User-agent lines share a group. The next User-agent after an Allow/Disallow line starts another group. Comments, blank lines, and unsupported records do not split groups.
2. Select crawler	Match the product token exactly, ignoring case. Merge all matching specific groups; use all * groups only when no specific group matches.
3. Match paths	Match from the beginning of the normalized path and query, with case preserved. Ignore the URL fragment.
4. Compare rules	Choose the longest normalized matching pattern; Allow wins an equal-length tie. File order cannot make a shorter rule win.
5. Apply the default	If no applicable rule matches, allow the URL by these rules. The exact /robots.txt path is implicitly allowed. Empty or invalid input is shown separately.

What normalized rule length means hereThe tester counts characters in the normalized ASCII pattern, including * and a terminal $. Raw non-ASCII characters become UTF-8 percent escapes; each preserved %HH sequence occupies three characters. Escaped unreserved ASCII is decoded before counting. Only ASCII space and tab are trimmed as syntax whitespace. Unicode spaces in paths, such as U+00A0 and U+3000, remain meaningful path characters even at the end. This is an explicit matching model, not a promise to reproduce every crawler's parser or extensions.

Pattern examples

Each pattern is a Disallow rule in an applicable group with no competing rule. URLs are shown as path plus query.

Pattern	Matches	Does not match
`/private`	`/private/report; /private-public`	`/Private/report; /public/private`
`/private/`	`/private/report`	`/private; /private-public`
`/report$`	`/report; /report#summary (fragment removed)`	`/report/; /report?format=csv`
`/*.pdf$`	`/guide.pdf; /docs/guide.pdf`	`/guide.pdf?download=1; /guide.PDF`
`/search?q=`	`/search?q=coffee`	`/search?page=1&q=coffee`
`/a*b`	`/ab; /a/long/path/b`	`/a/long/path/c`

Encoding distinctions

Rule pattern	Target path	Comparison
`/café`	`/caf%C3%A9`	Matches: both normalize to escaped UTF-8.
`/foo/%62ar`	`/foo/bar`	Matches: %62 is unreserved ASCII b.
`/foo/bar`	`/foo%2Fbar`	Does not match: an escaped slash remains distinct from a path separator.
`/foo%2Fbar`	`/foo%2fbar`	Matches: hexadecimal escape case is normalized.
`/file%2A.txt`	`/file*.txt`	Matches a literal asterisk; %2A is not a wildcard.
`/price%24`	`/price$`	Matches a literal dollar sign; %24 is not an end anchor.
`/search?q=a+b`	`/search?q=a%20b`	Does not match: query parameters are not form-decoded or reordered.
`/tag%23one`	`/tag%23one`	Matches a literal encoded #. An unescaped # in a rule starts a comment.

Merged crawler groups and a wildcard fallback

For Googlebot, /drafts and /preview are disallowed. For Bingbot, only /private is disallowed by this policy.

User-agent: *
Disallow: /private

User-agent: Googlebot
Disallow: /drafts

User-agent: googlebot
Disallow: /preview

Warnings and unsupported input

Input	Tester handling
`Disallow: (empty)`	Ignored with a warning. Does not override other Disallow rules.
`Sitemap; Crawl-delay; Noindex; Host`	Ignored with warnings. No effect on Allow/Disallow matching.
`User-agent: Googlebot/2.1`	Invalid token in this engine. Use Googlebot. Version suffixes, partial-token matching, and crawler-specific typo recovery are outside its scope.
`Disallow: /two words`	Ignored. Encode literal spaces as %20.
`Disallow: /bad%zz`	Ignored because the percent escape is invalid.
`Disallow before any User-agent`	Ignored because it has no valid group.
`500 KiB / 8,192 characters`	Maximum rules size / path-and-query length. Larger input is rejected without a partial result.

URL input behaviorThe tool accepts HTTP(S) URLs or paths beginning with /. Full URLs use the browser URL parser, which can normalize dot segments; the result shows the actual path used for matching. A slash-prefixed path is taken literally, even when it starts with //. Outer ASCII spaces and tabs are trimmed; internal ASCII whitespace, control characters, and backslashes are rejected. Unicode spacing characters in the path are preserved as UTF-8 percent escapes. Fragments are removed, and a query's order, plus signs, and empty ? delimiter are preserved.

A local result has a limited scopeThe tester does not fetch robots.txt, validate hosting, simulate cached copies or HTTP errors, check indexing, or guarantee that a crawler follows the policy. A Disallow is not authentication or a noindex instruction. Review warnings even when the displayed decision is allowed.

Related workflowUse the robots.txt Tester for source-line explanations, the testing guide for a walkthrough, and the robots.txt Generator to prepare a policy. The directives reference covers directive purpose and hosting context.

References

Questions

Are crawler tokens and URL paths both case-sensitive?

No. This tester compares crawler product tokens without case sensitivity, while preserving case for path matching. Googlebot and googlebot select the same group; /Admin and /admin remain different paths.

Is * a regular expression?

No. In a rule, * matches zero or more characters and a final $ anchors the end. Regex operators such as +, parentheses, and square brackets are literal characters here.

Does a specific group inherit User-agent: * rules?

No. Wildcard groups are fallback groups. When an exact specific group is present, this tester uses only the matching specific groups, including when those groups contain no effective rule.

Can I use this as a Googlebot simulator?

It covers the stated matching model, not every Google-specific parser behavior. For example, it rejects versioned User-agent fields instead of accepting them leniently. It also does not model retrieval, caching, or indexing.