Does the first matching robots.txt line win?

No. This tester compares all matching rules in the selected groups, chooses the longest normalized pattern, and prefers Allow for an equal-length tie. The reference explains what normalization counts.

Can I paste a website URL to fetch its robots.txt?

No. Paste the file contents yourself. The URL field is only a local matching input; the tester does not contact that site.

Why does Googlebot use different rules from another crawler?

An exact Googlebot group takes precedence over wildcard fallback in this tester. A different token uses its own exact group if present, otherwise the wildcard groups. Duplicate matching groups are merged.

Does an empty Disallow cancel an earlier Disallow?

No. Empty rules are ignored. An empty Disallow can leave a group unrestricted only when no other applicable non-empty rule restricts the URL.

Tutorial3 min readUpdated September 15, 2026

How to test robots.txt rules for a URL

Short answer

Paste the policy into the robots.txt Tester, enter a URL or slash-prefixed path, and choose a crawler product token such as Googlebot. Read the selected groups and winning source line, then review warnings. The result explains the pasted rules; it does not verify a live file or indexing status.

Start with the policy you intend to publish

A local test helps you find a mistaken path, unexpected crawler group, or missing exception before publishing. Copy the complete file into the robots.txt Tester. Include the User-agent lines: an isolated Disallow line has no group and is ignored with a warning.

Paste the complete rulesUse Load example for a starting policy, or paste your own plain-text robots.txt. The tester accepts up to 500 KiB and reports ignored lines.
Enter the URL or pathTry an HTTP(S) URL such as https://example.com/private/report, or just /private/report. The visible matched path includes the query string and excludes any fragment.
Choose the crawler tokenEnter Googlebot, Bingbot, another token made of letters, hyphens and underscores, or * for wildcard groups. A full browser User-Agent string is not accepted.
Trace the resultCheck which groups were selected and which source line won. Expand All matching rules to compare competing patterns, then review every warning before using the result.
Test nearby URLsChange case, add or remove a trailing slash, and try a query string. Test the intended exception as well as the URL you want to restrict.

Try it: robots.txt TesterCompare a pasted policy with one URL at a time; editing an input updates the explanation immediately.Open tool

Example: allow a public folder inside a restricted one

User-agent: *
Disallow: /private
Allow: /private/public

Two overlapping rules for crawlers without a specific group

/private/report is disallowed by line 2.
/private/public/report is allowed by the longer pattern on line 3.
/private-public still matches /private, because these are prefixes rather than folder names.
/Private/report does not match the lowercase patterns. Path matching is case-sensitive.

A specific crawler group changes the rules in play

User-agent: *
Disallow: /private

User-agent: Googlebot
Disallow: /drafts

Googlebot uses its own group

For Googlebot, /drafts is restricted by its selected group; /private has no matching restriction. The wildcard group is a fallback, so its rules are not added to a matching specific group. Multiple groups with the same product token are combined. An empty specific group also prevents wildcard fallback.

Queries, fragments, and escaped characters

With Disallow: /*.pdf$, /manual.pdf matches but /manual.pdf?download=1 does not: $ reaches the end of the path and query. /manual.pdf#page=2 matches because fragments are excluded. Use the matching reference for examples of %2F, literal asterisks, and UTF-8 paths.

Use warnings to find a policy the tester cannot interpret

The result considers only accepted User-agent, Allow, and Disallow lines. Sitemap, Crawl-delay, Noindex, and other records produce warnings and do not affect the decision. Empty Allow or Disallow values are ignored; they do not erase other restrictions. A file with no valid group receives an invalid-input result.

Know the engine's scopeThis tester uses case-insensitive exact product-token selection and the documented matching model in the reference. It does not reproduce crawler-specific typo correction, versioned User-agent parsing, or every crawler extension. It accepts 500 KiB of rules and an 8,192-character path/query; oversized input is rejected rather than partially evaluated.

A crawling decision is not an indexing resultAn allowed result does not mean a crawler will visit or index the page. A disallowed URL may still appear in search results. The tool does not fetch your published robots.txt or model redirects, HTTP errors, caches, noindex directives, or authentication. robots.txt itself does not enforce access control.

After correcting the policy, use the robots.txt Generator to prepare a file, and check the directives reference for hosting and directive context. Verify the actual file served for the intended scheme, hostname, and port through your deployment workflow.

Local testingThe pasted rules, URL, and crawler token stay in this browser tab's memory. The tester does not fetch the entered URL, upload inputs, or save them between visits. Copy test result copies only when you choose it.