Updated July 19, 2026

robots.txt directives reference

robots.txt is a plain-text file at the root of a site that tells crawlers which paths they may request. This sheet lists each directive, the pattern syntax, and the rules that decide which crawler a group applies to.

Directives

A group starts with one or more User-agent lines followed by its rules. The most specific matching User-agent group wins.

Directive	Applies to	Purpose
`User-agent`	Starts a group	Names the crawler the following rules apply to. `*` matches any crawler without its own group.
`Disallow`	Path prefix	Blocks matching paths. An empty value (`Disallow:`) allows the whole site.
`Allow`	Path prefix	Permits a path that a broader Disallow would block. Honored by Google and Bing.
`Sitemap`	Whole site	Absolute URL of a sitemap. Independent of User-agent groups; may appear multiple times.
`Crawl-delay`	Per group	Seconds a crawler should wait between requests. Bing and Yandex honor it; Google ignores it.

Path pattern syntax

Token	Meaning
`/path`	Matches any URL that starts with /path.
`*`	Wildcard: matches any sequence of characters.
`$`	Anchors the match to the end of the URL.

Example

Block two folders for all crawlers and point to a sitemap.

User-agent: *
Disallow: /admin
Disallow: /cart
Allow: /admin/public

# Block AI training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Location and hostingrobots.txt must be at the domain root — https://example.com/robots.txt — and served as text/plain. A file in a subfolder is ignored.

Advisory, not access controlCompliant crawlers obey robots.txt, but it does not enforce anything. Because the file is public, never list sensitive URLs in it — a Disallow tells everyone the path exists. Use authentication to protect private pages.

Disallow is not noindexDisallowing a URL stops crawling but does not guarantee it stays out of search results; it can still be indexed from external links. To keep a page out of results, allow crawling and use a noindex meta tag or header.

References

Questions

Which robots.txt directives are required?

None are strictly required, but a useful group needs at least a User-agent line and one Disallow (which may be empty to allow everything). Sitemap is strongly recommended.

Are robots.txt paths case-sensitive?

Yes. Paths are matched exactly as written, so `/Admin` and `/admin` are different. Directive names like User-agent are case-insensitive.

Do all crawlers support * and $ wildcards?

Google and Bing support `*` and `$` in paths. They are extensions to the original standard, so some older or smaller crawlers may ignore them.