Does robots.txt keep a page out of Google?

Not reliably. Disallow stops crawling, but a URL can still be indexed if other sites link to it. To keep a page out of search results, use a noindex meta tag or header and let it be crawled.

Can robots.txt stop AI crawlers?

You can add Disallow rules for AI user agents such as GPTBot and ClaudeBot. Compliant bots respect it, but robots.txt cannot force a crawler to comply.

Does this send my data anywhere?

No. Utilumo's developer tools parse and transform input inside the browser tab. Nothing is uploaded, stored, or logged.

Explainer1 min readUpdated July 19, 2026

What is a robots.txt file?

Short answer

robots.txt is a plain-text file at the root of a site (/robots.txt) that tells crawlers which paths they may or may not fetch. It uses User-agent, Disallow, Allow, and Sitemap directives, and it is advisory — not access control.

A map of what crawlers may fetch

robots.txt is the first place well-behaved crawlers look when they visit a site. It lists rules that say which URL paths a given crawler is allowed to request, so you can keep bots out of admin pages, search results, or duplicate URLs.

User-agent: *
Disallow: /admin
Disallow: /cart

Sitemap: https://example.com/sitemap.xml

A simple robots.txt

Where it goes

The file must live at the root of the domain, at https://example.com/robots.txt, and be served as plain text. A robots.txt in a subfolder is ignored.

Core directives

User-agent — which crawler the following rules apply to (* means all)
Disallow — a path the crawler must not fetch; an empty value allows everything
Allow — a path that overrides a broader Disallow (honored by Google and Bing)
Sitemap — the absolute URL of a sitemap; it applies site-wide
Crawl-delay — seconds between requests (Bing and Yandex; Google ignores it)

It is advisory, not access controlCompliant crawlers obey robots.txt, but it does not block access — anyone can read your robots.txt, so never use it to hide sensitive URLs. Use authentication for that.

Try it: robots.txt GeneratorGenerate a valid robots.txt, including an option to block AI crawlers, locally.Open tool