Robots.txt Generator

What is robots.txt in SEO?

A robots.txt file is the doorman of your website. Before a crawler fetches a page, it usually checks robots.txt to see whether that page is open to it. The file does not hide pages from people, and it is not security — anyone can read it. Its job is to guide well-behaved crawlers: keep them out of admin areas, login pages, internal search results and other parts of the site that should never appear in search.

In 2026 the file has a second job: controlling AI crawlers. Bots from OpenAI, Anthropic, Google, Perplexity and others fetch pages for AI search answers and model training. With per-bot rules you can let them answer questions from your content, block them entirely, or let them crawl while opting out of training use.

What is robots.txt for, and who uses it?

Site owners use it to protect crawl budget and keep unfinished or private areas out of search. Developers use it to keep staging copies and admin panels away from crawlers. Bloggers use it to stop internal search pages and tag archives from being indexed. And since AI crawlers arrived, publishers use it to decide who may train on their content.

The file follows the robots exclusion protocol, formalised as RFC 9309 in 2022. It is a voluntary standard: honest crawlers obey it, scrapers and bad bots ignore it.

Benefits of a correct robots.txt

A correct file earns its place in four ways:

• Crawl budget saved — crawlers spend their visits on pages that matter instead of admin areas and duplicate URLs.
• Cleaner search results — internal search pages, login screens and staging copies stay out of the index.
• Faster indexing of new pages — with fewer low-value URLs to crawl, important pages get visited sooner.
• AI control — per-bot rules decide which AI crawlers may fetch your content and whether it may be used for training.

Purpose of directives

A robots.txt file is built from a few simple directives:

• User-agent — which crawler the following rules apply to. * means every crawler.
• Allow — a path the crawler may visit. Allow: / opens the whole site.
• Disallow — a path the crawler should not visit, e.g. Disallow: /admin/.
• Crawl-delay — how many seconds a polite crawler should wait between requests. Not all crawlers respect it.
• Sitemap — the address of your XML sitemap, so crawlers can find it without guessing.
• __usage_ai-training — an emerging signal to opt out of AI training use of your content.

Difference between a sitemap and a robots.txt

They do opposite jobs and you usually need both. A robots.txt file tells crawlers what NOT to visit. An XML sitemap tells crawlers what DOES exist and is worth visiting. Robots.txt is a gate; a sitemap is a map.

Put the sitemap address inside robots.txt as a Sitemap: line, and crawlers will find both in one place.

How to make a robots.txt file

Use the generator above: set the rules, copy the result, and save it as a file named exactly robots.txt in your site’s root folder. Then open example.com/robots.txt in a browser to check it loads. Google Search Console shows your robots.txt under Settings, with any errors it found.

Best practices — and common mistakes

Most robots.txt disasters come from a short list of avoidable errors:

• Never write Disallow: / for all crawlers unless you mean to block the whole site — it is the most common way to accidentally de-index a site.
• Do not use robots.txt to hide private content — the file is public, and blocked pages can still appear in results if other sites link to them. Use noindex or a password instead.
• Do not block CSS, JavaScript or image files that Google needs to render the page — blocking them can hurt how the page is understood.
• Always add a Sitemap: line pointing at your sitemap, so crawlers find it without guessing.
• Test before uploading — paste the file into the validator in this tool and check it in Search Console after publishing.