Robots.txt Generator: Complete Guide to Controlling Web Crawlers
Every website needs a robots.txt file — it’s the first thing search engine crawlers look for when they visit your domain. This simple text file acts as a set of instructions telling bots which parts of your site they can access and which they should avoid. Our free robots.txt generator tool lets you visually configure your rules and produce a properly formatted file ready to deploy, without needing to memorize the syntax or worry about formatting errors.
What is Robots.txt?
Robots.txt (the Robots Exclusion Protocol) is a standard used by websites to communicate with web crawlers and other automated agents. Created in 1994 by Martijn Koster, it was formalized as an internet standard (RFC 9309) in September 2022 after nearly three decades as a de facto convention.
The file must be named exactly robots.txt (lowercase) and placed at the root of your domain: https://example.com/robots.txt. Crawlers fetch this file before crawling any other page on your site. If the file doesn’t exist (returns 404), crawlers assume they can access everything. If it exists, they parse the rules and respect them (most legitimate crawlers do, though malicious bots may ignore them).
Important to understand: robots.txt is an advisory protocol, not a security mechanism. It relies on crawler cooperation. It should never be used to hide sensitive information — use authentication, IP restrictions, or server-side access controls for actual security. Robots.txt is for SEO management and server load control.
User-Agent Directives Explained
The User-agent directive specifies which crawler a set of rules applies to. Each block of rules begins with one or more User-agent lines followed by Allow/Disallow directives:
User-agent: *
Disallow: /private/
User-agent: Googlebot
Allow: /private/public-page
Disallow: /private/
Common user-agent values include:
- * — All crawlers (wildcard)
- Googlebot — Google’s main web crawler
- Googlebot-Image — Google Images crawler
- Bingbot — Microsoft Bing crawler
- Slurp — Yahoo crawler (now uses Bing)
- DuckDuckBot — DuckDuckGo crawler
- Baiduspider — Baidu (Chinese search engine)
- YandexBot — Yandex (Russian search engine)
- facebot — Facebook crawler
- Twitterbot — Twitter/X link preview crawler
- GPTBot — OpenAI’s crawler for training data
- CCBot — Common Crawl (dataset for AI training)
Rules are matched by the most specific User-agent block. If both a * block and a Googlebot block exist, Googlebot follows only the Googlebot-specific rules and ignores the wildcard block.
Allow and Disallow Rules
Disallow: /path/ prevents crawlers from accessing any URL starting with that path. Allow: /path/ explicitly permits access, useful for creating exceptions within broader disallow rules.
Key syntax rules:
Disallow: /— Blocks the entire siteDisallow:(empty value) — Allows everything (same as no robots.txt)Disallow: /admin/— Blocks /admin/ and all paths beneath itDisallow: /file.html— Blocks that specific file*wildcard — Matches any sequence (e.g.,Disallow: /*.pdf$blocks all PDFs)$end anchor — Matches end of URL (e.g.,Disallow: /page$blocks /page but not /page/sub)
When Allow and Disallow conflict, the most specific (longest) path wins. If they’re equal length, Allow takes precedence. This lets you create patterns like “block everything in /api/ except /api/public/”:
Allow: /api/public/
Disallow: /api/
Crawl-Delay Directive
The Crawl-delay directive tells crawlers how many seconds to wait between successive requests. This is useful for smaller servers that might be overwhelmed by aggressive crawling:
User-agent: *
Crawl-delay: 10
Important caveats: Google does not support Crawl-delay — use Google Search Console’s “Crawl rate” setting instead. Bing, Yandex, and some other crawlers do respect it. Values can be integers or decimals (e.g., 0.5 for half-second delays). Setting this too high (30+ seconds) effectively stops a crawler from indexing much of your site within reasonable timeframes.
Sitemap Reference
The Sitemap directive tells crawlers where to find your XML sitemap — a structured list of all pages you want indexed. This is placed outside any User-agent block (typically at the end of the file):
Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-posts.xml
Sitemap: https://example.com/sitemap-products.xml
You can include multiple Sitemap directives. The URL must be absolute (include the full https:// domain). While search engines can discover sitemaps through Search Console or by convention (/sitemap.xml), including it in robots.txt ensures all crawlers find it automatically. The Sitemap directive is supported by Google, Bing, Yahoo, and most other search engines.
Common Robots.txt Patterns
Allow everything (default):
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml
Block everything (staging/development sites):
User-agent: *
Disallow: /
WordPress typical configuration:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /*?s=
Disallow: /*&s=
Sitemap: https://example.com/sitemap_index.xml
E-commerce (prevent faceted navigation indexing):
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*&page=
Sitemap: https://store.com/sitemap.xml
Block AI training crawlers (2024+ concern):
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Robots.txt vs Meta Robots vs X-Robots-Tag
These three mechanisms serve different purposes in crawler control:
robots.txt: Controls crawl access at the directory/file level. Prevents the crawler from fetching pages. Does not guarantee exclusion from search indexes (linked pages may still appear with limited information).
Meta robots tag: <meta name=“robots” content=“noindex, nofollow”> in HTML. Controls indexing behavior after a page is crawled. More granular: noindex (don’t index), nofollow (don’t follow links), nosnippet (no snippet in results), noarchive (no cached version).
X-Robots-Tag: HTTP header version of meta robots. Useful for non-HTML files (PDFs, images) and for applying rules without modifying page content. Example: X-Robots-Tag: noindex, nofollow.
Critical interaction: If robots.txt blocks a URL, crawlers can’t see the meta noindex tag on that page. The page might still appear in search results (without snippet) if other pages link to it. To reliably prevent indexing, use noindex but allow crawling — don’t block with robots.txt AND add noindex.
Testing and Validating Robots.txt
After creating your robots.txt, test it before deployment:
- Google Search Console: Use the Robots.txt Tester tool to verify which URLs are blocked for Googlebot
- Bing Webmaster Tools: Includes a robots.txt validator and tester
- Browser check: Visit
https://yoursite.com/robots.txtdirectly to confirm it’s accessible - HTTP status: Ensure it returns 200 OK (not 301 redirect or 404)
- Encoding: File must be UTF-8 encoded plain text (not HTML, not wrapped in tags)
Common mistakes to avoid: placing the file in a subdirectory (it must be at the root), using HTML formatting, blocking CSS/JS files that Google needs for rendering, blocking the sitemap URL itself, and creating overly broad rules that accidentally block important content.
SEO Best Practices for Robots.txt
A well-configured robots.txt improves SEO by directing crawl budget toward your most important pages. “Crawl budget” is the number of pages a search engine will crawl on your site per visit — for large sites, this is a real constraint. Block pages with no SEO value (admin panels, search result pages, session-specific URLs, thin/duplicate content) to focus crawlers on pages you want indexed.
Don’t block resources needed for rendering: Google recommends allowing access to CSS, JavaScript, and images so Googlebot can render pages as users see them. Blocking these can harm your rankings because Google can’t assess the user experience.
Keep your robots.txt simple and maintainable. Overly complex rules with dozens of patterns are hard to debug and easy to break. When in doubt, fewer rules are better — let Google crawl most of your site and use meta noindex for pages you specifically want excluded from results.
Frequently Asked Questions
What is robots.txt?
Robots.txt is a plain text file at your website’s root (example.com/robots.txt) that instructs web crawlers which pages they can access. It’s an advisory protocol — legitimate crawlers respect it, but it’s not a security mechanism.
What are user-agent directives?
User-agent specifies which crawler the rules apply to. “User-agent: *” targets all crawlers. Specific values like “Googlebot” or “Bingbot” let you create crawler-specific rules. The most specific match wins.
What is the difference between Allow and Disallow?
Disallow blocks crawlers from accessing a path. Allow explicitly permits access, useful for exceptions within a broader Disallow. When they conflict, the longest (most specific) path wins; if equal length, Allow takes precedence.
What is crawl-delay?
Crawl-delay specifies seconds between requests. “Crawl-delay: 10” means wait 10 seconds between fetches. Google ignores this (use Search Console instead). Bing and Yandex respect it. Useful for preventing server overload.
How do I add a sitemap to robots.txt?
Add “Sitemap: https://example.com/sitemap.xml” on its own line, typically at the file’s end. It’s not tied to any User-agent block. Multiple Sitemap directives are allowed. URLs must be absolute.
Does robots.txt block pages from appearing in search results?
Not necessarily. It prevents crawling, but pages can still appear in results (without snippets) if other sites link to them. For true indexing prevention, use the “noindex” meta tag and allow crawling so bots can see the noindex directive.
Where should robots.txt be placed?
At the domain root: https://example.com/robots.txt. It applies only to that exact domain and protocol. Subdomains (blog.example.com) and different protocols (http vs https) each need their own file.
What are common robots.txt patterns?
Block admin areas (Disallow: /admin/), search pages (Disallow: /search), duplicate content from filters (Disallow: /*?sort=), staging environments (Disallow: /), and AI crawlers (User-agent: GPTBot, Disallow: /). Allow everything with an empty Disallow value.
Related Tools
- URL Encoder & Decoder — Encode URLs for web use
- Unicode Converter — Handle special characters in text
- Tweet Character Counter — Optimize social media content
- Text to Speech — Make your web content accessible