Robots.txt Generator
Robots.txt is a crawl-control file. It tells compliant crawlers which URL paths they may request from a particular host. That makes it useful for keeping crawlers away from administrative areas, duplicate crawl paths, internal search results or other sections that do not need to be fetched.
It is also easy to misuse. One `Disallow: /` rule can block an entire host from a crawler. Blocking a URL does not reliably remove that URL from search results, and it can stop a search engine from seeing a `noindex` directive placed on the blocked page.
The Wild Creek Robots.txt Generator focuses on a small set of widely understood directives: `User-agent`, `Disallow`, `Allow` and `Sitemap`. You can start with a preset, edit the paths, add crawler-specific blocks and download the finished plain-text file.
Robots.txt Generator
Build a robots.txt file using widely supported directives. Start with a safe preset, add crawler-specific rules if needed, then review the output before publishing it.
Quick Start Preset
Sitemap
Use a full HTTP or HTTPS URL. You can submit more than one Sitemap directive if you edit the downloaded file manually.
User-Agent Rules
Choose from major search and AI crawler tokens or enter a custom token. Crawler names and policies can change, so verify an AI crawler's current documentation before relying on a rule.
robots.txt Preview
Configure your rules and generate the file to preview it.
Before You Publish
- Review every Disallow rule. A single
Disallow: /can stop a crawler from accessing the entire host. - Publish the file at the root of the host, for example
https://example.com/robots.txt. - Open the URL in a browser to confirm the file is publicly accessible as plain text.
- For Google, use the robots.txt report in Search Console to test a live file. Google normally discovers robots.txt automatically.
Robots.txt is a crawl-control mechanism. Do not use it as a substitute for noindex when your goal is to prevent a page from appearing in search results.
Available presets:
- Allow Crawling – explicitly allows all paths for all crawlers.
- Block All Crawling – disallows the entire host. Use only when that is genuinely the intended result.
- WordPress Core Pattern – disallows `/wp-admin/` while allowing `/wp-admin/admin-ajax.php`.
- Custom – starts with no predefined crawler rules.
Why the WordPress preset is deliberately small:
WordPress core’s virtual robots.txt output does not block `/wp-content/`, themes, plugins, `wp-includes`, feeds, tag archives or author archives by default. Blocking those areas simply because they look technical can interfere with rendering or hide useful crawlable resources. The generator therefore follows the core WordPress pattern instead of adding a long list of speculative exclusions.
Robots.txt and Indexing Are Different
A robots.txt rule tells a compliant crawler whether it may fetch a URL. It does not tell the search engine whether the URL may appear in search results.
If Google discovers a blocked URL through links or other sources, the URL can sometimes remain known even though Google cannot crawl its content. When the goal is to prevent a page from being indexed, use the appropriate indexing control on a crawlable page, such as a `noindex` directive, rather than assuming a robots block will do the same job.
Crawler-Specific Rules
Different services publish their own crawler tokens. Add a crawler-specific rule only when you know which token the service currently uses and what that token controls.
The generator does not ship with a static “block every AI crawler” button. AI search, training and browsing crawlers are changing quickly, and one vendor may publish several different agents for different purposes. A fixed list becomes stale and can give a false sense of completeness.
For example, if you want to control a specific OpenAI, Google, Anthropic or other crawler, check that provider’s current documentation and enter the exact user-agent token deliberately.
Why Crawl-delay Is Not Included
`Crawl-delay` is not part of Google’s supported robots.txt directives. Other crawlers may support it differently. The generator therefore avoids presenting it as a universal control.
If crawl load is causing a genuine server problem, investigate the crawler involved, caching, server capacity and the controls published by that crawler rather than adding a directive that some bots will ignore.
Publishing the File
A robots.txt file applies only when it is available at the root of the host and protocol it controls. For example:
`https://www.example.com/robots.txt`
A file at `https://www.example.com/folder/robots.txt` does not control the rest of that host.
After publishing:
- Open the robots.txt URL directly and check the exact text served.
- Confirm that important pages and resources are not unintentionally blocked.
- Use the relevant search-engine testing tools for crawler-specific verification.
- For Google, Search Console’s robots.txt report can help test the live file and request a refresh of Google’s cached copy.
Google normally discovers robots.txt automatically. Publishing or refreshing the file does not force immediate crawling or indexing of the site.
Sitemap Directive
You can include an absolute sitemap URL in robots.txt. This gives crawlers another route to discover the sitemap. Use the full URL, including protocol and host.
A sitemap directive does not make the listed sitemap valid and does not guarantee indexing. Validate the sitemap separately and make sure it remains publicly accessible to the crawlers that need it.
Frequently Asked Questions
Can robots.txt remove a page from Google?
Not reliably. Robots.txt controls crawling. If you need a page kept out of Google’s index, use an appropriate indexing control while allowing Google to crawl and see that control.
Should WordPress sites block wp-includes or wp-content?
Not as a blanket default. Search engines may need CSS, JavaScript, images and other resources from WordPress directories to understand and render pages. This generator follows the much smaller WordPress core pattern instead.
Can I block AI crawlers?
You can add specific user-agent rules for crawlers that publish a robots.txt token and honour the protocol. Check the provider’s current documentation first because crawler names and purposes change.
Why is there no Crawl-delay field?
Google does not support Crawl-delay in robots.txt, and support varies across other crawlers. The generator focuses on directives with broader utility.
Where should robots.txt be uploaded?
At the root of the host it controls, such as `https://example.com/robots.txt`. WordPress can also generate a virtual robots.txt response when no physical file overrides it.
Does Disallow mean a URL is bad for SEO?
No. Disallow simply limits crawler access for the specified user-agent. Whether a path should be blocked depends on why the crawler should or should not fetch it.