SEO
robots.txt Generator
Create a robots.txt with templates for WordPress, online shops or AI crawlers – and test right away whether an address is allowed or blocked for Googlebot & co.
- Runs entirely in your browser
- No data upload
- No sign-up
- Free
Template
Sitemaps absolute URLs
robots.txt
Test a URL
Checks
Group
Crawl-delay ignored by Google
Guide
How it works
- Choose a template
Start with a template – such as “WordPress” or “Block AI crawlers” – or build your rules yourself.
- Adjust the rules
Specify for which crawlers (user-agent) which paths are blocked (Disallow) or allowed (Allow), and add your sitemap.
- Test addresses
Use the tester to check whether a specific address can be accessed by Googlebot or another crawler – and which rule applies.
- Upload the file
Download the robots.txt and place it in the root directory of your domain, i.e. at https://www.your-domain.com/robots.txt.
FAQ
Frequently asked questions
Does robots.txt prevent a page from appearing on Google?
Not reliably. Disallow only prohibits crawling. If the page is linked elsewhere, Google may still show the address without content. To reliably keep a page out of the index, use <meta name="robots" content="noindex"> instead – and do not block the page in robots.txt, otherwise Google never sees the noindex.
What do * and $ mean in paths?
* stands for any number of characters, $ for the end of the address. Disallow: /*.pdf$ therefore blocks all addresses ending in .pdf. Without $, a rule applies to all addresses that start with the path.
Which rule applies if Allow and Disallow contradict each other?
Google applies the longest (most specific) matching rule. If both are the same length, Allow wins. The tester calculates exactly the same way.
Do AI crawlers respect robots.txt?
Well-known providers such as OpenAI (GPTBot), Anthropic (ClaudeBot), Google (Google-Extended) or Common Crawl (CCBot) state that they respect robots.txt. However, the file is a request, not technical protection – dubious crawlers can ignore it.
Is crawl-delay respected?
Google ignores crawl-delay. Bing and some other search engines take the value into account.