Robots.txt Generator
Generate a valid robots.txt file for your website with allow/disallow rules, crawl-delay, and sitemap. Free and easy — no coding needed.
How to use the Robots.txt Generator
- Set the User-agent (use * to target all crawlers), then add any paths you want to block under Disallow paths.
- Add any exceptions under Allow paths — useful for allowing one file inside an otherwise-blocked folder.
- Optionally set a Crawl-delay and add your Sitemap URL so crawlers can find it.
- Copy the generated robots.txt, or download it and upload it to the root of your website.
Frequently asked questions
Where does robots.txt need to go?
It must be placed at the root of your domain, e.g. https://example.com/robots.txt — it won’t be recognized anywhere else, including subfolders.
What do the “Allow everything” and “Block everything” presets do?
“Allow everything” clears all rules so crawlers can access your whole site. “Block everything” adds a Disallow: / rule to stop compliant crawlers from indexing any page.
Does every search engine respect Crawl-delay?
No. Google ignores the Crawl-delay directive entirely (use Search Console settings instead), though Bing and some other crawlers do honor it.
Is this file generated or checked online?
No. The robots.txt text is built entirely in your browser from the fields you fill in — nothing is uploaded or sent to a server.