Robots.txt Generator
Generate a valid robots.txt in seconds. Choose which crawlers to allow or block, set disallowed paths, add a crawl delay, and point bots to your sitemap — then copy the result.
User-agent: * Disallow: /admin Disallow: /cart Disallow: /checkout Disallow: /account Sitemap: https://example.com/sitemap.xml
What robots.txt controls
robots.txt is the first file most crawlers request. It tells well-behaved bots which paths they may crawl, which to skip, and where your sitemap lives. Used correctly it keeps crawl budget on your important pages and out of admin, cart, and search-results URLs.
Built for modern crawlers
Beyond Googlebot and Bingbot, this generator lets you write rules for AI crawlers like GPTBot, ClaudeBot, and PerplexityBot — useful if you want to allow or restrict how your content is used by answer engines.
Handle with care
robots.txt is a crawling directive, not a security or privacy control — disallowed URLs can still be indexed if linked elsewhere, and the file is publicly visible. To keep a page out of search results entirely, use a noindex meta tag or proper authentication instead.
Frequently asked questions
Where does robots.txt go?
At the root of your domain, exactly at https://yourdomain.com/robots.txt. Crawlers only check that location.
Does Disallow remove a page from Google?
No. Disallow stops crawling, but a blocked URL can still appear in results if other sites link to it. To remove a page from the index, allow crawling and add a noindex tag, or require authentication.
Can I block AI crawlers like GPTBot?
Yes. Add a rule targeting the crawler's user-agent (for example GPTBot, ClaudeBot, or PerplexityBot) with the paths you want to disallow. Compliance depends on the crawler honoring robots.txt.
Should I include my sitemap?
Yes — adding a Sitemap directive helps search engines discover all your URLs. You can list more than one sitemap.
Stop doing SEO by hand
These tools handle one step. Essel runs the whole pipeline — it researches, writes, optimizes, and publishes SEO content autonomously, so you don't have to.
Try Essel free