Skip to content
toolsbox

Robots.txt Generator

Build a valid robots.txt visually — presets, AI bot rules and a built-in path tester.

  • Free
  • No sign-up
  • Runs in your browser

What is a robots.txt generator?

A robots.txt generator builds the plain-text file that tells crawlers which parts of your site they may request. Instead of typing directives by hand and hoping the syntax is right, you pick a default policy, add rules per user-agent group, list your sitemaps and watch the finished file update as you go. Everything runs in your browser, so nothing you type is uploaded.

The file lives at the root of a host, for example example.com/robots.txt, and applies only to that exact protocol, host and port. A file on www.example.com does not cover shop.example.com, and a robots.txt placed in a subfolder is ignored. Each group starts with one or more User-agent lines, followed by Allow and Disallow rules that match URL paths.

This generator validates the output against RFC 9309, the robots.txt standard published in 2022. It flags rules that appear before any User-agent line, paths that do not start with a slash, unknown directives and duplicate groups. A built-in path tester lets you pick a crawler and a URL path and see whether it would be allowed or blocked, along with the exact rule that decided it.

How to create a robots.txt file

  1. 1

    Choose a default policy

    Start with allow all or disallow all for every crawler. Most public sites should allow all and then block specific paths; disallow all suits staging environments.

  2. 2

    Apply a preset or build groups

    Load a preset for WordPress, Shopify or Next.js, block AI training bots, allow AI search bots or block everything for staging. Then add or edit user-agent groups with their own Allow and Disallow paths.

  3. 3

    Add your sitemap URLs

    Enter the full absolute URL of each XML sitemap or sitemap index. Sitemap lines sit outside any group and apply to every crawler that reads the file.

  4. 4

    Test important paths

    Use the path tester to check key URLs such as product pages, /wp-admin/ or /api/ against Googlebot, Bingbot or GPTBot. Fix any rule that blocks something you need crawled.

  5. 5

    Copy or download and deploy

    Download robots.txt and upload it to the root of your domain, or copy the text into your platform's settings. Then run the AI Crawler Checker on the live site to confirm crawlers read it as expected.

When to use a robots.txt generator

Launching a new site

Create a clean file that keeps crawlers out of cart, search-results and admin paths while pointing them at your sitemap from day one.

Controlling AI crawlers

Block training crawlers such as GPTBot, ClaudeBot, CCBot and Google-Extended while leaving AI search bots like OAI-SearchBot and PerplexityBot open, so you can still be cited in AI answers.

Cutting crawl waste on large sites

Faceted navigation, sort parameters and internal search pages can generate millions of near-duplicate URLs. Wildcard Disallow rules keep crawlers focused on pages that matter.

Protecting a staging environment

The staging preset blocks every crawler with a single Disallow: / rule, useful as one layer alongside password protection.

Testing changes before they go live

Paste in your current rules, edit them and run the path tester before deploying, so a single misplaced slash never blocks your whole blog.

How crawlers read robots.txt, and common mistakes

A crawler obeys only the most specific group that names it and ignores the rest. If you add a group for Googlebot, Googlebot stops reading the * group entirely, so any shared rules must be repeated in its group. Within a group, the longest matching path wins; * matches any sequence of characters and $ anchors the end of the URL. When an Allow and a Disallow rule match with the same length, Google applies the less restrictive Allow.

robots.txt controls crawling, not indexing. Google can still index a disallowed URL without its content if other pages link to it, and it will show the bare URL in results. To keep a page out of search, allow crawling and use a noindex meta tag or X-Robots-Tag header, because Google cannot see a noindex on a page it is not allowed to fetch.

Google ignores crawl-delay, so the generator lets you set it for crawlers that honor it, such as Bingbot, but labels it accordingly. Google also reads only the first 500 KiB of a robots.txt file and generally caches it for up to 24 hours, so changes are not instant. Never block CSS or JavaScript files your pages need to render, since Google renders pages to evaluate them.

On Shopify, robots.txt is generated by the platform and customized through the robots.txt.liquid theme template, so use the preset as a reference for the rules you add there. On Next.js, you can serve the output as a static file in the public folder or recreate it in an app/robots.ts route.

Frequently asked questions

Is the robots.txt generator free?

Yes, the robots.txt generator is completely free, with no account and no usage limit. You can build as many files as you need, use every preset, run the path tester and download the result. It is designed for site owners, developers and SEO teams who want a valid file quickly without memorizing syntax.

Is anything I enter sent to your server?

No, the generator runs entirely in your browser, and nothing you type is uploaded or stored on our servers. Rules, paths, sitemap URLs and test results stay on your device until you copy or download the file. If you want to test a live site's robots.txt, the AI Crawler Checker fetches the public file from the domain you enter.

Does Disallow stop a page from appearing in Google?

No, Disallow only stops crawling, and Google can still index a blocked URL if other pages link to it. The result usually appears without a description. To remove a page from search, let Google crawl it and add a noindex robots meta tag or an X-Robots-Tag HTTP header instead.

Does Google support crawl-delay?

No, Google ignores the crawl-delay directive. Googlebot adjusts its crawl rate automatically based on how quickly your server responds and slows down when it sees errors such as 500 or 503 responses. Some other crawlers, including Bingbot, do honor crawl-delay, which is why the generator still lets you set it per group.

Should I block AI crawlers in robots.txt?

It depends on whether you want AI visibility. Blocking training crawlers such as GPTBot or CCBot opts you out of future training collection, while blocking AI search crawlers such as OAI-SearchBot or PerplexityBot removes you from their cited answers. The presets let you block training bots and allow search bots separately.

Where do I upload the robots.txt file?

Upload robots.txt to the root of each host, so it loads at a URL such as https://example.com/robots.txt. Every subdomain and protocol needs its own file. Files in subfolders are ignored by crawlers, and the filename must be lowercase. After deploying, open the URL in a browser to confirm it returns a 200 status.

Related tools

More SEO tools →