Create robots.txt File Online Free

About this tool

This tool does three jobs that usually need three separate tools. The Generate tab builds a properly formatted robots.txt file without writing any code — choose the bots you want to configure, add the paths to block or allow, and include your sitemap URL, with a live preview that updates as you type. The AI Crawlers tab turns the confusing world of GPTBot, ClaudeBot, PerplexityBot and Google-Extended into a set of deliberate choices, telling you in plain English what each block actually costs you. The Test & Validate tab checks any URL against any site's rules and shows you the exact line that decides the outcome.

The builder, the AI crawler panel and the rule matcher all run inside your browser — nothing you type is uploaded. The one exception is the "Load a live site's robots.txt" button: browsers are not allowed to read another domain's files directly, so that request passes through our server. Only the domain you enter is sent, and the fetched file is cached for five minutes.

Reviewed by the ToolBrainy Team · Rule matching runs in your browser · Last updated August 2026

How to use

Why use it

Use cases

What is robots.txt?

Robots.txt is a small plain-text file that lives in the root folder of your website, at an address like https://yourdomain.com/robots.txt. Its job is to tell search-engine crawlers which parts of your site they are welcome to visit and which parts they should leave alone. When a crawler such as Googlebot arrives, checking this file is usually the first thing it does.

The file follows the Robots Exclusion Protocol, a long-standing agreement that the major search engines respect. Each robots.txt file is built from one or more groups of rules. A group starts by naming a crawler with a User-agent line, then lists the paths that crawler may or may not request. It is a set of polite instructions rather than a lock, so it works best for guiding well-behaved bots, not for keeping secrets.

How robots.txt works

Before a crawler fetches your pages, it downloads your robots.txt file and reads the rules that apply to it. Each rule is a single line made up of a directive and a value. Here are the directives you will actually use:

• User-agent — names the crawler a group of rules applies to. * means every bot.
• Disallow — a path that the named crawler should not request. Disallow: / blocks the whole site.
• Allow — an exception that opens a path back up, even inside a blocked folder.
• Sitemap — the full URL of your XML sitemap so crawlers can find your pages faster.

A crawler matches paths from left to right, and the most specific rule wins. In the example below, the whole /private/ folder is closed off, but one file inside it stays open.

User-agent: *
Disallow: /private/
Allow: /private/public-notice.html

Sitemap: https://yourdomain.com/sitemap.xml

One line worth remembering: robots.txt controls crawling, not indexing. It asks a bot not to fetch a page. If you need a page kept out of search results, use a noindex meta tag instead, which we explain further down.

Why robots.txt matters for SEO

Search engines give every site a rough crawl budget — the number of pages they will fetch in a given period. A tidy robots.txt file helps them spend that budget on the pages you actually want ranked. This matters more as a site grows and picks up thin or duplicate URLs that add nothing to search results.

• Focus crawl budget. Keep bots away from cart, filter and internal search pages so they reach your real content sooner.
• Reduce server load. Blocking heavy or endless URL patterns stops aggressive crawling from slowing your site down.
• Point to your sitemap. A sitemap line helps engines discover new and updated pages quickly.
• Cut duplicate crawling. Steering bots away from tag archives and print versions reduces wasted requests.

Used carefully, robots.txt is a quiet SEO win. Used carelessly, a single wrong line can hide your best pages, so it pays to check your rules before you publish them.

AI crawlers and robots.txt

Robots.txt used to be a conversation with search engines. It is now also how you talk to AI companies. OpenAI, Anthropic, Google, Apple, Meta, Perplexity and others all publish user-agent names and say they honour robots.txt, which makes this small file the main lever site owners have over how their content is used.

The mistake almost everyone makes is treating "AI bots" as one group and blocking all of them. They are not one group. They fall into two categories that pull in opposite directions.

Training crawlers: blocking these costs you nothing

These collect text to train models. They do not send you visitors, and blocking them has no effect on how you rank or whether you get cited anywhere. If your only concern is your work being used as training data, these are the ones to block.

• GPTBot — OpenAI's training crawler. Blocking it does not stop ChatGPT citing you.
• ClaudeBot — Anthropic's training crawler, separate from the agents that fetch pages for users.
• Google-Extended — not a crawler at all. It is a switch telling Google whether content Googlebot already fetched may be used to train and ground Gemini. Blocking it has zero effect on Google Search rankings or indexing. This is the safest AI block a site owner can make, and the one most people are afraid to touch.
• Applebot-Extended — the same idea for Apple Intelligence. Siri and Spotlight use the separate Applebot crawler, so blocking the "Extended" token does not remove you from Apple's search features.
• CCBot — Common Crawl. It does not train a model itself, it builds the open dataset that a large number of AI companies train on. One rule here cuts off many models at once, which makes it the highest-leverage block on this list.
• Bytespider, Meta-ExternalAgent, Amazonbot, cohere-ai, Diffbot, AI2Bot — training crawlers for ByteDance, Meta, Amazon, Cohere, Diffbot and academic research respectively.

Answer engines: blocking these costs you traffic

These index or fetch your pages so that a real person gets your content, with a visible link back to you. They are a referral channel, and an increasingly important one. Blocking them is the AI equivalent of blocking Googlebot.

• OAI-SearchBot — indexes pages so ChatGPT search can surface and link to them. Distinct from GPTBot.
• ChatGPT-User — fetches a page live when someone asks ChatGPT about that specific URL. Block it and anyone pasting your link into ChatGPT gets an error.
• Claude-SearchBot and Claude-User — the same split for Anthropic: one indexes for citations, one fetches on a user's request.
• PerplexityBot and Perplexity-User — Perplexity shows prominent clickable sources, so citations here convert to visits.
• YouBot — You.com's answer index.

The rule of thumb

If a bot's job is to learn from your content, blocking it is free. If its job is to show your content to a person, blocking it costs you a visitor. That is why the AI Crawlers tab above offers "Block training, keep answer engines" as its recommended preset — it is the setting most publishers actually want, and almost no tool offers it in one click.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Disallow:

Two caveats worth knowing. First, this is voluntary — robots.txt is an instruction, not a lock, and a bot that ignores it faces no technical barrier. If content must not be taken, put it behind a login. Second, blocking a training crawler does not remove anything a model already learned; it only applies to future crawls.

Common robots.txt mistakes

Most robots.txt problems come from a few repeat offenders. Watch out for these:

• Blocking the entire site by accident. A stray Disallow: / left over from a staging setup can wipe you out of search.
• Expecting it to hide private data. Anyone can open your robots.txt, so listing a secret folder there simply advertises it. Use passwords or server rules for anything sensitive.
• Blocking CSS and JavaScript. Google needs these files to render your pages. Blocking /wp-includes/ or asset folders can hurt how your pages are understood.
• Assuming Disallow removes a page. A blocked URL can still appear in results if other sites link to it. Removing a page from search needs noindex, not Disallow.
• Case and slash slips. Paths are case-sensitive, and /Folder is not /folder. A trailing slash also changes the match.
• Putting the file in the wrong place. It only works at the domain root, never in a subfolder.

Robots.txt examples

Real files are usually short. Here are the patterns you will reach for most often — copy one as a starting point and adjust the paths to match your site.

User-agent: *
Disallow:

User-agent: *
Disallow: /

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /search/

Sitemap: https://yourdomain.com/sitemap.xml

User-agent: Googlebot
Disallow:

User-agent: AhrefsBot
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow:

WordPress robots.txt example

WordPress creates a virtual robots.txt automatically, so a fresh install already has one even if there is no file on the server. It is basic, but for most sites it is enough. If you want more control, you can replace it with your own rules. A sensible starting point for a WordPress site looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourdomain.com/wp-sitemap.xml

Notice that /wp-admin/ is blocked while admin-ajax.php stays open, because some themes and plugins rely on it to load content. Avoid blocking /wp-includes/ or your uploads folder — doing so can stop Google from rendering your pages properly. If you use an SEO plugin like Yoast, Rank Math or All in One SEO, each one has a built-in robots.txt editor, so you can paste these rules straight into the dashboard without touching FTP.

Robots.txt best practices

• Keep it simple. Only add rules you understand. A short, clear file is easier to maintain and less likely to break.
• Always include your sitemap. One Sitemap line helps search engines find everything worth crawling.
• Never block CSS, JS or images you want rendered. Blocking them can change how your pages are judged.
• Use noindex for pages you want out of search. Reach for robots.txt to manage crawling, not to remove indexed pages.
• Match paths carefully. Remember that rules are case-sensitive and that a trailing slash matters.
• Test before and after every change. A quick check saves you from a costly mistake.
• Review it now and then. As your site changes, old rules can quietly block the wrong things.

How to upload robots.txt

Once you have generated your rules above, getting the file live takes only a minute. The exact steps depend on how you manage your site:

• Save the file. Copy your rules into a new plain-text file named exactly robots.txt — all lowercase, nothing extra.
• Upload it to the root. Using FTP or your hosting file manager, place it in the top-level folder (often public_html) so it opens at yourdomain.com/robots.txt.
• On WordPress, use a plugin. Yoast, Rank Math and AIOSEO each include a robots.txt editor, so you can paste and save your rules without any file transfer.
• Confirm it works. Visit yourdomain.com/robots.txt in your browser. If your rules show up, you are done.

How to test robots.txt

Google retired its robots.txt Tester in December 2023. The report that replaced it, inside Search Console, only shows the last version Google fetched for a property you have verified — it dropped the interactive "does this URL match these rules" check entirely, and you can no longer test another site's file at all. That is the gap the Test & Validate tab above fills.

Testing takes seconds and catches mistakes before search engines act on them. A few ways to check your file:

• Use the tester above. Paste your rules or fetch any live domain, drop in a list of URLs, pick a crawler, and every URL is marked allowed or blocked with the exact line that decided it. It follows Google's matching rules: the longest matching path wins, and a tie goes to Allow.
• Open it directly. Load yourdomain.com/robots.txt and read the rules. It should return the file, not a 404.
• Use Google Search Console. The URL Inspection tool still tells you whether one specific page on your own verified property is blocked, and the robots.txt report shows fetch errors.
• Check a competitor's file. Fetching another site's robots.txt shows you which sections they keep out of search and which AI crawlers they allow — often a faster read on their strategy than any tool.
• Re-check after every edit. Search engines cache the file, so confirm the new version is live and test the paths you care about.

Frequently asked questions

Related tools