How to create a robots.txt file

You launch a new site, and a week later Google has indexed your admin login page, three test pages and a folder called old-old. A five-line text file would have prevented most of that. Here’s how to write one, and how to check it before it goes live.

What is robots.txt

robots.txt is a plain text file at the root of a website that tells crawlers which paths they may and may not fetch. It follows the Robots Exclusion Protocol, which has been in use since 1994 and was published as RFC 9309 in 2022.

Before a crawler such as Googlebot fetches pages on your site, it requests https://yoursite.com/robots.txt and follows the rules it finds. If the file doesn’t exist, the crawler assumes every page is allowed.

The file is a request, not access control. Googlebot, Bingbot and other well-behaved crawlers honor it, but nothing forces a bot to. Think of it as a “please keep off the grass” sign, not a fence. To keep something private, put it behind authentication or block it on the server.

Where to put robots.txt

The file must be called robots.txt, in lowercase, and sit at the root of the host:

https://www.example.com/robots.txt        correct
https://www.example.com/pages/robots.txt  wrong location, ignored
https://www.example.com/Robots.txt        wrong case, ignored

Each host needs its own file. www.example.com/robots.txt doesn’t apply to blog.example.com, so a blog on its own subdomain needs a separate robots.txt.

Robots.txt syntax and directives

A file is made of groups. Each group starts with one or more User-agent lines naming the crawlers it applies to, followed by Allow and Disallow rules. Lines starting with # are comments.

User-agent

User-agent names the crawler. * matches every crawler that doesn’t have a more specific group of its own.

User-agent: *          # every crawler
User-agent: Googlebot  # Google Search
User-agent: Bingbot    # Bing
User-agent: GPTBot     # OpenAI

These are alternatives, not one group. Several User-agent lines in a row share the rules that follow them.

Disallow

Disallow tells the crawler not to fetch paths that start with the given value. An empty value blocks nothing.

Disallow: /admin/        # the /admin/ directory
Disallow: /private.html  # a single page
Disallow:                # nothing is blocked

Allow

Allow permits a path inside a disallowed one. When rules conflict, Google and Bing use the most specific (longest) matching rule, and Allow wins a tie.

User-agent: *
Disallow: /private/
Allow: /private/public-page.html

Here everything under /private/ is blocked except public-page.html.

Sitemap

Sitemap points crawlers to your XML sitemap. It takes a full URL, can appear anywhere in the file, and isn’t tied to a group.

Sitemap: https://www.example.com/sitemap.xml
Sitemap: https://www.example.com/sitemap-blog.xml

Crawl-delay

Crawl-delay asks a crawler to wait a number of seconds between requests. Bing and Yandex respect it. Google ignores it and sets its crawl rate automatically based on how your server responds.

User-agent: Bingbot
Crawl-delay: 10

Common configurations

Allow all crawlers

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

Block all crawlers

This is what you want on a staging site, so a half-finished redesign doesn’t show up in search results:

User-agent: *
Disallow: /

Make sure it never reaches production. Search engines will stop crawling the site, and its pages will slowly drop out of results. This happens more often than anyone likes to admit, usually right after a big launch.

Block AI crawlers

To keep AI training crawlers out while still allowing search indexing, give each one its own group. Google-Extended isn’t a separate crawler: it’s a token that controls whether Google may use your content for Gemini, without affecting Google Search.

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

A typical production site

Block admin areas, API endpoints, internal search and account pages, and allow everything else:

User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /search?
Disallow: /cart/
Disallow: /account/
Disallow: /*.json$

Sitemap: https://www.example.com/sitemap.xml

Wildcard patterns

Google, Bing and RFC 9309 support two special characters in paths:

• * matches any sequence of characters. Disallow: /folder*/page blocks /folder1/page and /folder-abc/page.
• $ marks the end of the URL. Disallow: /*.pdf$ blocks URLs ending in .pdf but not /pdf-guide/.

# Block all PDF files
Disallow: /*.pdf$

# Block all URLs with a query string
Disallow: /*?

# Block Word and Excel files
Disallow: /*.doc$
Disallow: /*.xls$

What robots.txt means for SEO

• Blocked pages aren’t crawled. Disallow a page you want ranked and search engines can no longer read it.
• Disallow isn’t noindex. A blocked URL can still appear in results if other pages link to it. To keep a page out of the index, allow crawling and add a noindex meta tag, since a crawler can’t see the tag on a page it isn’t allowed to fetch.
• Crawl budget matters on large sites. Blocking low-value URLs such as filters, session IDs and duplicate pages lets crawlers spend their time on the pages that matter.
• List your sitemap. A Sitemap line helps crawlers find new and deeply nested pages.

How to test robots.txt

• Google Search Console has a robots.txt report that shows which file Google fetched, when, and any parsing problems. The URL Inspection tool tells you whether a specific URL is blocked.
• Bing Webmaster Tools includes a robots.txt tester.
• The robots.txt generator and analyzer works before you deploy. Paste a file in Analyze mode to see its groups and sitemaps and get warnings for unknown directives, rules outside a group, duplicate user-agents, invalid crawl delays and a missing sitemap. Then test a path such as /admin/dashboard against a crawler to see whether it’s allowed or blocked, and which rule decided it.

How to build a file with the generator

1. Open the robots.txt tool in Generate mode.
2. Set the user-agent for the first block, typing a name or picking a preset such as Googlebot, GPTBot or CCBot. Add Allow or Disallow rules with Add rule, and an optional crawl delay.
3. Use Add user-agent block for more crawlers, and enter your sitemap URL.
4. Select Generate, then Copy the result into a robots.txt file at your site root.

Common mistakes

• Getting the trailing slash wrong. Disallow: /admin blocks /admin, /admin/ and /admin-panel. Disallow: /admin/ only blocks paths under /admin/.
• Blocking CSS and JavaScript. Google renders pages like a browser, so it needs these files to understand your content.
• Listing secret paths. Anyone can read your robots.txt, so naming a hidden directory there tells everyone it exists.
• Shipping staging rules. A Disallow: / that was right on staging will remove your site from search if it reaches production.