Robots.txt Generator
Build a robots.txt file from a preset or your own rules: which crawlers may fetch which paths, plus your sitemap URLs. Review the result, then copy or download it.
How to Build a robots.txt File
A robots.txt file tells crawlers which URLs on your site they may fetch. The format is defined in RFC 9309, and major search engines follow it. This generator writes the file from the groups and rules you set up, and warns about entries crawlers are likely to ignore or misread.
Using the Generator
- Start from a preset or the default (allow everything). A preset replaces the current rules, and you can edit them afterwards.
- Add a group per crawler you want to treat differently. Type
*for "all crawlers", or pick a name from the suggestions, such asGooglebotorGPTBot. - Add Allow and Disallow rules. Paths start with
/and are case-sensitive.*matches any characters and$marks the end of the URL. - Add your sitemap URLs as full addresses, for example
https://www.example.com/sitemap.xml. - Generate, review the warnings, then copy or download the file.
Example: the WordPress preset with a sitemap
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://www.example.com/wp-sitemap.xmlThese are the same rules WordPress serves by default. Theme, plugin and /wp-includes/ files stay crawlable because search engines need the CSS and JavaScript to render pages.
How Crawlers Read the Rules
- One group per crawler. A crawler uses the group whose
User-agentline matches its name most specifically, and ignores the rest. If you add aGooglebotgroup, Googlebot stops reading the*group, so repeat any rules from*that should still apply to it. - The most specific path wins. When an Allow and a Disallow both match a URL, the rule with the longer path applies. If they're the same length, Allow wins. The order of lines doesn't matter. For example, with
Disallow: /shop/andAllow: /shop/sale, the page/shop/sale/shoescan be crawled. - Rules match from the start of the path.
Disallow: /cartalso blocks/cartoons. To block only the page and its folder, use/cart$and/cart/, as the e-commerce preset does. - Sitemap lines apply to the whole file, not to a group, and you can list several.
- Crawl-delay asks a crawler to wait between requests. Bing supports it; Google ignores it and adjusts its crawl rate automatically.
- Host was a Yandex-only directive that Yandex itself has retired. Use redirects and canonical tags to choose a preferred domain.
Google's robots.txt specification describes how Googlebot applies these rules.
What robots.txt Can and Can't Do
It controls crawling
Use it to keep crawlers out of areas that waste their time or your server's: internal search results, cart and checkout pages, endless sort and filter combinations, or a staging site.
It doesn't remove pages from search
A blocked URL can still appear in Google, without a description, if other pages link to it. To keep a page out of results, let it be crawled and add a noindex robots meta tag or header. Google can't see a noindex on a page it isn't allowed to fetch.
It isn't security
Anyone can read your robots.txt, and crawlers that don't follow the rules simply ignore it. Protect private areas with a login, not a Disallow line, and remember that listing them in robots.txt advertises where they are.
Don't block CSS, JavaScript or images
Search engines render pages much like a browser. Blocking the files a page needs can stop them from seeing your content or layout correctly.
Blocking AI Crawlers
AI companies publish separate robots.txt names for different jobs, so you can opt out of one without the others:
- Training crawlers collect pages to train models:
GPTBot(OpenAI),ClaudeBot(Anthropic),CCBot(Common Crawl, whose data many AI projects use),meta-externalagent(Meta) andBytespider(ByteDance). The "Block AI Training" preset disallows these. - Control tokens aren't separate crawlers.
Google-Extendedcontrols whether content Google crawls can be used for its Gemini models, andApplebot-Extendeddoes the same for Apple's models. Disallowing them doesn't affect Google Search or Apple's search features. The preset includes both. - AI search and user-requested fetches, such as
OAI-SearchBot,ChatGPT-User,Claude-SearchBot,Claude-UserandPerplexityBot, decide whether your pages can appear or be cited in those assistants' answers. The preset leaves them alone; add groups for them if you want to block those too.
Crawler names change from time to time, so check each company's crawler documentation for the current list. Following robots.txt is voluntary: reputable crawlers do, others may not.
Frequently Asked Questions
Where does the file go?
At the top level of the host, so it opens at https://www.example.com/robots.txt. A file in a subfolder is ignored. Each subdomain, and each protocol or port, needs its own file. It must be plain UTF-8 text, and Google reads only the first 500 KiB.
What if my site has no robots.txt?
If the URL returns 404, crawlers assume everything may be crawled, which is fine for many sites. Server errors are different: if robots.txt returns a 5xx error, Google treats the whole site as blocked for a while, and if the errors continue it falls back to the last copy it fetched successfully.
How do I check what Google sees?
The robots.txt report in Search Console shows the file Google last fetched, when, and any lines it couldn't parse. The URL Inspection tool tells you whether a specific URL is blocked.
Should I block URLs with query parameters?
Only those that create endless near-duplicates, such as sort orders or filter combinations, and only when you don't need those pages in search. A blanket Disallow: /*? can block pages that matter, including paginated listings on some sites.
Is what I enter here stored?
Your rules are sent to Orbit2x's server to build the file and aren't saved. See the privacy policy.