// admin tools · 28
robots.txt & sitemap
Build a robots.txt with AI-crawler presets, test any URL against it with exact RFC 9309 matching, and generate a valid sitemap.xml — all in your browser.
// runs locally — nothing is uploaded
groups
evaluated
■ deciding rule ■ also matched ■ selected group
How it works
What it does
Three tools in one. --build writes a robots.txt from groups of user-agents and Allow/Disallow rules, with presets for allow-all, block-all, AI crawlers and WordPress. --test tells you whether a crawler may fetch a URL and highlights the exact rule that decided it. --sitemap turns a list of URLs into a valid sitemap.xml, splitting into several files plus a sitemap index when needed. Everything runs in your browser; nothing is fetched or uploaded.
How the tester decides
The matcher follows RFC 9309, the Robots Exclusion Protocol standard, which is also how Google documents its own behaviour:
- Pick the group. User-agent values are compared case-insensitively against the crawler's product token (paste a full UA string and the tokens inside it are tried). Every group naming that crawler is combined. Only if none does are the
User-agent: *groups used. - Normalise. Non-ASCII characters are percent-encoded as UTF-8, percent-encoded unreserved characters such as
%62are decoded, and other escapes keep their meaning — so%2Fis not the same as/. - Match. Rules match from the start of the path including the query string.
*matches any run of characters and a trailing$anchors the end. - Longest match wins. The matching rule with the most octets decides. If an Allow and a Disallow tie, Allow wins. No match means allowed, and
/robots.txtitself is always allowed.
Directives
- User-agent
- Starts a group. Consecutive lines share the rules that follow.
- Disallow / Allow
- A path pattern. An empty
Disallow:restricts nothing. - Sitemap
- An absolute URL, independent of any group.
- Crawl-delay
- Not part of RFC 9309. Google ignores it; Bing and Yandex honour it.
About the AI crawler preset
The preset lists training crawlers and control tokens such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider and meta-externalagent. Google-Extended and Applebot-Extended are tokens, not separate bots: blocking them opts out of AI training without affecting Search. A tick box also blocks search and user-triggered fetchers such as OAI-SearchBot and ChatGPT-User, which removes your pages from AI answers and citations.
FAQ
Does Disallow stop a page appearing in Google?
Not reliably. Disallow stops crawling, but a blocked URL can still be indexed from links pointing to it. To keep a page out of results, allow crawling and serve a noindex meta tag or X-Robots-Tag header.
Will blocking AI crawlers actually work?
robots.txt is voluntary. Major operators publish their tokens and say they honour them, but nothing technically enforces it. For hard blocking, use your server, CDN or firewall rules.
What are the sitemap limits?
Per the sitemaps.org protocol: up to 50,000 URLs and 50 MB uncompressed per file, all URLs absolute and on the same host. Larger sets are split, with a sitemap index pointing at each file. Upload every file to your site root and reference the index from robots.txt.
Do changefreq and priority matter?
Google says it ignores both and uses lastmod only when it is consistently accurate. They are optional here and omitted by default.