admin@likeapro.dev: ~/robots — zsh

// admin tools · 28

robots.txt & sitemap

Build a robots.txt with AI-crawler presets, test any URL against it with exact RFC 9309 matching, and generate a valid sitemap.xml — all in your browser.

// runs locally — nothing is uploaded

presets

groups

NORMAL local only · 0 bytes sent --:--:--

How it works

What it does

Three tools in one. --build writes a robots.txt from groups of user-agents and Allow/Disallow rules, with presets for allow-all, block-all, AI crawlers and WordPress. --test tells you whether a crawler may fetch a URL and highlights the exact rule that decided it. --sitemap turns a list of URLs into a valid sitemap.xml, splitting into several files plus a sitemap index when needed. Everything runs in your browser; nothing is fetched or uploaded.

How the tester decides

The matcher follows RFC 9309, the Robots Exclusion Protocol standard, which is also how Google documents its own behaviour:

  1. Pick the group. User-agent values are compared case-insensitively against the crawler's product token (paste a full UA string and the tokens inside it are tried). Every group naming that crawler is combined. Only if none does are the User-agent: * groups used.
  2. Normalise. Non-ASCII characters are percent-encoded as UTF-8, percent-encoded unreserved characters such as %62 are decoded, and other escapes keep their meaning — so %2F is not the same as /.
  3. Match. Rules match from the start of the path including the query string. * matches any run of characters and a trailing $ anchors the end.
  4. Longest match wins. The matching rule with the most octets decides. If an Allow and a Disallow tie, Allow wins. No match means allowed, and /robots.txt itself is always allowed.

Directives

User-agent
Starts a group. Consecutive lines share the rules that follow.
Disallow / Allow
A path pattern. An empty Disallow: restricts nothing.
Sitemap
An absolute URL, independent of any group.
Crawl-delay
Not part of RFC 9309. Google ignores it; Bing and Yandex honour it.

About the AI crawler preset

The preset lists training crawlers and control tokens such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider and meta-externalagent. Google-Extended and Applebot-Extended are tokens, not separate bots: blocking them opts out of AI training without affecting Search. A tick box also blocks search and user-triggered fetchers such as OAI-SearchBot and ChatGPT-User, which removes your pages from AI answers and citations.

FAQ

Does Disallow stop a page appearing in Google?

Not reliably. Disallow stops crawling, but a blocked URL can still be indexed from links pointing to it. To keep a page out of results, allow crawling and serve a noindex meta tag or X-Robots-Tag header.

Will blocking AI crawlers actually work?

robots.txt is voluntary. Major operators publish their tokens and say they honour them, but nothing technically enforces it. For hard blocking, use your server, CDN or firewall rules.

What are the sitemap limits?

Per the sitemaps.org protocol: up to 50,000 URLs and 50 MB uncompressed per file, all URLs absolute and on the same host. Larger sets are split, with a sitemap index pointing at each file. Upload every file to your site root and reference the index from robots.txt.

Do changefreq and priority matter?

Google says it ignores both and uses lastmod only when it is consistently accurate. They are optional here and omitted by default.