Robots.txt and Sitemap Toolkit

How to use this toolkit

1. Start the robots file from a preset, or from one open group. Load the preset closest to your site — WordPress, a shop, or one of the AI-crawler presets — knowing it replaces the builder's current contents, then edit the rules it loaded. If no preset fits, keep the single * group and add only the paths you genuinely need to close.
2. Shape each group. Set the crawler token, add Disallow lines for the paths that crawler should not fetch, and add an Allow line whenever one page inside a blocked area must stay open. Read the plain-language line under the group to confirm it says what you intended.
3. Add your sitemap addresses and check the validation list. The preview updates as you type. Work through every error and warning: each message names its line and explains the fix, from a rule placed before any User-agent line to a group that names a crawler but blocks nothing.
4. Test the URLs that matter before you download. Run your checkout, account, help and best product pages through the simulator for * and for any crawler you gave its own group. The verdict names the deciding rule, so a surprise is a finding, not a mystery.
5. Generate the sitemap in the second tab. Paste your URL list (or an old sitemap), add honest last-modified dates where you know them, and generate. Fix every invalid line the report names, note the duplicates it removed, and download the XML.
6. Publish both files at your site root. The robots file belongs at https://yoursite.example/robots.txt and the sitemap at the address declared in its Sitemap line. After publishing, fetch both addresses yourself to confirm what the world actually receives, then test a few URLs once more against the live file.

Worked example: a small shop's rules and sitemap

The sample loads a compact rule set for a small shop and the URL list that goes with it. The robots side has a * group that blocks /checkout/ and /account/ but re-opens /account/help, a separate Googlebot group that blocks /tmp/, and one Sitemap line. Load it, then run these five tests in the simulator — every verdict below is the verdict the tool returns:

Load sample

• https://example.com/checkout/ for AnyBot — Blocked by Disallow: /checkout/. The crawler name "AnyBot" has no group of its own, so the * group decides.
• https://example.com/account/help for AnyBot — Allowed by Allow: /account/help. Both account rules match, but the Allow pattern is 13 characters against the Disallow's 9, and the longest match wins.
• https://example.com/account/settings for AnyBot — Blocked by Disallow: /account/. The Allow covers only the help page, not the whole account area.
• https://example.com/tmp/file.html for Googlebot — Blocked by the Googlebot group's Disallow: /tmp/. The named group beats the * group, and only its rules are consulted.
• https://example.com/tmp/file.html for AnyBot — Allowed. AnyBot is judged by the * group, which contains no /tmp/ rule — proof that a block written for one crawler does not leak onto the others.

The sample URL list in the Sitemap tab has seven lines: five become valid, unique sitemap entries; one line (not-a-url, line 5) is reported invalid because it is not an absolute address; and one line repeats the homepage, so it is counted as a duplicate and listed once. The widget product carries its own last-modified date of 2026-10-01 from the URL | date format, and the search address — https://example.com/search?q=shoes&sort=price — appears in the XML with its ampersand correctly escaped as &, because a raw & would make the file invalid XML.

What this toolkit does, and why it matters

Two small text files quietly decide how search engines and, increasingly, AI systems experience your site. Robots.txt is read before crawling begins, and it answers one question per crawler: which paths may I fetch? A sitemap answers the opposite question: which pages exist, and when did each last change? Sites usually produce both files once — often from a plugin default or a copied template — and never look at them again, while the site underneath changes for years. The toolkit exists to make both files deliberate: built rule by rule with each rule explained, checked for the syntax mistakes that silently change their meaning, and tested against real URLs before anything is published.

The robots side is a builder rather than a text box because the file's failures are structural, not typographical. Rules belong to the group named above them, the most specific group wins outright, and among matching rules the longest pattern decides — so a file can be perfectly spelled and still do the opposite of what its author meant: an Allow placed in the wrong group protects nothing, and a block written for one crawler says nothing to any other. Building group by group, with a plain-language restatement under each group and a simulator that names the deciding rule for any URL, turns those invisible mechanics into something you can see and correct. The preset library encodes the same care: the AI-crawler presets name GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent and Bytespider individually, because in robots.txt there is no such thing as blocking "AI" in general — there are only product tokens, one crawler at a time.

The sitemap side is deliberately unglamorous: validation and honest dates. A pasted list is checked line by line, invalid lines are named with their reasons instead of being silently dropped, duplicates are counted rather than quietly multiplied, and the output is well-formed XML with correctly escaped addresses. The 50,000-URL protocol limit is handled the way the protocol intends, with an index and child files, rather than by truncating your list at an invisible boundary. None of this guarantees indexing — a sitemap is a hint, not a command — but a file that is valid, current and truthful is the version of the hint crawlers are built to trust.

Common errors and limitations

• The staging block that reached production. Disallow: / is correct on a development copy and removes the whole live site from crawling when copied across. It is the first line to check whenever crawling collapses after a launch.
• Expecting robots.txt to hide or protect content. It cannot. Blocked paths remain fetchable by anything that ignores the file, blocked URLs can still appear in search results as bare links, and the file itself is public. Use authentication for anything private.
• Assuming Crawl-delay slows Google. Googlebot ignores the Crawl-delay line entirely; use Search Console's crawl-rate settings for Google. Bing and Yandex do honour the line, which is the only reason to include it.
• Expecting a Google-Extended block to change Google Search. Google-Extended governs AI training and grounding use of crawled content. Search crawling is governed by the Googlebot token. Blocking one never does the other's job.
• Case and anchors. Paths are case-sensitive, so /Private/ does not block /private/; and a trailing $ anchors a pattern so strictly that /page.php$ stops matching the moment a query string is added. The simulator demonstrates both behaviours.
• Simulator scope. The tester implements the RFC 9309 core — group selection, longest match, wildcards and anchors — against one file at a time. Individual crawlers add their own extensions and size limits on top, and a crawler that fetches a stale cached copy of your file will not see today's edit. Treat a verdict as the standard's answer, then confirm critical paths in each engine's own testing tool.
• Sitemap honesty. The generator validates form, not truth: it cannot know whether a listed page exists, whether a date is accurate, or whether a page deserves indexing. Listing error pages, redirects or near-duplicate addresses wastes the trust the file is meant to build.

Frequently asked questions