A robots.txt file sits at the root of your website and tells search engine crawlers which parts of the site they may visit. For a small site, you do not need a long or clever file. You need a short, tested one — and this guide builds exactly that, using the Robots.txt and Sitemap Toolkit.

What robots.txt actually does

When a crawler arrives at your site, it usually checks yourdomain.com/robots.txt first. The file contains groups of rules. Each group starts with a User-agent line naming the crawler it applies to — * means all crawlers — followed by Allow and Disallow lines naming paths.

A Disallow line does not delete a page, hide it from people, or remove it from an index that already lists it. It asks well-behaved crawlers not to visit that path. A disallowed page can still be linked to from elsewhere, and a search engine may still show its bare URL in results if it learns about it another way. That distinction shapes every decision below: use robots.txt to guide crawling, and use passwords, login controls, and noindex tags when you actually need to keep something private or out of an index.

Start by listing what matters

Before you write a rule, make two short lists. The first is the pages you want search engines to find: your homepage, main service or product pages, key articles, and contact page. The second is the paths crawlers do not need: the admin area, login pages, internal search results, shopping cart and checkout pages, and account pages.

Most small sites share the same second list. Admin paths such as /wp-admin/ and /admin/ exist for the site owner, not for visitors. Internal search pages such as /search/ generate a new URL for every query typed into the search box, so letting crawlers follow them can create hundreds of thin, near-duplicate pages. Cart, checkout, and account pages are personal to each visitor and contain nothing a search engine should index. Blocking those paths is not about secrecy; it is about keeping the crawler's attention on content worth showing to the public.

There is one path you should never block casually: your site's root, /. A rule that reads Disallow: / blocks everything. It is sometimes used deliberately on a staging site that must not be found, but on a live site it is almost always a mistake, often made by copying a staging file to production without checking it. Read the finished file line by line before publishing, and check that it does not contain Disallow: / unless you truly mean it.

Build the file in the toolkit

Open the Robots.txt and Sitemap Toolkit and start with a plain template rather than typing the whole file from memory. The toolkit explains each rule in plain language as you add it, which helps you catch a rule that sounds right but does something different from what you intended. A wildcard such as *, for example, matches more than many beginners expect, and a missing trailing slash changes which paths a rule covers.

A safe first file for a typical small site might look like this:

User-agent: *
Disallow: /wp-admin/
Disallow: /admin/
Disallow: /search/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /login/
Allow: /

Sitemap: https://example.com/sitemap.xml

Replace example.com with your own domain and adjust the blocked paths to match your platform. A site without WordPress will not have /wp-admin/, and a site without a shop will not need /cart/ or /checkout/. If your platform uses a different admin path, add that one instead. The file should describe your site, not someone else's template.

Paths in robots.txt are case-sensitive, so /Admin/ and /admin/ are different paths, and the most specific rule wins when rules overlap. The Sitemap: line points crawlers to your sitemap, but it does not replace submitting it in a search console.

Test every URL before you publish

This is the step most people skip, and it is the step that prevents the embarrassing mistakes. The toolkit lets you test URLs against the rules you have just built, so you can see whether each important address is allowed or blocked before the file ever reaches your live site.

Test at least these, using your own real URLs:

  • Your homepage, for example https://example.com/ — this must be allowed.
  • A main content page, for example https://example.com/services/ — this must be allowed.
  • A blog post or product page — this must be allowed.
  • Your admin path, for example https://example.com/wp-admin/ — this must be blocked.
  • An internal search result, for example https://example.com/search/?q=shoes — this must be blocked.
  • A cart or checkout page — this must be blocked.

If any result surprises you, do not publish yet. Edit the rule, re-test, and only publish when every test matches your two lists. Testing takes a few minutes. Recovering from a live file that blocks your homepage can take much longer, because crawlers may not revisit immediately.

Once the tests pass, upload the file to the root of your site so it is reachable at yourdomain.com/robots.txt, then visit that address in a browser and read it once more. Confirm the file you see is the file you tested, not an older version served from cache. If you use a search console, submit the file for its own robots.txt test as a second opinion.

Common first-file mistakes

The most common mistake is blocking the whole site with Disallow: /, usually copied from a staging environment. The second is blocking assets that pages need to render, such as a CSS or JavaScript folder, which can make pages look broken to a crawler even when the pages themselves are allowed. Do not block theme or asset folders unless you have a specific reason and have checked the effect.

A third mistake is treating robots.txt as a privacy tool. Never list a genuinely sensitive path in robots.txt as your only protection, and never assume a disallowed path is hidden. The file itself is public — anyone can read yourdomain.com/robots.txt — so it can actually draw attention to the paths it names. Sensitive areas need a login or password, not a disallow line. The fourth mistake is the opposite: leaving a staging copy of the site crawlable, so search engines index half-finished pages alongside the real ones. A staging site should be password-protected, with a robots.txt that blocks everything.

Honest limitations

Robots.txt is a request, not a security barrier. Well-behaved search engine crawlers respect it, but malicious bots, scrapers, and anyone visiting your site directly will ignore it completely. It cannot keep a page private, protect a password, or remove a page from search results on its own. A sitemap does not guarantee indexing either, and a page that is allowed in robots.txt may still not appear in results if it is thin, duplicated, or never linked to. Treat robots.txt as one signal in a larger setup: good content, clear internal links, and correct indexing controls do the rest. To remove a page from an index, use a noindex tag or remove the page, rather than relying on a disallow line alone.

Revisit the file whenever you add a shop, a membership area, or a new section, and re-test after every change.

For a deeper explanation of syntax, user-agent groups, and matching rules, read How to Configure robots.txt Correctly. To check the rest of the page — titles, headings, canonical tags, and other on-page signals — run the site through the SEO Checker, and if you add structured data to key pages, build it with the Schema Generator.

Frequently asked questions

Do I need a robots.txt file at all?

No, a site can work without one, and crawlers will then assume they may visit everything. A file is still worth having because it steers crawlers away from admin, search, and cart paths and points them to your sitemap in one place.

Will blocking a page in robots.txt remove it from search results?

Not by itself. A disallowed page may still appear as a bare URL if other pages link to it. To keep a page out of results, use a noindex tag or remove the page, and use robots.txt only to control crawling.

Can I block just one crawler and allow the rest?

Yes. Add a separate group starting with that crawler's user-agent name, with its own rules, alongside the general * group. Test carefully, because a group for a named crawler overrides the general group for that crawler.

What should I do if I accidentally blocked my whole site?

Fix the file immediately, upload the corrected version to the root, confirm it at yourdomain.com/robots.txt, and re-test your homepage and key pages. Then request re-crawling in your search console if that option is available, and allow time for crawlers to revisit.

human review pending client designation