Robots.txt basics for a small site, with a copy-ready example

A robots.txt file sits at your domain root and tells crawlers which paths they may fetch. On a small site you rarely need dozens of rules. You need a clear robots txt example: a User-agent block, sensible Disallow paths, optional Allow exceptions, and a Sitemap line that points at a real XML file. Wrong rules hide admin noise or, worse, block CSS and JavaScript so Google cannot render your pages. Staging leftovers that disallow everything keep a live site invisible. This guide walks through a copy-ready sample, what each line means, how to add a sitemap URL, mistakes to avoid, and when meta robots or noindex beat crawl rules. Draft fast with the robots txt generator, then verify https://yoursite.com/robots.txt in the browser.

What robots.txt does on a small site

Search crawlers request /robots.txt before they spend time on the rest of your URLs. The file is public text. Anyone can open it. Treat it as a polite map of fetch rules, not a password wall. If a path must stay secret, use authentication or keep it off the public host.

Small sites win with short files. A brochure site on Webflow, a ten-page WordPress install, or a Next.js marketing site usually needs a few Disallow lines for admin, search, or cart URLs, plus a Sitemap pointer. You do not need a crawl-budget treatise written for million-page stores. Start open, hide the obvious junk, and leave CSS, JS, and image folders crawlable.

robots.txt limits crawling. It does not guarantee a URL stays out of Google’s index. Other sites can still link to a disallowed URL, and Google may show a URL-only result. When a page must not appear in search at all, use a noindex tag or header. Pair that with the meta tag generator when you need a ready <meta name="robots" content="noindex"> block.

Hosts and CMS platforms sometimes serve a virtual robots.txt. WordPress, Wix, and Shopify may override a file you uploaded by FTP. After you publish, open the live URL and read what visitors actually get. If the live file differs from your draft, edit in the SEO plugin or host panel instead of fighting the disk file.

A robots txt example you can adapt

Paste this starter for a typical small marketing site. Replace the sitemap host with your domain. Add or remove Disallow lines to match your CMS.

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /cart
Disallow: /checkout
Disallow: /search
Disallow: /thank-you/

Sitemap: https://www.example.com/sitemap.xml

Read the block top to bottom. User-agent: * applies the following rules to every crawler that respects the standard. Allow: / keeps the site open by default. Each Disallow hides a path prefix. The Sitemap line is a hint, not a crawl rule; it helps crawlers find your URL list.

Skip rules you do not have. A static site without WordPress should drop /wp-admin/. A brochure site without commerce should drop /cart and /checkout. Extra Disallow lines for missing folders do little harm, but they clutter reviews when someone audits the file later.

Comments start with #. Use them for your future self: # staging only; remove before launch. Keep production comments short so the file stays easy to scan.

If you want CMS-shaped defaults instead of hand-editing, open the robots txt generator, pick WordPress, Shopify, Webflow, Next.js, or Staging, then copy the output and change the Sitemap URL.

Allow, Disallow, and User-agent lines

Syntax is line-based and picky about spaces. Most small-site files use one group for all bots. You only split User-agent blocks when one crawler needs different rules.

User-agent groups

User-agent: * is the default group. Put shared Disallow and Allow lines under it. Start a new group only when you must name a specific bot, for example a staging block that applies to everyone the same way still uses *.

Blank lines separate groups. Rules under one User-agent do not carry into the next group. If you copy samples from several blogs into one file, check that you did not leave orphan Disallow lines without a User-agent above them.

Crawler names are case-sensitive in practice for many bots. Prefer documented names from the vendor. For a small public site, one * group is enough until you have a concrete reason to special-case a bot.

Disallow path prefixes

Disallow: /admin/ blocks any URL that starts with /admin/. Trailing slashes matter for how you think about folders, but crawlers match prefixes. Disallow: /search also catches /search?q=shoes on many parsers. Test the paths you care about in Google Search Console’s robots testing tools when traffic justifies the setup.

An empty Disallow (Disallow:) means allow everything for that group. Disallow: / under User-agent: * blocks the whole site. That pattern belongs on staging, not on a live storefront you want indexed.

Do not Disallow your CSS, JS, or font folders. Rendering needs those assets. Blocking /wp-includes/ on WordPress is a common template line for code paths; blocking /static/ or /_next/static/ on a modern front end can break how Google sees layout. Prefer hiding admin and duplicate search URLs over hiding assets.

Allow as an exception

Allow narrows a broader Disallow. WordPress samples often disallow /wp-admin/ then allow /wp-admin/admin-ajax.php so plugins keep working for crawlers that need that endpoint. On a tiny custom site you may never need Allow lines if you never Disallow a parent path.

Order and longest-match rules vary by crawler. Keep exceptions obvious: Disallow the folder, Allow the one file. If the logic feels hard to read, simplify the Disallow list instead of stacking clever Allow lines.

Add a Sitemap line without guessing the URL

Sitemap: takes an absolute HTTPS URL. Relative paths are a frequent mistake. Prefer Sitemap: https://www.example.com/sitemap.xml over /sitemap.xml.

Find the real file first. Open /sitemap.xml in a browser, or check your CMS SEO settings for the sitemap index URL. WordPress with Yoast or Rank Math often uses /sitemap_index.xml. Shopify usually exposes /sitemap.xml on the storefront domain. If the URL 404s, fix the sitemap generator before you advertise it in robots.txt.

You may list more than one Sitemap line. A small site usually needs one. Large sites split by post type; you can ignore that until you outgrow a single index file.

Submitting the same sitemap in Google Search Console still helps. The robots.txt line is a discovery hint for any crawler that reads the file. It does not replace Search Console verification, and it does not force a crawl.

Common mistakes on small sites

Most damage comes from copy-paste presets that never get reviewed after launch. Two failure modes show up again and again: blocked assets and staging rules left on production.

Blocking CSS and JavaScript

Disallowing /static/, /assets/, /_next/, or broad /*?* rules can stop crawlers from loading stylesheets and scripts. Google may still index HTML, but rich results and mobile usability suffer when the renderer cannot see the page you ship to users.

Before you publish a strict preset, open the draft and ask whether each Disallow hides a content URL or a build asset. Content URLs (admin, cart, internal search) are fair game. Asset URLs are not. If a CMS preset blocks a folder your theme needs, delete that line.

After deploy, use URL Inspection or a fetch-as-Google style test when you have Search Console access. If the rendered screenshot looks unstyled, inspect robots.txt for asset blocks first.

Staging Disallow-all left on production

A staging robots.txt often looks like this:

User-agent: *
Disallow: /

That file is correct on staging.example.com. On www.example.com it tells polite crawlers to fetch nothing. Teams clone the host, promote the database, and forget the file. Rankings stall for weeks while everyone debugs meta tags.

Make staging removal a launch checklist item. Switch the robots txt generator from the Staging preset to your CMS preset, paste the new file, and confirm the live URL no longer contains Disallow: / alone. Add password protection or IP allowlists on staging so you are not relying on robots.txt as the only shield.

Crawl rules vs noindex and meta robots

Use robots.txt when you want to reduce wasted crawls on login, cart, faceted search, or thank-you URLs. Use noindex when the HTML may be fetched but must not rank. Thank-you pages, thin tag archives, and private docs that still need a shareable link often need noindex even if they are allowed in robots.txt.

If you Disallow a URL, crawlers that obey the file may never fetch it, so they may never see a noindex tag on that page. For “do not show in search,” prefer allowing the crawl and setting noindex, or use both carefully with a plan. Soft-private pages that should stay out of the index belong in the meta tag generator workflow: robots meta plus a canonical when a preferred URL exists.

Canonical tags and robots meta live in the HTML head. robots.txt lives at the site root. They solve different jobs. Fixing only one when the problem is the other wastes a week of guessing.

Draft the file, publish it, then check the live URL

Build the text in a generator or editor. Save as robots.txt with plain UTF-8 text and a trailing newline. Upload to the site root so https://yoursite.com/robots.txt returns 200 and text/plain (or an equivalent the host serves for that path).

On WordPress, paste through Yoast, Rank Math, or the theme’s file manager when the host blocks root uploads. On Webflow, Squarespace, and Wix, find the SEO or robots controls in the dashboard. Headless Next.js sites often generate the file in app/robots.ts or public/robots.txt; deploy that path with the rest of the app.

Verification is three clicks. Open the live URL in a private window. Confirm your Sitemap line and Disallow list match the draft. Search for Disallow: / as a lone production rule. Then run a robots.txt checker or Search Console tester against a sample URL you care about.

If the file “does nothing,” check caching, CDN rules, and whether the CMS overwrote your upload. robots.txt not working reports often turn out to be a different file served at the edge, or a staging host tested by mistake.

Related on-page checks after robots.txt

Crawl access is only one slice of how a page appears in search. A clean robots.txt gets bots to the right URLs. Titles, descriptions, and structured data still decide how those URLs look in results and whether rich features fire.

Once the file is clean, tighten titles and descriptions so snippets read well. Wave peers cover meta title length for Google and meta description truncation. Generate head tags with the meta tag generator when you need a paste-ready block.

For FAQ rich results without a WordPress plugin, see FAQ schema JSON-LD. Social previews need separate Open Graph work, including OG image size. Do that after crawl rules stop hiding the pages you want shared.

Frequently asked questions

What is a good robots txt example for a small website?

Start with User-agent: *, keep the site allowed, Disallow admin, cart, checkout, and internal search paths you actually use, then add one absolute Sitemap: URL. Delete lines for folders your stack does not have. A ten-line file that matches your CMS beats a hundred-line paste from an enterprise blog.

Where should I put robots.txt on my site?

Place it at the domain root so crawlers fetch https://yourdomain.com/robots.txt without a subdirectory. Subfolder installs still need the file on the host that answers for the public hostname. If a CDN or reverse proxy sits in front, confirm the edge serves your file and does not cache an old staging copy.

Does Disallow stop Google from indexing a URL?

No. Disallow asks crawlers not to fetch the URL. Google may still index a URL it discovered through links, sometimes without a snippet. For true exclusion from results, use noindex on a crawlable page or keep the content behind auth. Use robots.txt to save crawl time on junk paths, not as your only privacy control.

Should I block CSS and JavaScript in robots.txt?

No for normal small sites. Crawlers need those files to render pages the way users see them. Blocking asset folders creates false “mobile usability” and content mismatches. Disallow login, admin, and duplicate parameter URLs instead. Review any preset that blankets /static/ or query strings before you ship it.

How do I write a robots.txt disallow all rule for staging?

A disallow-all file uses one User-agent group and a single path rule that blocks the whole site. Use this pattern:

User-agent: *
Disallow: /

Host that file only on staging or pre-launch domains. On production, replace it with an open or CMS-specific file the same day you point DNS at the live site. Add auth on staging so polite robots.txt is not your only barrier.

How do I add a sitemap to robots.txt?

Add a line Sitemap: https://www.example.com/sitemap.xml with your real absolute URL. Confirm the sitemap loads in a browser and returns XML. WordPress SEO plugins often use /sitemap_index.xml; copy that exact path. You can list multiple Sitemap lines if you maintain separate indexes.

Do I need a robots.txt generator or can I write the file by hand?

Either works. Hand-writing is fine when you understand Allow and Disallow. A robots txt generator speeds CMS presets and reduces typos in User-agent blocks. Generate, read every line, change the Sitemap host, then paste into your host or SEO plugin. Never publish a preset you have not skimmed.

What is the difference between robots.txt and meta robots?

robots.txt is a site-root fetch policy. Meta robots (and X-Robots-Tag headers) are per-page indexing instructions inside the HTTP response. Disallow reduces crawling; noindex reduces indexing after a fetch. Small sites often need both: a short robots.txt plus noindex on thank-you or private HTML pages built with a meta tag generator.

Why does my WordPress robots.txt look different from the file I uploaded?

WordPress may serve a virtual file, and SEO plugins can override root uploads. Edit rules inside Yoast, Rank Math, or your host’s recommended path, then re-open the live URL. If a plugin and a physical file fight, the plugin usually wins. Treat the browser view of /robots.txt as source of truth.

How do I check that robots.txt is working?

Open https://yoursite.com/robots.txt and read it. Use Google Search Console’s robots testing tools when the property is verified. Compare a sample Disallow path against a URL you expect to block. If results still show staging content, search the live file for Disallow: / and confirm you are not testing the wrong hostname.

Clean crawl rules make room for clearer titles, descriptions, and schema on the pages you do want crawled. Keep the robots txt generator for the next CMS preset change, and pair indexing tags with the meta tag generator when a page must stay out of results.