Robots txt generator

Favorite

Robots txt generator builds a crawl-ready robots.txt file with CMS presets. Pick your platform, add rules, and copy the file to your site root. Free.

Site originCMS / platformBlock admin, login, and internal search — keep AJAX for plugins.

Matches common WordPress + Yoast-style rules. Upload as /robots.txt or use an SEO plugin if your host blocks the file.

Include Sitemap lineBlock internal search URLsDisallow common AI training crawlers (GPTBot, CCBot, …)

Extra Disallow paths (one per line)

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-includes/
Disallow: /wp-login.php
Disallow: /?s=
Disallow: /search/
Disallow: /author/
Disallow: /cgi-bin/
Disallow: /search
Disallow: /*?s=

Sitemap: https://example.com/sitemap.xml

Save as robots.txt at your site root (e.g. example.com/robots.txt). Some CMS platforms override this file in the dashboard — publish there if uploads are blocked. robots.txt guides crawlers; it does not guarantee indexing. Use noindex on staging pages too.

How it works

1
Enter your data
Text, a link, or settings — right on this page.
2
Instant result
Everything runs on your device. We do not get a copy.
3
Copy or save
Use it right away.

Need more room?

Pro lifts daily file caps. Plus adds keyword research when you outgrow free checks.

See plans — from $9/mo

Why people use Toolsy

Fast

You get the result in a second. No waiting in a line.

Private

Browser tools stay on your device. Uploaded files are processed and discarded with the response.

No sign-up

Open the page and start. An account is only for paid plans.

Other tools

FAQ schema generator Title & meta length Keyword density Keyword desk

Generate robots.txt with the robots txt generator

A robots.txt file sits at your domain root and tells search engines which URLs they may crawl. Wrong rules waste crawl budget or expose staging paths.

Pick a CMS preset, set your site origin, and copy a starter file. Add custom Disallow lines, a Sitemap URL, and optional AI crawler blocks.

Use the robots txt generator as a practical template. Pair it with noindex on pages that should never rank, then test in Google Search Console.

Presets include

WordPress
Shopify
Webflow & Squarespace
Next.js / headless
Drupal & Joomla
Magento & Ghost
HubSpot & Wix
Staging (block all)

Options

Sitemap line
Internal search blocks
AI crawler blocks
Custom Disallow paths
One-click copy
Private — no upload

Common questions

What does a robots txt generator do?

It writes a robots.txt file that tells crawlers which paths they may fetch. You pick presets, edit rules, and copy the file.

Where does a robots txt generator file go?

At your site root, for example https://yoursite.com/robots.txt. Upload via FTP, host panel, or CMS SEO settings.

Does a robots txt generator block indexing?

It limits crawling, not indexing. Use meta robots or noindex when a page must stay out of search results.

Which CMS preset should I pick in the robots txt generator?

Choose the platform that hosts your site. Presets add common Disallow paths for admin, cart, and search URLs.

Does the robots txt generator upload my site URL?

No. The file is built only in your browser.

WordPress already has robots.txt — do I need this?

WordPress may serve a virtual file or let an SEO plugin override it. Use this generator to draft rules, then paste into Yoast, Rank Math, or a physical robots.txt if your host allows it.