Build a clean robots.txt for WordPress or any website. Choose which AI and SEO crawlers to block, add your sitemap and copy or download the file.
What robots.txt does (and what it doesn’t)
robots.txt is a plain text file at the root of your site, for example example.com/robots.txt. Well-behaved crawlers read it before they visit and skip the folders you list. It is a request, not a lock: polite bots like Googlebot follow it, scrapers ignore it.
The most common mistake is using it to hide pages from Google. Blocking a URL stops Google reading the page, but the URL can still appear in results if other sites link to it. To keep a page out of search, let Google crawl it and add a noindex tag instead.
A safe starting point for WordPress
For most WordPress sites, the best robots.txt is short:
- Block
/wp-admin/, but allow/wp-admin/admin-ajax.php, which some themes and plugins need on the front end. - Leave
/wp-content/and/wp-includes/open. Google needs your CSS, JavaScript and images to see the page properly. - Add a
Sitemap:line pointing to your XML sitemap.
WooCommerce stores can also block cart, checkout and account pages, which have no value in search and waste crawl time. The generator adds those when you pick “WordPress + WooCommerce”.
Should you block AI crawlers?
That depends on what you want from your content. Blocking GPTBot, ClaudeBot, CCBot and Google-Extended asks those companies not to use your pages to train their models. It does not affect your normal Google rankings. Blocking a search-style bot such as PerplexityBot is different: it can stop your pages being cited, and linked, in that tool’s answers.
A middle path many publishers take is to block training crawlers and allow search ones. Either way, it is your call, and you can change it at any time.
Blocking SEO tool bots to save server load
AhrefsBot, SemrushBot, MJ12bot and DotBot crawl millions of sites to build backlink databases. On small shared hosting plans they can eat a real share of your CPU. Blocking them costs you nothing in Google, though it means those tools will show less data about your own site.
Uploading and testing the file
On WordPress, Rank Math and Yoast both have a robots.txt editor in their settings, so you don’t need FTP. After saving, open yoursite.com/robots.txt in a browser to check it, then use the robots.txt report in Google Search Console. Our guide on viewing, editing and testing robots.txt on WordPress walks through each step.
Common questions
Where do I put robots.txt?
At the root of your domain, so it opens at yoursite.com/robots.txt. A file in a subfolder is ignored. Subdomains such as shop.yoursite.com need their own file.
Does Google follow Crawl-delay?
No. Googlebot ignores Crawl-delay, which is why this generator leaves it out. Bing and some other crawlers respect it, but blocking heavy bots is usually more effective.
Will blocking Google-Extended hurt my rankings?
No. Google says Google-Extended only controls whether your content is used for Gemini and related AI features. Normal crawling and ranking by Googlebot are unaffected.
Can a wrong robots.txt remove my site from Google?
Yes. A single “Disallow: /” under “User-agent: *” blocks the whole site. It happens most often when a staging site’s file is copied to the live site, so always check the file after launching.