This guide is for bloggers, marketers and small business owners who run WordPress and want to see, change or check their robots.txt file without breaking how Google crawls the site. You do not need to write code. You do need to understand one thing first: robots.txt controls crawling, not indexing, and mixing the two up is the most common mistake.
Contents
What robots.txt does (and does not do)
According to Google Search Central, a robots.txt file tells search engine crawlers which URLs they can access on your site. It is used mainly to avoid overloading your site with requests. Google is explicit that it is not a mechanism for keeping a web page out of Google. To keep a page out of search results, use noindex or password-protect the page.
Google also lists some limits worth remembering:
- Not every crawler obeys robots.txt, and crawlers may interpret the syntax differently.
- A disallowed URL can still be indexed if other sites link to it, and its address and anchor text may still appear in results, usually without a description.
- It is not a security tool. Private content needs a password, not a robots.txt rule.
Virtual vs physical robots.txt in WordPress
WordPress does not need a robots.txt file sitting on your server. If none exists, WordPress generates one on the fly. The developer documentation for the do_robots() function describes it as displaying the default robots.txt content, sent as plain text. With typical paths, that default looks like this:
Enjoying this? Get our best tips by email
Practical SEO, link building and content strategies, straight to your inbox.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpDevelopers can change this output through the robots_txt filter, and the function fires a do_robotstxt action first. This is how SEO plugins add their own rules. The same documentation notes that since WordPress 5.3, the “Discourage search engines” setting no longer adds Disallow: / to this file; it is handled with a robots meta tag instead.
The practical difference:
| Type | Where it lives | How you change it |
|---|---|---|
| Virtual | Generated by WordPress when /robots.txt is requested | An SEO plugin editor or a code filter |
| Physical | A real text file in your site’s root folder | File upload or your host’s file manager |
Rank Math’s help pages say it creates a virtual file and that an actual robots.txt in the root folder stops you editing from the dashboard. Rank Math does not state outright which wins, but its fix is to delete the physical file. Treat a physical file as the thing to check first if your edits seem to do nothing.
Step 1: View your current robots.txt
Type your domain followed by /robots.txt into a browser, for example https://yourdomain.com/robots.txt. Google suggests using a private browsing window to confirm the file is publicly accessible. The file must be at the root of the host it governs; a copy in a subfolder does nothing, and each subdomain needs its own file.
Copy whatever you see into a safe place before you change anything. That gives you an easy way back.
Step 2: Edit it
Option A: Rank Math’s robots.txt editor
Per Rank Math’s knowledge base:
- Go to WordPress Dashboard → Rank Math SEO and make sure Advanced Mode is enabled (top right).
- Click General Settings, then Edit robots.txt.
- Edit the text in the box.
- Click Save Changes.
Rank Math’s default rules target all bots, disallow the WordPress admin directory, allow admin-ajax.php, and list your sitemap URL. In the free version a Robots.txt Tester link under the editor opens an external tool; the Pro version has a built-in Validate Code option.
If the editor will not save or the option is missing, Rank Math lists these causes: a physical file in the root folder, another plugin generating the file, server permission problems, a “robots.txt file is not writeable” message caused by file editing being disabled in wp-config.php, or a security plugin that disables built-in file editors. The fix depends on the cause, and the file-permission and wp-config.php ones are worth passing to your host or developer if you are unsure.
Option B: Upload a physical file
Google’s guidance for creating the file is simple. Use a plain text editor, not a word processor, because word processors can insert curly quotes or formatting that break rules. Save it as UTF-8 and name it exactly robots.txt. Then upload it to the site root through your host’s file manager or FTP. Google notes the exact upload method depends on your hosting, so check your host’s documentation.
A physical file means your SEO plugin’s editor may stop working for robots.txt, so choose one method and stick with it.
What the syntax looks like
Google’s documentation describes groups that begin with a User-agent line, followed by Disallow or Allow rules, one per line. Rules are case-sensitive, # starts a comment, and paths start with /. A Sitemap line is optional, can appear more than once, and must be a fully qualified URL. A simple WordPress example:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourdomain.com/sitemap_index.xmlReplace the sitemap address with your real one; the filename varies by plugin. For background on what a sitemap does, see our guide Understanding Sitemaps: A Complete Guide for SEO Success.
What to block, and what never to block
Reasonable things to block
- The admin area (already in the default), with the AJAX exception.
- Unimportant or duplicate URL patterns that waste crawl requests, such as internal search results, if your site generates many of them.
Google’s intro says robots.txt can keep crawlers away from unimportant or duplicate pages, but its main purpose is managing crawler traffic. For most small sites, the default is enough.
Things not to block
- CSS, JavaScript and images your pages need. Google says you may block resource files only if pages work fine without them. If losing them makes a page harder for Google to understand, do not block them. Blocking theme or plugin folders is a common way to cause this.
- Pages you want removed from search. This is the trap. If you disallow a page, Google may never see a
noindextag on it, and the URL can still show up if others link to it. Google also warns that combining crawling and indexing rules can make them counteract each other. To deindex a page, leave it crawlable and addnoindex(many SEO plugins offer a per-page robots setting; check yours), or put it behind a password. - The whole site, unless you truly mean it.
Disallow: /on a live site is a classic launch-day mistake, often copied over from a staging site.
If a page you blocked is already showing in results and you want it to display normally, Google says to remove the robots.txt entry. For the related indexing statuses you may see afterwards, read Crawled vs Discovered: Not Indexed in Search Console.
Step 3: Test with the Search Console robots.txt report
Google’s help page says the robots.txt report shows which robots.txt files Google found for the top 20 hosts in a Domain property, when each was last checked, and any errors or warnings. It is available for domain-level properties: a Domain property, or a URL-prefix property with no path.
- Open Settings for your property in Search Console and select the robots.txt report.
- Click your file to see its status. Fetched means the last attempt returned a file; any parsing problems appear under Issues. Errors stop a rule from being used, while warnings do not. Not Fetched – Not found (404) means there is no file, and Google can then crawl all URLs.
- Click Open live robots.txt to see the file in your browser.
- After fixing an error or unblocking important URLs, use the more settings icon next to the file and choose Request a recrawl. Google says this does not guarantee an immediate recrawl.
Google’s help page points to the URL Inspection tool for checking whether a specific URL is blocked, and to its open-source robots.txt library or third-party validators for local testing. Google crawlers find the file automatically, so you do not submit it.
Related reading: Google’s planned shared AI training control on robots.txt and AI crawlers.
Key takeaways
- WordPress serves a virtual robots.txt by default; a physical file in the root folder changes how you edit it.
- In Rank Math, go to Rank Math SEO → General Settings → Edit robots.txt (Advanced Mode on).
- Robots.txt manages crawling. Use
noindexor a password to keep pages out of search. - Do not block CSS, JavaScript or images that your pages need to render.
- Add a fully qualified Sitemap line, then check the file in the Search Console robots.txt report.
FAQ
Does my WordPress site need a robots.txt file?
No. If no file exists, WordPress generates a default one. Google also says that having no robots.txt means it can crawl all URLs, so the file is only needed if you want to change that.
Will Disallow remove a page from Google?
No. A blocked page can still be indexed if other sites link to it. Use noindex on a crawlable page or password protection.
Why can’t I edit robots.txt in Rank Math?
Rank Math’s help page lists a physical file in the root folder, another plugin generating the file, server permissions, file editing disabled in wp-config.php, or a security plugin disabling file editors.
How quickly does Google pick up changes?
Google says it recrawls robots.txt files often, and you can request a recrawl in the Search Console report, though it is not guaranteed to be immediate.
