Putting a site behind a content delivery network (CDN) is one of the most common speed upgrades for WordPress sites and small business sites. It can also go wrong for search: a bot challenge that shuts out Googlebot, or a cold cache that sends a burst of crawl requests to your hosting account. This checklist is for site owners, bloggers and small agencies who are about to make that move, or who made it recently and want to check nothing is broken. It is based on a Google Search Central blog post from November 2026 and on Google’s own documentation, listed under Sources.
Why a CDN move can change crawling
Google’s post explains that its crawlers estimate how much a site can handle (a crawl capacity limit, also called hostload) from server health and response times. Slow responses make crawlers back off. Fast, stable responses let the crawl rate rise, as long as there is crawl demand.
When your DNS points at a fast CDN edge, response times usually drop. According to the post, crawlers can read that as a hardware upgrade and request more. Two things make this awkward at cutover:
- Cold cache. The first request for each URL at an edge location is a cache miss and goes back to your origin server. Only later requests are served from cache.
- Crawlers and visitors behave differently. Visitors cluster on a few popular pages, which warm up fast. Crawlers try to cover your whole URL inventory, so many more URLs miss the cache at the same time.
The same post notes that crawlers usually ramp up gradually and that many CDNs pace connections to smooth spikes, so a problem is not guaranteed. The point of the checklist is to avoid the two failure modes: blocking crawlers by accident, and overloading the origin while the cache fills.
The checklist
1. Before cutover: decide what bot protection you actually need
Google’s post warns that turning on broad bot blocking or CAPTCHA-style challenges also blocks verified crawlers such as Googlebot. Crawlers then land on error or challenge pages, the site looks restricted or broken, and if that continues they back off and pages may drop out of Search.
- Find out whether your CDN or firewall has bot protection switched on by default. Check the provider’s own documentation for where that setting lives, because names and locations differ between vendors and change over time.
- If you block bots, use the verified-bot allowance your CDN or web application firewall (WAF) offers, rather than allowing anything that merely claims to be Googlebot.
- Prefer targeted rules (for example against a specific unwanted bot) or robots.txt over a blanket challenge on all automated traffic. The post lists both as alternatives.
If you need a refresher on editing robots.txt safely, see our guide on how to view, edit and test robots.txt on WordPress.
2. Verify Googlebot instead of trusting the user agent
Google’s documentation on verifying Googlebot says spammers and other bad actors may send requests while claiming to be from Google, so a claimed identity is not proof. It describes two methods.
- Manual check: take the IP address from your server logs, run a reverse DNS lookup, confirm the hostname ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that hostname and confirm it returns the original IP. For common crawlers, expect hostnames of the form crawl-…googlebot.com or geo-crawl-…geo.googlebot.com.
- Automatic check: compare the requesting IP with the CIDR ranges Google publishes as JSON files. There are separate files for common crawlers, special-case crawlers and user-triggered fetchers, so use the one that matches the traffic you are allowing.
The manual method suits a one-off check on a handful of log lines. If your CDN offers a verified-bot allowance, it is usually the easier route for ongoing protection, and Google’s post recommends using it alongside Google’s published IP ranges.
3. Pre-warm the cache before routing all traffic
Google’s post recommends requesting important URLs through the CDN, at a controlled pace, before you route all traffic through it. It names three groups to start with:
- High-traffic pages
- Upcoming peak or promotional pages
- Frequently crawled URLs
For a small WordPress site, a practical source for that list is your XML sitemap plus your top pages in Analytics. Keep the pace gentle so the warm-up itself does not hammer the origin. Our guide to speeding up a WordPress website covers page caching that works hand in hand with a CDN.
4. Keep a temporary brake ready: 429 or 503
If crawl load still threatens the origin during warm-up, Google’s post suggests rate-limiting with a 429 (Too Many Requests) or 503 (Service Unavailable) response, kept in place for a few hours or one to two days at most. Google’s page on reducing crawl rate adds detail:
- Errors reduce the crawl rate for the whole hostname, not only for the URLs that returned the error.
- Once the errors stop, crawling ramps back up automatically.
- For 503 and 429 you can send a Retry-After header, either as a number of seconds or as an absolute date and time in UTC.
- Do not keep error responses in place for long. Google warns that if the same URL returns these codes over several days, it may be dropped from the index.
This is an emergency tool. Google also warns that slowing the crawl means fewer new pages found and slower refreshes, so changes such as prices or stock may lag in Search.
5. Do not use 403 or 404 to slow Googlebot
Google’s HTTP status code documentation says 4xx responses other than 429 do not affect crawl rate, and it warns against using 401 or 403 to limit crawling. Instead, Google treats content from such URLs as nonexistent and removes indexed URLs over time. A WAF rule that answers Googlebot with 403 is therefore a quiet way to lose pages.
| Response | How Google treats it | Use to slow crawling? |
|---|---|---|
| 429 | Treated as a sign of overload; crawlers slow down | Yes, briefly |
| 503 | Crawl rate drops; URLs that keep returning it can be removed from the index | Yes, briefly |
| 403 / other 4xx | Content treated as nonexistent; indexed URLs removed over time | No |
6. After cutover: watch the Crawl Stats report
In Search Console, open Property settings (the gear icon) and select Crawl stats. Google’s help page says the report is only available for root-level properties (a Domain property or a URL-prefix property at the root), and describes it as aimed at advanced users; it says sites with under a thousand pages usually do not need it. If you have just moved to a CDN, though, it is a handy before-and-after check. Look at:
- Host status: flags problems with robots.txt fetching, DNS resolution and server connectivity. Red means an issue in the last week.
- Crawl responses: the share of 200, redirect, 404 and 5xx responses. A jump in 5xx or in robots.txt failures after cutover deserves attention.
- Average response time and total crawl requests: compare with the period before the move.
Google notes the numbers can differ slightly from your server logs, so treat the report as a trend indicator. If pages show up as crawled but not indexed afterwards, our post on Crawled vs Discovered: Not Indexed explains how to read that status.
Common mistakes
- Enabling the strictest bot setting on day one and never testing what Googlebot sees.
- Allowing any request whose user agent says Googlebot, which is trivially faked.
- Leaving a temporary 503 or 429 rule switched on for days.
- Switching DNS for the whole site at once without warming the cache.
Key takeaways
- A faster site can invite faster crawling, and a cold CDN cache sends first requests to your origin.
- Exempt verified crawlers from bot challenges, and verify Googlebot by DNS or Google’s published IP ranges, not by user agent.
- Pre-warm popular, promotional and frequently crawled URLs at a controlled pace.
- If the origin struggles, use 429 or 503 for hours or a day or two at most, never 403 or 404.
- Check Crawl Stats after the move for 5xx spikes, robots.txt failures and response-time changes.
FAQ
Will a CDN hurt my rankings?
Nothing in Google’s guidance says a CDN itself is a problem. The risks it describes come from configuration: blocking verified crawlers with challenges, or overloading the origin while caches are cold.
How do I know if Googlebot is being challenged?
Check your server or CDN logs for requests from verified Google IPs that received error or challenge responses, and look at Crawl Stats for rising error shares.
How long can I return 503 or 429?
Google’s guidance says a couple of hours, or one to two days at most. Beyond that, URLs that keep returning errors may be dropped from the index.
Do small WordPress sites need to pre-warm?
Google’s post does not set a size threshold. A small site with a capable host may never notice a crawl surge, but warming your top pages and sitemap URLs is cheap, and it is easier than diagnosing an overloaded origin later.