Last updated on October 23rd, 2026 at 12:44 pm
when many pages disappear from search, check whether the CDN or web application firewall is blocking crawlers before rewriting the theme or content.
Why it happens
A page can look correct in the application and still be inaccessible at the network edge. A CDN or web application firewall sits in front of the origin server and may return an HTTP 403 response before WordPress handles the request.
HTTP 403 means the request was understood but refused. If a search crawler repeatedly receives that response, it cannot reliably fetch the page for indexing.
A simple diagnostic process
- Check the status code. Request one affected URL and confirm whether it returns 200, 403, or another status.
- Compare clients. Test a normal browser request and a request using a crawler user agent. A difference points toward an edge security rule.
- Check shared files. Verify that robots.txt and the XML sitemap are also reachable without a challenge.
-
Inspect the CDN. Review firewall events, bot protection, rate limits, and managed challenge rules.
-
Retest externally. After changing the rule, confirm that affected pages return 200 without cookies or an interactive challenge.
Cause and fix
Cause: a broad bot rule challenges or blocks legitimate crawlers.
Fix: narrow the rule and allow verified search crawlers using the CDN provider’s supported verification method.
Cause: rate limiting treats crawler traffic as abusive.
Fix: adjust the threshold for public pages while keeping protection on sensitive endpoints.
Cause: the homepage works, but deeper routes receive 403 responses.
Fix: test representative URLs from each template instead of checking only the homepage.
Common mistakes
Changing titles, templates, or page text will not solve an access-layer 403. Editing robots.txt also will not help when the crawler cannot reach the server successfully.
Do not permanently disable the firewall as a shortcut. The safer approach is to identify the matching rule, narrow its scope, and verify the result.
Verify the crawler, not just its user agent
A user-agent header is easy to spoof. Changing it in curl can reveal a rule difference, but it does not reproduce a verified Googlebot request. Use Search Console URL Inspection and edge event logs together. After crawl access is restored, indexing can take time; a 200 response does not guarantee inclusion in search.
When de-indexing affects many pages at once, start with crawl access. The next action is simple: test one affected URL as a crawler and confirm that it returns HTTP 200 without a challenge.
Reference: Google crawl requirements.