A price change goes live on a product page at 9 a.m. The origin serves the new price immediately, yet three hours later the old price is still rendering, and the version Googlebot last fetched is the stale one. Nothing is broken at the origin. The stale response is coming from the CDN, where an HTML caching rule set too aggressively is holding the previous version until its TTL expires. The layer that makes the site fast is quietly deciding which version of a page reaches users and search engines.
Most sites route traffic through a CDN by default now: Cloudflare, Akamai, Fastly, AWS CloudFront, Google Cloud CDN, BunnyCDN. Teams pick one for speed, security, and cost. The less obvious point is that the configuration inside that CDN shapes crawl efficiency, Time to First Byte, and which version of a page gets indexed. This is edge-layer technical SEO, and it usually gets configured by people who are not thinking about search at all.
Where a CDN touches SEO #
A CDN sits between users and the origin. A request hits the CDN first; if the content is cached and still valid, the CDN returns it directly, otherwise it fetches from origin, caches the response per the rules, and passes it back. That single decision point controls several things search cares about: caching rules (what gets stored and for how long), TTFB (which feeds Googlebot’s crawl capacity, since Google adjusts crawl rate to server response time), cache headers (what Googlebot’s renderer holds between fetches), edge geographic distribution (how fast each region experiences the site), bot management (whether Googlebot gets through or gets challenged), and edge logic (whether redirects and rewrites run at the edge or origin).
A correctly configured CDN accelerates everything the crawler does. A misconfigured one can break indexation in ways that are hard to trace, because the origin still returns correct responses when tested directly: the fault lives at the edge.
Caching rules by content type #
Caching rules determine what the CDN stores and for how long, and the right answer differs by content type.
Static assets (CSS, JS, images, fonts). Long TTLs work because these assets are versioned with a content hash in the filename; when the file changes its URL changes, so the old cache entry stops being requested. The standard pattern is Cache-Control: public, max-age=31536000, immutable, a one-year cache marked immutable so caches never revalidate it. Googlebot benefits too, since it does not refetch unchanged assets during rendering.
HTML pages. This is where most CDN-related SEO damage happens. Short TTLs (roughly 5 to 60 minutes) suit pages that change regularly: long enough to absorb a spike, short enough that edits propagate quickly. Longer TTLs are fine for genuinely static pages like terms of service; personalized pages should not be cached at all. The failure pattern is caching HTML too long. A product page on a 24-hour TTL means a price or inventory change can take a day to reach Googlebot’s view: the site gets crawled on schedule, but the crawled version is stale.
API responses. Short TTLs fit most cases, since the underlying data changes often. Cache keys can include query parameters or headers, but each dimension of variation lowers the hit rate, so keys should stay as narrow as the content allows.
TTFB and why it moves both directions #
Time to First Byte measures how long after a request reaches the network the server starts sending the response, and a CDN moves it in both directions. It comes down through geographic proximity (a user in Singapore hitting a nearby edge sees far lower TTFB than one reaching an origin in Virginia), on cache hits where the edge answers without contacting origin, and when TLS termination at the edge offloads the handshake. It goes up on cache misses, which add CDN-to-origin latency on top of origin processing time, and when origin-shield indirection adds a hop or edge functions run on every request.
For SEO, low and consistent TTFB matters for two reasons. Google’s crawl-rate calculation factors in response time, so a faster server tends to earn more crawl capacity. And TTFB is the first component of Largest Contentful Paint; a slow first byte drags LCP down before any rendering begins. LCP has its own threshold and its own topic, but the CDN’s contribution to it starts here. Measuring TTFB by region reveals whether the edge network is delivering: 200ms at home but two seconds in Asia means the edge coverage is not reaching that geography.
Cache headers and Googlebot’s renderer #
HTTP cache headers tell every intermediary (CDN, browser, and Googlebot’s renderer) what to cache and for how long. Cache-Control is the modern standard, carrying directives such as public/private, max-age, s-maxage (the freshness window for shared caches like a CDN, overriding max-age there), immutable, no-cache, and no-store. ETag and Last-Modified enable cheap revalidation through conditional requests (If-None-Match and If-Modified-Since). Vary declares which request headers change the response, which matters for cache correctness when content differs by language or user agent.
Google’s Web Rendering Service caches resources it fetches during rendering for up to 30 days, regardless of the max-age the headers specify. Long TTLs with proper versioning let the renderer reuse cached files across pages; no-cache or no-store on JS and CSS forces a refetch every time, slowing rendering and burning crawl capacity. Because a resource’s URL is its cache key, changing that URL forces a refetch even when the contents are identical, so cache-busting through unnecessary URL churn quietly costs crawl budget. The pattern that breaks: no cache headers at all, or contradictory ones such as Cache-Control: max-age=3600 alongside Expires: 0.
Cache invalidation and the staleness tradeoff #
Caching rules set the refresh schedule; invalidation handles content that needs to refresh sooner than the TTL allows. The tradeoff is simple: long TTLs maximize hit rates and cut origin load but widen the stale-content window, while short TTLs keep content fresh at the cost of caching benefit. Invalidation lets a site keep long TTLs and still push updates on demand. Three patterns cover most needs.
Purge by URL. The site tells the CDN to drop one cached response, and the next request refetches from origin. This fits targeted updates: an edited article, a restocked product, a price change. Deployment pipelines commonly trigger these as part of a release.
Purge by tag or surrogate key. The site tags related responses during caching, then purges everything carrying a tag in one call. This fits changes that ripple across pages, where one product update should invalidate the product page, category page, and homepage at once. Fastly’s surrogate keys and Cloudflare’s cache tags both implement this; it depends on consistent tagging in the application layer, or purges miss pages that should have refreshed.
Stale-while-revalidate. The CDN serves the stale response immediately, then fetches a fresh copy in the background so the next request gets the update. The directive Cache-Control: max-age=60, stale-while-revalidate=86400 means fresh for 60 seconds, with a stale copy acceptable for up to 24 hours while revalidation runs. Users never wait; the tradeoff is a brief window where they may see content from just before the latest change.
Four SEO-specific concerns sit on top of these mechanics. Long-cached HTML means Googlebot may crawl the stale version, with the index reflecting old content until the cache expires or is purged, so active pages need short HTML TTLs or a purge on every meaningful update. Sitemaps cached for hours delay discovery of new URLs, so they want short TTLs (5 to 15 minutes) or none. A cached 404 from a transient deploy error keeps returning 404 after origin recovers, so 4xx responses should not be cached, or only briefly. And a purge takes a moment to propagate globally, briefly showing different content by region; most CDNs converge within a minute, but high-traffic sites should expect the gap.
How to keep the CDN aligned with SEO #
Treat the CDN as a system whose state needs explicit management rather than a passive speed boost. A short recurring review, quarterly or after any major site change, catches these failures before they compound:
- Check HTML TTLs. Confirm dynamic pages are on short TTLs and no product or content template is inadvertently cached for hours.
- Confirm Googlebot is not blocked. Bot-management features, “under attack” modes, rate limiting, and IP blocks can all catch the crawler, since Googlebot does not solve challenges. Enable the CDN’s built-in verified-bot category (Cloudflare and most major CDNs ship one that includes Googlebot) and verify crawlers the way Google documents: a reverse DNS lookup confirmed by a forward lookup, or a match against Google’s published IP-range files. Allow verified search crawlers regardless of other rules.
- Measure TTFB by region. Geographic field or synthetic data reveals thin edge coverage and misplaced origin shields.
- Watch edge logic for cloaking risk. Edge functions that redirect or rewrite are fine when they treat all traffic equally; logic that serves Googlebot different content than users is a cloaking violation. Vary behavior by signals other than user agent.
- Scope AI-crawler blocking carefully. Many sites now enforce AI-crawler policy at the edge rather than trusting robots.txt compliance. Cloudflare’s AI Crawl Control (launched as “AI Audit” in September 2024, generally available August 2025) categorizes AI crawlers and offers one-click blocking; Fastly, Akamai, and AWS WAF managed rules are comparable. Broad blocks can catch Googlebot or Bingbot too, so the rule should allow verified search crawlers, block verified AI crawlers, and challenge only unverified traffic.
Frequently asked questions #
Will caching HTML cause Google to index stale content? It can. If a page is cached for 24 hours and changes, Googlebot may fetch the cached copy, and the index holds the old version until the cache expires or is purged. Keep HTML TTLs short for pages that change, or purge on update.
What Cache-Control values should static assets use? public, max-age=31536000, immutable is the standard for hash-versioned assets, since a content change produces a new URL and retires the old cache entry.
The CDN is rarely the dramatic part of technical SEO, and its settings usually belong to performance, security, or DevOps teams rather than search. But those settings decide TTFB, which pages Googlebot sees, and how fast changes reach the index. Getting the SEO team into CDN configuration reviews, and watching for TTFB drift, indexation gaps, and sudden crawl-rate changes, is often where the difference between a site that performs and one that quietly does not gets settled.