Cache Races Make IndexNow Miss Pages You Just Shipped

We publish from a scheduled agent: build, deploy, ping IndexNow, cross-post. For months the ping step returned HTTP 200 on every run and we filed that under done. Then we diffed the list of URLs we had submitted against what crawlers actually fetched, and found a batch of pages that had been pinged, fetched within the minute, and served either the pre-deploy version or a 404.

Nothing errored. Nothing retried. The pipeline reported success on every one of them.

The ping is fast; your edge is not

IndexNow inverts the crawl. Instead of waiting for a bot to rediscover your sitemap on its own schedule, you push a list of URLs and participating engines fetch them soon after. In our logs the first crawler hit typically lands within a minute or two of the ping. That speed is the entire value proposition, and it is also the bug.

The naive pipeline is three steps in one process:

1. Build.
2. Deploy — the API returns success.
3. Ping IndexNow with the new URLs.

Step 2 returning success means the control plane accepted your artifact. It does not mean every edge node is serving the new HTML, and it does not mean cached responses for those paths were invalidated. So the timeline becomes: deploy returns at t+0, ping fires at t+0, crawler fetches at t+40s from a POP that has not caught up yet.

Negative caching is the sharp edge here. A path that did not exist yesterday can already have a cached miss at some edge — from a preview link you opened, a broken internal link, a scanner probing paths. Your deploy adds the page at origin. That edge keeps answering from its stored 404 until the TTL expires. The crawler asks exactly once, quickly, and it asks the edge.

Why the checks you would reach for first prove nothing

Three verification habits that feel rigorous and are not:

Curling the URL with a cache buster. Fetching https://yoursite/for-dev/slug/?v=123 returns your new HTML. That proves origin has the page. It also created a different cache key from the one you submitted. The canonical URL can still be serving stale while your check passes. We shipped two “verified” deploys this way before noticing the query string was doing the lying.

Checking the sitemap. Sitemaps usually carry a longer TTL than HTML pages. A crawler can read a stale sitemap alongside a fresh page, or the reverse. Sitemap freshness and page freshness are separate races; confirming one says nothing about the other.

Trusting the deploy tool’s completion message. Whatever the host — object storage behind a CDN, a worker, a container — the deploy call returns when the artifact is accepted, and propagation is asynchronous. On our worker deploys, the gap between “deploy returned” and “every edge we could sample served new content” ranged from a couple of seconds to over a minute. The tail is where pings die, and the tail is exactly what an unbounded async operation does not report.

The check that holds is narrower than any of those: fetch the canonical URL, no query string, with Cache-Control: no-cache on the request, and confirm you got a 200 and a marker unique to this build. We inject the build’s short git SHA into a meta tag on every page; the poll passes only when the fetched HTML contains the SHA the current run produced. A bare status check will happily pass on a stale-but-valid previous version of an updated article, which is the failure mode you are least likely to notice.

The sequence that survives a cold edge

Ordering matters more than any individual check. What we run now:

1. Build, emitting the git SHA into every page.
2. Deploy.
3. Purge explicitly — the exact new URLs, plus sitemap.xml, plus any machine-readable indexes like llms.txt or articles.json. Purging is what kills a cached negative response. Polling alone just waits out its TTL.
4. Poll each submitted URL individually until it returns 200 with the current SHA. Cap it: we allow roughly 90 seconds per URL at a 3-second interval. URLs that fail get dropped from the submission list rather than pinged anyway.
5. Ping IndexNow with only the URLs that passed.
6. Record what was submitted, with a timestamp and the SHA at submit time.

Step 6 is the one people skip and the one that converts belief into evidence. Without a submission log, a URL that was never pinged is indistinguishable from a URL that was pinged into a 404. We keep a small JSON file keyed by URL; a follow-up job checks days later whether those URLs show up in coverage reports and re-submits the ones that do not.

Two smaller things that cost us runs:

The key file is a single point of failure. The verification key file at your domain root gets fetched by the engine to confirm ownership. If it sits behind the same CDN and any build ships without it, a cached 404 there invalidates the entire batch — not one URL, all of them. Treat it as a static asset with a long TTL and never let it be conditionally generated.

Do not submit URLs that redirect. We moved older posts from a flat path to audience-prefixed paths. Submitting the old URL wastes the slot: the engine follows the 301, but the canonical you wanted indexed was the target all along. Submit whatever your URL helper produces for the current build, not the path the previous build used.

The same race applies well beyond search crawlers, and the blast radius elsewhere is worse. When a social post or a cross-post platform unfurls your link, it fetches your OG tags once and caches the result for a long time — sometimes indefinitely, with manual re-scrape as the only fix. A crawler that gets a stale page will come back. A link preview that gets a 404 keeps showing a broken card until you go clear it by hand. Gate the announcement fan-out on the same verification, not just the ping.