Incremental IndexNow pings for a 15-locale static site: hash the HTML, not the timestamps

RefCodeHub — Fri, 30 Oct 2026 02:41:25 GMT

RefCodeHub is a static site built with Astro: 15 language versions, 375 indexable URLs in 15 per-locale sitemaps (state of the build dated 22 October 2026). Every page carries hreflang alternates, so a single data change (a new fee table, a refreshed regulator register) can touch dozens of HTML files at once. Pinging every URL on every deploy would be noisy; pinging nothing means Bing and other IndexNow engines see changes late. Here is the small setup we ended up with.

1. Make the build deterministic

If two builds of the same content produce different bytes, every page looks "changed". We removed the sources of noise: no build timestamps in HTML, sitemap lastmod equals the page's dateModified, and that date is the newest data date for the page (referral terms check, newest code, register download, fee check), not the deploy date. Before hashing, the script strips framework-generated data-astro-* attributes, which can differ between builds.

2. Hash pages and diff against the last successful ping

scripts/indexnow.mjs walks dist/, skips non-indexable pages, hashes the rest and compares with .indexnow-state.json. Only new, changed or removed URLs are POSTed to https://api.indexnow.org/indexnow. A rebuild without content changes sends nothing.

3. Save state only after the API accepts

Before sending, the script checks that the key file is actually live on the production domain.
A 403 right after a deploy usually means the key file is not visible yet, so it retries (up to 3 times, 30 s apart).
State is written only on 200/202. Any failure means the next run resends the same diff.
--dry-run lists what would be sent; --all resubmits everything.

4. Know what IndexNow does not do

Bing, Yandex, Seznam and Naver use IndexNow; Google does not. For Google, the accurate lastmod values in the sitemaps do the job, so step 1 pays off twice.

Takeaways

Fix determinism first; incremental pings are useless without it.
Persist "what was accepted", not "what was built".
Treat a 403 after deploy as a propagation problem before blaming the key.

The site itself compares crypto exchange referral programmes: what a code actually gives, explained on how referral codes work, and the side-by-side exchange referral bonus comparison.