Incident Report
At 10:00 UTC, CI deployed a routine workspace schema migration (v2) along with new content-hashed JavaScript bundles and pruned old v1 chunks from the CDN bucket. Fresh incognito smoke tests passed immediately.
Within 12 minutes, RUM telemetry and customer support escalated a multi-tab failure affecting long-lived sessions.
- T+00:00:
workspace-app@2026.11.16.2rolls out; CDN purgesv1lazy chunks. - T+04:00:
ChunkLoadErrorrate spikes to 4.8% among users who opened a second tab after the deploy. - T+12:00: 310 support tickets report missing offline drafts after users clicked the Back button from an external OAuth redirect.
- T+18:00: IndexedDB telemetry confirms
v1tabs writing deprecated record shapes intov2object stores.
The Red Herring
The first Slack reflex was to call self.skipWaiting() inside the Service Worker install handler and attach a window.location.reload() listener on ChunkLoadError. Locally, hard-reloading a single tab cleared the error.
In production, that patch worsened the outage. skipWaiting() hijacked active tabs mid-transaction, while back-button navigations still restored frozen v1 JavaScript heaps from bfcache without triggering a network fetch or Service Worker intercept, writing legacy records into the upgraded v2 database.
Causal Chain
A deploy skew incident is a five-layer state divergence across worker registration, HTTP cache headers, the ECMAScript module map, browser history snapshots, and client-side storage schemas.
- The active Service Worker serves the cached
v1index.htmlmanifest to existing tabs. - CDN
Cache-Controlrules either 404 pruned chunks or serve mixed-version assets. - The browser ESM module map preserves live bindings to
v1exports alongsidev2dynamic imports. -
bfcacherestores a frozenv1heap after another tab upgraded the origin's database. - The zombie
v1runtime writes unmigrated records into av2IndexedDB object store.
Failure Simulator
Layer Dissection
01 → 02: Service Worker Ownership Meets HTTP Cache Skew
Under the Service Worker Lifecycle, a newly deployed sw.js enters waiting until every client controlled by v1 unloads. While v1 controls Tab A, it intercepts navigation requests and returns cached index.html referencing v1 chunk hashes.
When traffic hits the CDN edge directly, HTTP Caching Directives compound the split:
-
index.htmlwithstale-while-revalidatereturns yesterday's manifest during the stale window. - Hashed chunks carry
Cache-Control: public, max-age=31536000, immutable, so the browser skips revalidation. - Once the CDN prunes old build artifacts,
v1HTML requests av1lazy chunk and receives HTTP 404.
02 → 03: HTTP Split Corrupts Live ESM Bindings
When a dynamic import() resolves against a newer build while a shared runtime chunk resolves from cache, the module loader does not clone values. Under ES Modules Live Bindings, every importer holds a read-only reference to the exporter's live memory slot.
If an unhashed entrypoint survives in cache while a feature chunk loads from v2, v2 code reads v1 schema constants or throws SyntaxError on missing named exports during module linking.
03 → 04: Module Skew Survives Inside bfcache
When a user navigates to an external OAuth page, the browser snapshots the entire v1 JS heap and DOM into memory. Under bfcache Navigation, pressing Back restores that snapshot synchronously (pageshow with event.persisted === true).
No HTTP request fires and no Service Worker fetch event executes. Meanwhile, Tab B has already booted v2 from the network and executed onupgradeneeded on the shared origin database.
04 → 05: Zombie Runtime Corrupts Client Storage Migrations
IndexedDB is origin-scoped, not tab-scoped. When Tab B bumps the database version from 1 to 2, Client Storage Schema Migrations require every active tab to close its handle on versionchange before the upgrade proceeds.
A bfcache-restored tab that ignored versionchange still executes v1 serialization code. Opening a new readwrite transaction against the upgraded v2 store writes v1-shaped objects that violate v2 index key paths and corrupt shared origin state.
Architectural Fix & Invariants
const APP_VERSION = '2026.11.16.2'; window.addEventListener('pageshow', (event) => { if (event.persisted && localStorage.getItem('active_build') !== APP_VERSION) { window.location.reload(); } }); const openReq = indexedDB.open('workspace_db', 2); openReq.onsuccess = () => { const db = openReq.result; db.onversionchange = () => { db.close(); window.location.reload(); }; }; Architectural Invariants:
- Immutable chunks, zero-stale HTML: Serve
index.htmlwithCache-Control: no-cacheand retain previous build chunks on the CDN for at least 24 hours after deploy. - No blind
skipWaiting(): Activate a waiting Service Worker only when no active transactions are open, then reload all controlled clients atomically. - Mandatory
versionchangeteardown: Implementonversionchangeon everyIDBDatabasehandle to calldb.close()and halt writes from outdated bundles. - Revalidate on
persistedrestore: Compare the runningAPP_VERSIONagainst the active origin version on everypageshowevent whereevent.persisted === true.
Golden Rule: A frontend deploy is never atomic across open tabs and bfcache; fence every storage write behind a versioned onversionchange and pageshow check.


