Your first job on an erasure request is a search: where does this person's data sit? For a customer with one online order, the example estate in this series holds it in more than twenty places across the source database, the lakehouse, and the records around it. Some are obvious, such as the orders table and its bronze copy. Others are easy to miss: the change tables that feed replication, a month-end snapshot, the text of a query someone ran to look into a refund, and the table that caught a pipeline error. Each has its own rule for how long data stays after a delete, and several keep it for weeks after the DELETE statement succeeds.

This series is for teams that have built a medallion lakehouse on Google Cloud and now have to run it. It assumes the layers are in place, as described in our post on the medallion architecture. Part 1 starts with bronze, because bronze is where replay comes from and also where personal data is often kept longest, in its least processed form. The aim is an erasure process that finishes by a date the platform can compute for its own stores. That date applies to erased data. A lot of order data is kept on purpose under legal obligations, and this post keeps the two apart.

The estate is a composite built for this series, close to the one in our post on multi-cloud retail: Microsoft Entra ID for workforce identity (which a later part on access uses), Google Cloud for the data platform, and Azure SQL Database under an online retailer's operational systems.

The storage options this series refers to

Several parts of this series depend on how bronze, silver, and gold are stored, so we give three storage options a letter here. A later part compares them in depth.

  • A. BigQuery native tables, stored and managed entirely by BigQuery.
  • B. Apache Iceberg managed tables (formerly BigLake tables for Apache Iceberg in BigQuery). Data files sit in your Cloud Storage bucket and BigQuery manages them. Google garbage-collects data files after the time travel window, but not the files of a dropped table, and recommends keeping the bucket's default soft delete policy (Google Cloud, n.d.b).
  • C. Iceberg in an open catalog, such as Google's Apache Iceberg REST catalog endpoint, with other engines writing the files and you running the file maintenance. BigQuery can't create or modify tables in that endpoint with DDL or DML (Google Cloud, n.d.y).

This part works through the design on options A and B, where BigQuery runs both the split job and the erasure job. On option C, those jobs run in Spark against the catalog instead. The file chain (delete files, compaction, snapshot expiry, and orphan cleanup) belongs with the storage comparison.

Option B carries one constraint that shapes the design. Google lists authorised views as unsupported for managed Iceberg tables (Google Cloud, n.d.b). In this design, readers reach bronze, the retained store, and the identity table only through authorised views. So on option B those three stay native tables, as do the stores those views read (the suppression entries, the reopen table, the hand-off table, and the kept list). Managed Iceberg holds silver and gold, whose readers have table access anyway. If you put bronze on managed Iceberg, you have to give silver and analysts read access on the tables, and that lets them read deleted rows during the time travel window. The option B rows in the ledger and the worked example cover that case's timing. That read exposure comes on top.

Where this design follows the textbook medallion, and where it departs

Google's guide to the medallion architecture describes four implementation steps (Google Cloud, n.d.ag). The first three build the layers. Ingestion moves data into bronze "without changing it", so no information is dropped or accidentally altered on the way, and bronze acts as a permanent, historical archive. Building silver uses Spark or SQL to apply data quality rules, such as converting timestamps to UTC or removing test accounts. Gold comes from SQL queries that join tables into metrics like revenue by category, often as high-performance views that a tool such as Looker can read directly.

The fourth step, orchestration and governance, names Managed Service for Apache Airflow for scheduling. Its governance example is letting only a few authorised service accounts write to silver and gold. The page's list of Google Cloud products also names Dataform for building and scheduling the transformations that turn silver into gold (Google Cloud, n.d.ag).

This series follows those steps, with SQL for every move between layers on options A and B, and Spark SQL on option C. It's one approach, and we've seen it work well where the data already sits in BigQuery: the work runs where the data is, with no separate cluster to run and no data to move out. On the estates we've worked on, it has usually been cheaper and faster to run, and quicker to build and change, than the same work in pipeline code. The right pattern depends on the data, team, and existing tooling, and another estate can call for another.

The tool that runs that SQL varies by engagement. We've used Airflow, Dataform, and dbt (scheduled by the existing orchestrator or by dbt Cloud), usually whatever you already run. The example estate uses Managed Service for Apache Airflow. It schedules every job in this post and submits each SQL script to BigQuery as a job under that job's own service account. The infrastructure account creates the gold view once. The hardening section sets out the access rules for Airflow and what changes under Dataform or dbt.

Personal data calls for one departure. Google's page doesn't cover it, and a bronze layer that keeps every customer's identifiers as a permanent archive makes the storage limitation and erasure duties hard to meet. So this series splits Google's bronze in two:

  • A restricted raw dataset takes the source rows as received, except the email (which in the main design never leaves the source), and keeps them only for a short retention period.
  • Bronze holds the longer-lived history for replay, kept for its purpose periods, with direct identifiers replaced by a token.

Google's steps leave every change to the data until silver. This design tokenises before bronze, so the history kept longest holds the least.

There are two smaller differences. Google says data scientists often want raw bronze data to train models. Here they get tokenised bronze through authorised views, and no person or analyst reads raw. Google also names Cloud Storage as the bronze landing zone. In this design raw lands in BigQuery through Datastream, and on option B, bronze's data files sit in Cloud Storage under BigQuery's management.

What bronze is kept for, and what has to be kept anyway

After launch, bronze usually serves three purposes: replaying history when a rule changes or a bug is found, answering audit questions about what arrived and when, and debugging pipelines. Each needs a different amount of history. In the estates we've seen, retention is easier to justify per purpose than as one number for the whole layer.

GDPR's storage limitation principle in Article 5(1)(e) asks that personal data be kept in a form that permits identification of people "for no longer than is necessary for the purposes for which the personal data are processed" (European Parliament and Council, 2016). In California, section 1798.100(a)(3) of the Civil Code requires businesses to disclose how long they intend to keep each category of personal information or, if that isn't possible, the criteria used to decide it (California Legislature, n.d.a), and the regulations require the same in the notice at collection (California Privacy Protection Agency, n.d.a).

Retention runs the other way too. Orders, invoices, payments, and returns are often subject to tax and accounting retention, and both regimes allow for this. Under Article 17(3)(b) and (e), GDPR's right to erasure doesn't apply to the extent processing is necessary "for compliance with a legal obligation" under Union or Member State law, or "for the establishment, exercise or defence of legal claims" (European Parliament and Council, 2016). Section 1798.105(d) of the CCPA has an exception for complying with a legal obligation (California Legislature, n.d.b). So the first step in this design is to sort event families into two groups. In the running example:

  • Erased: saved carts, recently viewed items stored by the application, wishlists, and reviews. Marketing preferences are erased too, with one exception covered below.
  • Kept under a stated obligation: orders, order lines, invoices, payments, and returns, with shipment events following their orders. They're access-restricted and minimised, and recorded in the erasure log with their retention and basis. Where the obligation allows, they lose their token-to-person mapping.

Where an obligation requires identifying fields, such as the name and address on an invoice, those fields stay in the retained record under the record's own access and retention. You can still trace retained records to the person through the source system's own retained records, so they stay personal data for as long as those exist.

That split is an illustration. Which records a business has to keep, and for how long, depends on where it trades. This post doesn't set out any country's retention periods.

Three ways to make bronze erasable

Rewrite on delete. Keep identifiers in the events and delete or rewrite the person's rows wherever they are. It needs nothing extra at landing, but every erasure scans every table that might hold the person, and a missed table means a missed erasure.

Separate identifiers with a random token. Split identifiers from events soon after landing. It costs a split job and an identity table, and each erasure then starts from a short, known list of tokens.

Per-subject encryption. Encrypt each person's fields with their own key, and delete the key to make the data unreadable. Google describes deleting a key to render data unreadable as crypto-shredding, and BigQuery's AEAD functions can use keysets wrapped by Cloud KMS (Google Cloud, n.d.a). Each person's keyset would usually sit in a table. Deleting its row is then subject to that table's time travel, with fail-safe assumed as an upper bound.

If the keysets are wrapped, destroying the wrapping KMS key version means no keyset wrapped by it can be unwrapped again. That suits shredding a whole dataset more than one person. The key material is destroyed only after a scheduled period of 30 days by default, and Google says it can then remain in its systems for up to 45 days from the scheduled destruction time (Google Cloud, n.d.j).

Encryption also covers only the copies that stayed encrypted. Values decrypted into silver, gold, caches, or extracts aren't affected. When the KMS resource name and wrapped keyset are passed to KEYS.KEYSET_CHAIN as constants, they can appear in the diagnostic query plan, though Google says the keys are never exposed in plaintext and decrypting the wrapped keyset needs permission on the Cloud KMS key (Google Cloud, n.d.e). NIST classes cryptographic erase as a purge technique for media sanitisation (National Institute of Standards and Technology, 2025). Its conditions include, as NIST says ISO/IEC 27040 specifies, being able to sanitise all copies of the key. In our reading, keyset rows held in time travel are copies. We haven't found EU guidance that treats key deletion as erasure, so this design uses crypto-shredding, where it uses it at all, alongside deletion rather than in place of it.

Each fits a different context. Where your estate has many event tables and erasure requests arrive regularly, we've seen the token approach work well, with rewrite-on-delete for the few stores that have to hold identifiers. That's the design the rest of this post builds. If yours has few tables, or identifiers that can't leave the events, rewriting on delete may serve you better. Per-subject encryption suits retiring a whole dataset more than erasing one person.

The design in brief

These terms carry most of the post.

  • Source key: the customer's account ID in the source system.
  • Customer key: a pseudonymous join key, an HMAC of the normalised email address, computed by one key service. Guest orders and carts carry one even though they have no source key. The split gives each guest customer key an identity row of its own, with no source key, so erasure finds a guest's records by customer key. Retained records carry the token in place of the customer key, for guests too.
  • Token: a random value that stands in for the person on bronze events.
  • Identity table: the mapping from token to source key and customer key. Only the split and erasure jobs, and any named scanner, read or write it directly. Analysts and silver jobs get tokens and never the mapping.
  • Suppression entries: the keys of people who have asked for erasure, checked by the split job before anything new is written.
  • Retained store: the restricted home of event families kept under an obligation.
  • Erasure time: when the source system's erasure of the person took effect, fixed once by the erasure job and used to tell the person's old events from a returning customer's new ones.
  • Request intake, queue, and erasure log: the service account that records verified requests, the table where a request's keys wait for the erasure job, and the job's record of what it did, which holds no keys or tokens.
  • Hand-off table and kept list: where the erasure job passes a person's tokens, with a pass number, to the jobs that erase silver and gold; and the identity rows an obligation requires to be kept, with their basis.
  • Horizon: the computed date by which erased copies have left the pipeline's own stores.
  • Cutoff and landing check: the last split run whose identity rows count as the person's own, recorded by the erasure job at its first step; and the test that the source erasure has reached the raw dataset.

Figure 1 shows where one customer's record lands, with the backfill path drawn separately from CDC.

Figure 1. Where one customer's record lands in the example estate. The source holds application tables, change tables, and backups of both. CDC and backfill bring rows into the restricted raw dataset. The split job works from its staging tables, checks each event against the suppression entries and adds suppressed record keys, and writes bronze events, the retained store, identity rows, and marks for the raw rows it has finished with, identity-source rows among them, to which the erasure job adds 'erased' marks. Keys, mappings, and erasure records sit in an isolated group: the suppression entries and suppressed record keys, the erasure queue, the reopen table, the identity table, with one row per source key, or for guests one per customer key and token, and the hand-off table and kept list. Request intake appends the person's keys to the suppression entries and writes the queue row. Silver reads bronze through an authorised view and the retained store through a minimised view that nulls tokens on the kept list or in the hand-off table; gold reads silver, and other downstream recipients are fed from gold or the export view. The export view takes token and customer key from the identity table and excludes erased people and reopened tokens. Copies and records can hold any of these stores, raw and bronze among them. The landing, split, and isolated stores follow the isolation rule in the hardening section, and on option B bronze, the retained store, the identity table, and the stores their views read stay native tables.

One customer, end to end

Take one account holder with a wishlist and one order. In the application's customers table, their row holds their email next to the customer key the key service returned. Their wishlist and order rows hold their source key.

Datastream copies changes to all three rows into the restricted raw dataset, leaving out the email. From there the split job does three things:

  • It creates an identity row, one per source key, with a new token, the source key, and the customer key.
  • It writes the wishlist events to bronze with the token in place of the account ID.
  • It writes the order to the retained store, also with the token, because orders are kept under an obligation.

An analyst who counts wishlist items by category reads a gold metric with no token in it. That metric is built from a silver table that carries the token.

A month later, the customer asks for erasure. Request intake adds their source key and customer key to the suppression entries, so no new parent event keyed to them reaches bronze. Child rows of records already in bronze are removed in steps 1 and 3. The source system erases the customer row on its own schedule, and the erasure job works through its steps:

  • It deletes the wishlist events by token (step 1).
  • It waits for any split run already under way, and checks again (steps 2 and 3).
  • It hands the token to the jobs that erase silver and gold (step 4).
  • Once they confirm, it deletes the identity row and clears the token from the retained order (step 5).
  • When the source erasure has landed, it deletes the customer's rows from raw (step 6).

The order stays in the retained store, linked to its invoice by order ID. It has no token and no mapping back to the person, though it keeps any identifying fields its obligation requires. Bronze, the identity table, and raw still hold the deleted rows for their time travel and fail-safe windows, and the post works out when those windows close.

The tables, and the SQL between the layers

Table 1. Tables in the listings. It groups them by layer. We explain each column where its rule comes up: the erasure time and cutoff in stage 2, and split runs in the split job section. The names are illustrative. Each erased family, such as carts, has tables of the same shape.

Table Layer Holds Written by Read by
raw.* Raw Source rows as received, minus the email Datastream Split, erasure job, cleanup job
staging.wishlist_changes, staging.wishlist_item_changes Raw Raw rows the split hasn't processed yet Split, as new tables each run, named per run (shown without the suffix), each with a one-day expiry Split
identity.identity_map Keys Token, source key, customer key, and the split run that set the key (key_set_run): one row per source key, or for guests one per customer key and token, so a returning guest gets a second row Split; the erasure job deletes rows Split, erasure job, export view
suppression.source_keys, suppression.customer_keys Keys Erased keys with their request ID, request time (requested_at), and erasure time (erasure_ts) Intake appends; the erasure job fixes the erasure time Split, erasure job, export view
suppression.record_keys Keys Suppressed record keys with their family, request ID, and how far each reaches (suppress_through) Split, erasure job Split, erasure job
erasure.queue_keys Erasure The request's queue row: its keys and record keys, one table row each, every one carrying the request's cutoff (cutoff_run) once step 1 records it Intake, erasure job Erasure job
split_marks.marked_rows Raw Raw rows finished with, by row ID, reason, record key, and request ID Split, erasure job Split, cleanup job, erasure job
ops.split_runs, ops.committed_split_runs, ops.merge_watermarks Operations Run numbers and start times only Split; silver merge Split staging, silver merge, erasure job
bronze.wishlists, bronze.wishlist_items Bronze Events with the token, record keys, the raw row each came from (raw_row_id), the split run that wrote them (split_run), and an ordering field for changes in the same instant (change_seq) Split; the erasure and bronze retention jobs delete Split, erasure job; silver through bronze_views
bronze_views.wishlist_items Bronze An authorised view, storing nothing Infrastructure account Silver merge
silver.wishlist_items, silver.orders, silver.products Silver Cleaned rows; items and orders carry the token Silver merges (only the items merge is shown); silver's erasure job deletes items and clears order tokens Gold view (items and products); other gold jobs
gold.wishlist_items_by_category Gold A view of counts, storing nothing and carrying no token Infrastructure account Dashboards
erasure_handoff.tokens, erasure_confirm_silver.confirmations Erasure Tokens with request ID and pass number; confirmations with no token Erasure job; silver's erasure job Downstream erasure jobs; erasure job

Listing 1 traces the wishlist from bronze to gold, in the shape of Google's steps. A MERGE rebuilds each changed item in silver from its bronze history. Along the way it converts timestamps to UTC and drops test accounts. A view then joins silver tables into a gold metric. You'll find the split that writes bronze in Listing 2, in the split job section, and the erasure DELETEs in Listing 3, in stage 2. Each block runs as its own script. Silver reads and moves its watermark inside its own transaction, so it needs no per-run values.

SQL

-- Two scripts: the merge into silver, run after each committed split run, -- and the gold view, created once. -- Bronze to silver: recompute each changed item from its full bronze history, -- read through an authorised view, up to the latest committed split run. -- Timestamps become UTC, and an item whose latest change is a delete, or -- comes from a test account, leaves silver. The watermark is read and moved -- inside the transaction, so this script needs no per-run values. DECLARE last_merged INT64; DECLARE merge_to INT64; BEGIN TRANSACTION; SET last_merged = (SELECT IFNULL(MAX(last_merged_run), 0) FROM ops.merge_watermarks WHERE target = 'silver.wishlist_items'); SET merge_to = (SELECT IFNULL(MAX(split_run), 0) FROM ops.committed_split_runs); MERGE silver.wishlist_items AS s USING ( SELECT token, wishlist_id, product_id, change_type, added_at_utc, is_test FROM ( SELECT e.token, e.wishlist_id, e.product_id, e.change_type, TIMESTAMP(e.added_at_local, e.time_zone) AS added_at_utc, IFNULL(e.is_test_account, FALSE) AS is_test, ROW_NUMBER() OVER (PARTITION BY e.wishlist_id, e.product_id ORDER BY e.source_ts DESC, e.change_seq DESC) AS latest FROM bronze_views.wishlist_items AS e JOIN ( SELECT DISTINCT wishlist_id, product_id FROM bronze_views.wishlist_items WHERE split_run > last_merged AND split_run <= merge_to) AS changed ON changed.wishlist_id = e.wishlist_id AND changed.product_id = e.product_id WHERE e.split_run <= merge_to) WHERE latest = 1 ) AS b ON s.wishlist_id = b.wishlist_id AND s.product_id = b.product_id WHEN MATCHED AND (b.change_type IN ('DELETE', 'UPDATE-DELETE') OR b.is_test) THEN DELETE WHEN MATCHED THEN UPDATE SET token = b.token, added_at_utc = b.added_at_utc WHEN NOT MATCHED AND b.change_type NOT IN ('DELETE', 'UPDATE-DELETE') AND NOT b.is_test THEN INSERT (token, wishlist_id, product_id, added_at_utc) VALUES (b.token, b.wishlist_id, b.product_id, b.added_at_utc); MERGE ops.merge_watermarks AS m USING (SELECT 'silver.wishlist_items' AS target) AS t ON m.target = t.target WHEN MATCHED THEN UPDATE SET last_merged_run = merge_to WHEN NOT MATCHED THEN INSERT (target, last_merged_run) VALUES (t.target, merge_to); COMMIT TRANSACTION; -- Silver to gold: a plain view that joins silver tables into a metric for -- dashboards. It stores nothing and carries no token or key. The -- infrastructure account creates it once and authorises it on silver's -- dataset, so its readers need no access to silver; no job recreates it. CREATE OR REPLACE VIEW gold.wishlist_items_by_category AS SELECT DATE(w.added_at_utc) AS day_added, p.category, COUNT(*) AS items FROM silver.wishlist_items AS w JOIN silver.products AS p ON p.product_id = w.product_id GROUP BY day_added, p.category; 

Listing 1. Bronze to silver and silver to gold for one wishlist, in BigQuery SQL.

On option A, the gold view is authorised on silver's dataset, so dashboard readers don't need access to the silver tables that carry the token. If silver is a managed Iceberg table, the view can't be authorised. In that case gold is a table built by its own registered job.

A plain view like this gold one stores nothing, so it needs no erasure job of its own. Silver can only be rebuilt from bronze, and by the time the token is handed off, bronze holds nothing for it. Gold is different when it's a table, or, on option A, a materialised view built for speed (our reading of Google's "high-performance views"). Then it holds aggregates computed from the person's rows until it's refreshed or rebuilt. A table has its own time travel, and a materialised view needs the same check. That gold job registers for the hand-off and confirms only after the refresh. Where aggregates are published widely, this design also suppresses small counts, because a count of one can single someone out.

Silver tables built from retained families, such as orders, keep their rows at erasure. Silver's erasure job clears their token, as step 5 does in the retained store, so revenue in gold doesn't move. A later part of this series covers what gold readers see when published numbers do change.

The rest of the post fills in each part: keeping the email out of capture, what the split writes, the two stages of erasure and their edge cases, marketing preferences, the stores to account for, and the date. The design leans on access controls throughout, so we've gathered them in one section near the end.

Keeping the email address out at source

Our post on multi-cloud retail computed the customer key in one key service, the only identity with sign and verify permission on the secret key in Cloud KMS. It also set out the choice between replicating raw addresses into the data platform and keying at source. This part takes the keying-at-source route and adds the mechanism for an estate fed by Datastream.

The email gets this treatment because it's the input to the join key. Other identifiers, such as names, phone numbers, delivery addresses, and free text, do land in the raw dataset. From there the split moves them to the identity side, or keeps them in a retained record whose obligation needs them.

The source application stores the customer key the key service returns. That post allows records to be stored first and keyed asynchronously. Neither the database nor the application holds the secret key, though the application can still request keys. Because the source database holds each email next to its customer key, its change tables and backups are on the ledger. The hardening section covers the controls that keep the key service's key to itself.

Two controls keep the email out of the lakehouse, one for each way data leaves the source.

  • Change data capture. Datastream supports Azure SQL Database only with the change tables CDC method, and lists it at service tier S3 and above (Google Cloud, n.d.aa). The @captured_column_list parameter limits which columns reach a change table, and Microsoft documents it for Azure SQL Database where only some columns should be tracked "for privacy or performance reasons" (Microsoft, 2025b). Primary key columns, or the columns of the unique index named with @index_name, must be captured (Microsoft, 2026h), so the email mustn't be part of either.
  • Backfill. A Datastream backfill streams a snapshot of the existing data (Google Cloud, n.d.o). We don't rely on the capture instance's column list to govern it. The stream's API schema allows columns to be listed under its included and excluded objects (Google Cloud, n.d.v). We'd list the allowed columns under included objects and treat that list as the control for backfills. A column added later then stays out until someone adds it.

A table over 500 million rows can be backfilled only if it has a unique index with no nullable columns and every index column is in the stream (Google Cloud, n.d.aa). That's one more reason to keep the email out of such indexes.

When a stream starts, Datastream selects the latest capture instance, doesn't switch to one created afterwards, and skips a table whose selected instance is deleted (Google Cloud, n.d.aa). So the instance that excludes the email has to exist before the stream starts.

If CDC is already running with every column, you recreate the stream against the new instance, usually with a backfill, and only then drop the original instance. Dropping it removes its change table (Microsoft, 2026g), but backups taken before then still contain it. SQL Server allows two capture instances per table (Microsoft, 2026h), so adding a source column becomes a planned change. A later part covers it.

This design leaves Datastream's enable_ddl_support_for_ct label off. With the label on, Datastream creates and switches to a new capture instance when a column is added (Google Cloud, n.d.aa), and the documentation doesn't say which columns that instance captures. Without the label, columns added after the stream starts don't replicate, and that's what this design wants. Masking the column isn't a workaround either. Datastream doesn't support tables with masked columns, and where they are replicated, the values arrive unmasked (Google Cloud, n.d.aa). Datastream's own login on the source is a privileged account, and the hardening section covers it.

If you can't change the source application, one fallback is a BigQuery remote function that calls a batch endpoint of the key service from the split job. It comes at a cost. Because the email is captured, it stays in the change tables and the raw dataset for their full retention. The schema check allowlists that one column, and the split job keys it and doesn't carry it forward. The worked example in "Computing the date" assumes keying at source. Under the fallback, its change-table and raw chains would also be chains for the email itself. The hardening section covers the endpoint's controls.

The split job

Datastream writes to BigQuery in merge mode by default, which keeps no history of change events (Google Cloud, n.d.f). A lakehouse that replays from bronze needs that history, so the example estate uses append-only mode. There, an update becomes a single UPDATE-INSERT row, an update that changes the primary key becomes an UPDATE-DELETE and UPDATE-INSERT pair, and a delete becomes one DELETE row (Google Cloud, n.d.f). Datastream applies these through the Storage Write API over gRPC (Google Cloud, n.d.f). Rows written that way can be changed by UPDATE, DELETE, or MERGE, with no waiting period stated (Google Cloud, n.d.ac). The erasure steps below rely on that.

Datastream can write every table into a single dataset (Google Cloud, n.d.f). The example estate creates that raw dataset beforehand, because otherwise a new BigQuery dataset gives READER, WRITER, and OWNER access to project Viewers, Editors, and Owners, and OWNER access to its creator (Google Cloud, n.d.c). Its access list names only the split job, the erasure job, the raw cleanup job, and any named scanner. Datastream writes through its project-level service agent. Raw, bronze, the identity table, the suppression entries, and the other restricted stores all follow one isolation rule, set out in the hardening section.

Datastream applies partitioning only when it creates a table, and you configure it in the stream (Google Cloud, n.d.r). So ingestion-time partitioning goes in before the stream first runs. Expiration alone doesn't give a date: an expired partition is removed by a background process that "can take several days to complete", and that deletion isn't recorded in audit logs (Google Cloud, n.d.p). A deleted partition's data is then kept under time travel and fail-safe (Google Cloud, n.d.p).

So the erasure job deletes the person's raw rows itself, and a scheduled cleanup job deletes everything else older than the raw retention period. The cleanup job skips rows that the split is holding back or that are waiting for a parent. A hold older than a stated threshold pages someone.

Expiration stays as a backstop, set longer than the raw retention period plus that hold threshold. The split records the raw rows it has finished with in its marks table, split_marks.marked_rows, with a reason: 'copied' for retained-family rows it has sent to the retained store, 'suppressed' for rows it dropped, 'orphaned' for erased-family children whose parent never arrived, and 'identity' for rows from identity sources, such as the customers table, once their fields are on the identity side. On its next daily run, the cleanup job removes marked rows, including the erasure job's 'erased' rows. It deletes a mark once its raw row is gone. Expiration set as a dataset default affects newly created tables (Google Cloud, n.d.ad), so it goes in before the stream creates its tables, or per table afterwards with ALTER TABLE ... SET OPTIONS (Google Cloud, n.d.p).

The split job is written in BigQuery SQL, runs from Airflow on options A and B, and writes three things. Airflow starts each split run one at a time, with a run timeout. Each run begins with a task that runs as the split account. It takes the next run number from a counter table, ops.split_runs, and records its start time there. Numbers therefore increase by start time, and a retry gets a new number. Without that, Airflow would run the retry under the same run. That task also waits until INFORMATION_SCHEMA.JOBS shows no earlier split job still pending or running, so two runs never overlap. A run that times out is cancelled when Airflow kills its task, and a job timeout set on the BigQuery script ends it even if the worker running the task dies.

  • The identity table maps a random token to the source key and the customer key. Clear identifiers from erased families are kept only where a named purpose needs them (for example a coarsened delivery postcode). Each has its own retention, which the erasure job enforces, and the purpose reads it through its own authorised view. Free text counts as an identifier. Erasure deletes the whole identity row, these identifiers included.
  • Bronze events carry the token in place of both the direct identifiers and the customer's source key. Record keys such as order and cart IDs stay, because deduplication and child lookups need them. Those keys can be joined back to the person through the source system while its records are kept. That's one more reason to treat the events as personal data. The events are clustered by token, and only the split job inserts them. A bronze retention job deletes erased-family events once they pass the longest of bronze's purpose periods, so bronze isn't kept indefinitely for people who never ask.
  • The retained store holds event families kept under an obligation, including any identifying fields the obligation requires. They don't go to bronze. While the mapping exists, they carry the token like bronze events. Silver reads them through a minimised view that returns the token as null for any token on the kept list or in the erasure hand-off table (the view is authorised on both datasets). That way, a retained record merged into silver after silver's erasure job has run doesn't bring the token back. A retention job deletes records when their obligation period ends.

Child records, such as cart lines, wishlist items, order lines, and shipments, get their token from the token-keyed parent event. A child that arrives before its parent waits in the raw dataset. One whose parent never arrives is counted. In an erased family, it's then marked 'orphaned' for the cleanup job. In a retained family it goes to the retained store unlinked.

When the split suppresses a parent in an erased family, it writes that parent's record key, with its family (cart or wishlist) and the request ID from the matching entry, to a table of suppressed record keys kept with the suppression entries. A source-key match suppresses every event of that record. A customer-key match covers events up to the erasure time, or up to the request while that time isn't fixed. The erasure job's own entries, from step 1, cover every event. A suppressed parent's children are then dropped and counted separately. Every suppression entry carries its request ID.

Listing 2 is the split for the wishlist family. It writes a wishlist event to bronze only when no suppression entry covers it. For each wishlist it suppresses, it records the key and marks those rows and their items for the cleanup job. The other items take their token from the wishlist's latest event in bronze. Each split run's statements share one transaction, so a failed run leaves nothing in bronze. The listing leaves out the 'orphaned' marks, the retained-store writes with their 'copied' marks, and the iden...