Data deletion in a lakehouse is a lie if you only run a DELETE statement. In regulated environments, "erasure" isn't a logical filter; it is a physical guarantee that sensitive information no longer exists within your storage layer. Most engineers assume that once a record is removed from the current view of a table, they have met their compliance obligations. They are wrong.

Why this matters: Regulations like GDPR and CCPA require the permanent removal of personal data upon request. In modern table formats like Apache Iceberg, data is versioned; a simple delete command leaves the sensitive information in historical snapshots and metadata files, potentially exposing you to non-compliance penalties even after the "delete" is complete.

The challenge is that Lakehouse architectures are designed for immutability. When you issue a DELETE or MERGE command, the engine creates new data files and marks old ones as "replaced" in the metadata. The data remains in your object storage (like S3 or ADLS) until specific maintenance procedures are executed.

Prerequisites

To follow this tutorial, you need:

  • An environment running Apache Iceberg 1.4.0+.
  • A compute engine that supports the Iceberg Spark procedures (Spark 3.3+).
  • Access to the underlying object storage (S3/GCS/ADLS) to verify file deletion.
  • Assume a table named user_data partitioned by region.

Dead sea from above with desert mountains and salt ponds
Photo by Matvey Logachev on Unsplash

Step 1: Execute the targeted deletion

First, you must remove the record from the active state of the table. In Iceberg, this is typically handled via a DELETE FROM statement. This command creates a new snapshot where the specified rows are excluded.

-- Using Spark SQL to perform the logical delete DELETE FROM prod.db.user_data WHERE user_id = 'user_abc_123';

At this stage, the record is gone from all standard SELECT queries. However, the data file containing user_abc_123 still exists in your storage bucket. If you were to query the table using AS OF VERSION AS OF <snapshot_id>, the record would reappear.

Step 2: Expire snapshots to revoke temporal access

Iceberg keeps a history of snapshots to allow for time travel and rollback. To fulfill a "right to be forgotten" request, you must remove the snapshots that contain the sensitive data.

The expire_snapshots procedure removes the metadata entries for old snapshots and identifies the associated data files as candidates for deletion.

-- Expire snapshots older than a specific timestamp -- This makes historical versions of the record unreachable CALL catalog.system.expire_snapshots( table => 'prod.db.user_data', older_than => TIMESTAMP '2023-10-27 10:00:00.000' );

Check the Iceberg documentation for expire_snapshots to understand how this interacts with your write.metadata.delete-after-commit.enabled settings. Note that this does not immediately delete the files from your cloud provider; it only tells the Iceberg catalog to stop tracking those files.

Step 3: Remove orphaned files from object storage

Even after expiring snapshots, the physical Parquet or Avro files may persist in your cloud storage. These are known as "orphan files." If these files contain the sensitive PII, you have not actually deleted the data.

You must run the remove_orphan_files procedure to identify and physically delete files that are not referenced by any current snapshot.

-- Physically delete files not referenced by any metadata -- Use a retention interval long enough to avoid deleting files  -- from concurrent jobs (e.g., 24 hours is standard) CALL catalog.system.remove_orphan_files( table => 'prod.db.user_data', older_than => TIMESTAMP '2023-10-26 10:00:00.000' );

Step 4: Verify the erasure

Compliance is not based on your code; it is based on auditability. You can verify that the data has been physically removed by attempting to list the objects in the underlying storage path that match the pattern of your deleted data.

If you are using AWS S3, you can use the CLI to ensure the specific file paths associated with the deleted snapshots are gone:

# Hypothetical check for files in the prefix aws s3 ls s3://my-data-bucket/user_data/data/ | grep user_abc_123 # Expected result: No output

Common pitfalls

Aggressive retention settings: Do not set your expire_snapshots interval too short. If a long-running batch job is mid-execution when you expire a snapshot, that job may fail because the files it was reading disappeared. Always ensure your "older_than" timestamp accounts for your longest-running job.

Hidden PII in manifests: Remember that Iceberg metadata files can contain column-level statistics (min/max values). If you delete a user but their user_id was the min/max in a partition, that value might persist in the metadata file even after the data file is gone. In extreme regulatory cases, you may need to perform a full rewrite_data_files with rewrite-all enabled to ensure that every trace of the data is purged from metadata stats.

The "Time Travel" trap: If you expose time-travel capabilities to your BI tools (e.g., via VERSION AS OF in Trino or Spark), users can bypass your deletion. Ensure that your access control layer (like Apache Ranger) prevents users from querying historical snapshots if those snapshots have not yet been purged.

Conclusion

Right to erasure in a lakehouse is a multi-stage lifecycle process, not a single command. You must delete the record, expire the metadata, and purge the orphan files. If you skip any of these, you are simply hiding the data, not destroying it.

Try it: Run SELECT count(*) FROM prod.db.user_data.snapshots before and after your next maintenance cycle to observe how many snapshots you are actually keeping, and verify if your storage costs decrease after running remove_orphan_files.

Cover photo by Rizky Pangestu on Unsplash.

This article was drafted with AI assistance and is intended for technical review and fact-checking before publication.