This post is co-written with Aderant.

Aderant, a global provider of business management software for the legal industry, modernized the data pipeline powering its Stridyn analytics tool using Apache Iceberg, AWS Lambda, and AWS Glue. The new architecture delivers near real-time data availability for 700 client environments across 266 source tables. It replaces a batch-oriented pipeline that took 15 minutes per processing cycle with an event-driven system that processes change data capture (CDC) events in single-digit seconds.

In this post, we share how Aderant redesigned its multilayer data pipeline using a medallion architecture (bronze, silver, and gold layers) built on Apache Iceberg tables. This approach achieved 10–26x faster query performance and a 77 percent cost reduction compared to alternative approaches. We include architecture details, performance benchmarks, and the decision framework so that analytics teams managing multi-tenant SaaS tools can apply the same pattern.

The challenge: Batch processing limiting analytics freshness

Aderant’s Stridyn analytics application serves 700 client environments running Expert Sierra, its cloud-based legal practice management solution. The tool processes data from 266 source tables in Microsoft SQL Server databases, delivering financial reports, billing analytics, and operational dashboards to law firms globally.

The existing pipeline faced several challenges:

  1. Slow processing times. A full data processing cycle took approximately 15 minutes, with batch job overhead adding latency before data transformation could begin.
  2. Limited real-time data availability. Clients experienced delays between operational changes and analytics visibility, impacting time-sensitive billing and matter management decisions.
  3. Performance issues with complex queries. Reports involving window functions, joins across multiple fact tables, and common table expressions (CTEs) suffered from slow response times when querying external tables through Amazon Redshift Spectrum.
  4. Scalability constraints. The architecture struggled to handle custom data, schema changes, and a growing volume of CDC events that already reached 380 million per month across all tenants.

The catalyst for modernization came from Aderant’s commitment to delivering near real-time data availability, making sure data is accessible within minutes from source to target without changing the end-user experience.

The solution: Event-driven medallion architecture with Apache Iceberg

Aderant built a new data pipeline using a medallion architecture with three distinct layers (bronze, silver, and gold), all built on Apache Iceberg tables stored in Amazon Simple Storage Service (Amazon S3). The architecture uses AWS Lambda for event-driven CDC processing, AWS Glue for transformations, and Amazon Redshift for the final analytics serving layer.

How it works: The lifecycle of a data change

The following steps describe what happens when a record changes in a client’s source database.

  1. CDC capture and delivery. AWS Database Migration Service (AWS DMS) captures changes from Microsoft SQL Server databases and writes Parquet files to an S3 landing zone, organized by table and tenant.
  2. Event-driven ingestion (raw to bronze). An S3 event notification triggers an Amazon Simple Queue Service (Amazon SQS) message, which invokes a Lambda function that processes the CDC payload and writes to bronze-layer Iceberg tables in single-digit seconds per event.
  3. Bronze-to-silver transformation. AWS Glue jobs read incremental data from bronze tables using Amazon DynamoDB-based watermarking, apply normalization and deduplication through Iceberg MERGE operations, and produce current-state silver tables with ACID adherence.
  4. Silver-to-gold aggregation. The gold layer uses two approaches based on table complexity: incremental AWS Glue jobs for 77 standard gold tables, and materialized views with Iceberg snapshot comparison for 34 tables with complex query patterns that cannot be easily incrementalized.
  5. Gold-to-Amazon Redshift delivery. Delta changesets from materialized view snapshots are written to native Amazon Redshift tables, where columnar storage, zone maps, and sort keys deliver sub-100-millisecond query performance for end-user dashboards.

Why Apache Iceberg

Aderant chose Apache Iceberg as the table format across all pipeline layers for three reasons.

ACID transactions for concurrent operations. With 700 tenants generating CDC events simultaneously, Iceberg’s snapshot isolation helps prevent conflicts when multiple Lambda functions write to the same tables concurrently.

Time travel and snapshot comparison. For the 34 complex gold tables, Iceberg’s snapshot metadata identifies deltas without manual watermarking. The system compares consecutive snapshots to determine exactly which rows changed, then writes only those deltas to Amazon Redshift.

Schema evolution without pipeline disruption. As Aderant onboards new data sources and clients customize their schemas, Iceberg handles column additions and type changes without requiring full table rebuilds.

Why AWS Lambda for CDC processing

After evaluating three architectural approaches, the team chose Lambda based on cost, latency, and operational simplicity. The Docker container image deployment model (using Amazon Elastic Container Registry) accommodates the large library dependencies required for Iceberg operations, including PyIceberg and pandas. The pay-per-invocation model also aligns naturally with CDC workloads, where event volume fluctuates throughout the day, scaling to zero during quiet periods and handling 10x seasonal spikes without pre-provisioned capacity.

Architecture

The following diagram illustrates the end-to-end data flow from source databases through the medallion layers to the analytics serving layer.

Medallion data pipeline from source databases through the bronze, silver, and gold Iceberg layers to Amazon Redshift

Figure 1: End-to-end data flow through the medallion architecture, from source databases to the Amazon Redshift serving layer

The pipeline begins at the source layer, where AWS DMS replicates CDC events from Microsoft SQL Server databases into an Amazon S3 landing zone as Parquet files. When new files arrive, Amazon S3 Event Notifications publish messages to an Amazon SQS queue. The queue serves as a buffer to handle burst traffic and provides dead-letter queue capabilities for failed events.

AWS Lambda functions, deployed as Docker containers through Amazon Elastic Container Registry (Amazon ECR), consume messages from the SQS queue and write to bronze-layer Iceberg tables in Amazon S3. These Lambda functions use the PyIceberg library to perform ACID-aligned writes, and Amazon DynamoDB stores watermark state to track processing progress and help prevent duplicate writes. The AWS Glue Data Catalog registers Iceberg table metadata across the bronze, silver, and gold layers.

Amazon EventBridge triggers the next stage on a 5-minute schedule, invoking a decision Lambda function that checks whether bronze-to-silver AWS Glue jobs should run based on data availability. When new data is present, AWS Glue 5.0 jobs (running Apache Spark 3.5 on Python 3.11) read incrementally from bronze tables. These jobs apply MERGE operations for deduplication and normalization, then write clean records to silver-layer Iceberg tables. DynamoDB watermarks make sure each AWS Glue job processes only new data since its last successful run.

The silver-to-gold transition follows two paths. For the 77 standard gold tables, Amazon EventBridge detects the completion of bronze-to-silver jobs and triggers a second decision Lambda. That Lambda launches incremental AWS Glue jobs that write directly to Amazon Redshift native tables through the Amazon Redshift Data API. For the 34 complex gold tables, AWS Glue materialized views perform the transformation in SQL. A separate AWS Glue job compares Iceberg snapshots to extract only the changed rows and writes those delta changesets to Amazon Redshift native tables.

Amazon Redshift Serverless serves as the final analytics layer, where native tables with columnar storage, zone maps, and sort keys deliver sub-100-millisecond query latency. A GraphQL Data API layer sits between Amazon Redshift and the end-user dashboards, serving 42 pre-built reports including financial statements, billing analytics, and firm overview dashboards.

Amazon CloudWatch provides end-to-end observability across pipeline components, with custom metrics tracking processing latency, error rates, and throughput per tenant. AWS Identity and Access Management (IAM) enforces least-privilege access, and S3 data is encrypted using AWS Key Management Service (AWS KMS) with server-side encryption. The entire infrastructure is defined in Terraform and deployed through GitHub Actions continuous integration and continuous delivery (CI/CD) pipelines.

Materialized views: Solving the complex query problem

A notable aspect of the architecture is how Aderant handles the 34 gold tables that involve complex SQL patterns: LAG/window functions, joins across multiple fact tables, UNIONs with aggregations, and CTEs.

For these tables, traditional incremental processing would require redesigning each query’s logic. Instead, the team uses AWS Glue materialized views as an intermediate step:

  1. Create materialized views using standard Spark SQL (CREATE MATERIALIZED VIEW AS SELECT ...) against silver-layer Iceberg tables.
  2. Automatic change tracking through Apache Iceberg, which monitors source table changes without manual watermarking.
  3. Scheduled refresh every 2–5 minutes through AWS Glue jobs (bypassing the 1-hour minimum for automatic refresh).
  4. Delta extraction by comparing Iceberg snapshots between refreshes.
  5. Write to Amazon Redshift only the changed rows as native table updates.

This approach alleviates the need to rewrite complex analytical queries while still achieving incremental processing. The materialized view handles the computational complexity, and Iceberg’s snapshot comparison identifies what changed.

Quantifiable results

The architecture improvements delivered measurable gains across three dimensions: query performance, pipeline latency, and infrastructure cost.

Query performance

The team benchmarked 16 analytical queries, ranging from simple aggregations to complex pivots over 9 million rows, comparing Amazon Redshift native tables against external tables queried through Amazon Redshift Spectrum. Across all query types, native tables delivered 10–26x faster response times, consistently returning results in under 100 milliseconds regardless of dataset size. The largest improvement came on smaller, frequently accessed datasets (25.8x speedup), while complex pivot queries with window functions still achieved a 13.7x improvement. These benchmarks were conducted on Amazon Redshift Serverless at 16 RPU, meaning provisioned clusters with dedicated compute and persistent caching would perform even better.

The team also validated Amazon S3 Express One Zone for workloads that remain on S3-based Iceberg tables without Amazon Redshift. It reduced query latency by 31–78 percent compared to Amazon S3 Standard, with the greatest gains on complex queries involving multiple table scans.

Pipeline latency

The largest operational improvement is end-to-end processing time. The previous pipeline required approximately 15 minutes per full processing cycle. The new event-driven ingestion layer processes CDC events without batch job overhead. CDC events now flow from source database to bronze-layer Iceberg tables in single-digit seconds. Gold layer refreshes complete in under 5 minutes for initial loads and under 1 minute for incremental updates. The system sustains throughput of 186,000+ events per second across all 266 tables and 700 tenants.

Infrastructure cost

By choosing Lambda-based event processing over streaming alternatives, Aderant achieved a 77 percent reduction in pipeline compute costs. The serverless, pay-per-invocation model scales to zero during quiet periods and handles 10x seasonal spikes without pre-provisioned capacity, keeping costs proportional to actual data volume rather than reserved infrastructure.

In production: Multi-tenant operations

The architecture runs in production today, processing data for 700 client environments across 266 source tables. Three aspects of the implementation stand out.

Tenant isolation with shared infrastructure. Each tenant’s CDC events flow through the same Lambda functions and AWS Glue jobs, but Iceberg’s partition-by-tenant structure maintains data isolation. Configuration files drive per-tenant behavior without code changes.

Graceful handling of complex reports. The Expert Bill Report, one of Aderant’s most complex analytical outputs involving multiple dimension tables and fact aggregations, processes 900,000 records in approximately 5 minutes for initial load, with subsequent incremental runs handling only changed records in seconds.

Self-healing watermark management. DynamoDB-based watermarks track the last processed timestamp per table per tenant. If a Lambda function fails, the watermark remains unchanged, and the next invocation reprocesses from the correct point, providing exactly-once semantics without complex distributed transaction logic.

Looking ahead

Aderant plans to continue expanding the tool’s capabilities.

Streaming analytics for high-volume tenants. The Lambda-based architecture supports a hybrid model where tenants with the highest event volumes can migrate to Amazon Kinesis streaming for sub-second latency, while standard tenants remain on the cost-effective Lambda path.

AI-powered analytics. Integration with Amazon Bedrock for natural language querying of the gold layer, so law firm users can ask questions about their billing and matter data conversationally.

Automated data quality. Expanding the validation framework to continuously monitor data consistency between pipeline layers, with automated alerting when discrepancies are detected.

Conclusion

Aderant’s journey from batch-oriented data processing to an event-driven, Iceberg-based pipeline demonstrates what is possible when you combine modern open table formats with serverless compute. The new architecture processes CDC events in single-digit seconds instead of 15-minute batch cycles. It delivers 10–26x faster query performance through native Amazon Redshift tables and reduces infrastructure costs by 77 percent compared to streaming alternatives.

The key insight is that not every table needs the same processing strategy. Aderant uses incremental AWS Glue jobs for straightforward tables and materialized views with Iceberg snapshot comparison for complex analytical patterns. This approach delivers near real-time freshness across all 111 gold tables without rewriting complex SQL logic.

For analytics teams managing multi-tenant SaaS tools, Aderant’s experience shows the value of Apache Iceberg. Its ACID alignment, time-travel capabilities, and schema evolution support make it a strong foundation for production data pipelines at scale.

To get started with Apache Iceberg on AWS, see Using Apache Iceberg tables in AWS Glue. For more on the services used in this architecture, visit AWS Lambda and AWS Glue. For a reference architecture with sample code, see the AWS Samples repository on GitHub.


About the authors

Sham Birbahadur

Sham Birbahadur

Sham is the Director of Data Engineering at Aderant. As a Legal Technology leader with 30 years of experience spanning software development, cloud operations, data architecture, and AI-driven transformation. Sham has led global teams across development, infrastructure, and cloud operations, driving large-scale modernization initiatives and cloud-first strategies. With deep expertise in AWS and Azure platforms, his work focuses on data migration, data transformation, automation, scalability, and the practical adoption of AI to deliver innovative business solutions.

Nikolai Kolesnikov

Nikolai Kolesnikov

Nikolai is a Principal Delivery Consultant at AWS Professional Services, where he served as the technical lead for the Stridyn Data Pipeline Enhancement engagement. Nikolai specializes in designing scalable data architectures using Apache Iceberg, AWS Glue, and serverless compute patterns.

JJ Johnston

JJ Johnston

JJ is a Sr. Analytics Solution Architect at AWS. JJ helps customers leverage their data efficiently and quickly in order to stay competitive and successful in their industries. They come from a background in Financial Services, AIML Startups, and Alexa’s Data Platform, working with and advising on a multitude of data patterns giving them a wealth of experience they use to help customers leverage data for their business needs.

Peter Chung

Peter Chung

Peter is a Senior Solutions Architect at AWS, based in New York. Peter helps software and internet companies across multiple industries scale, modernize, and optimize. Peter is the author of “AWS FinOps Simplified” and is an active member of the FinOps community.