Sampling Ad Exchange Events at Scale: What We Kept, What We Dropped, and Why

At Unity, our ad exchange logged every stage of an auction into Kafka. This included the bid request, bid responses, filters applied, auction decision, fill, impression, and click events. When I looked at the data volume, I found that we were processing around 10 GB per second.

The problem was that every downstream Spark job had to process this entire volume, even when it only needed a small part of the data.

I was the only data engineer in a virtual pod working on this project, along with two product managers, two senior software engineers, and an architect. I proposed sampling the stream and also suggested changes to the Spark clusters running the reports. Here is how we approached the problem and what we learned along the way.

Where was the money actually going?

The data was costing us money at multiple stages. Kafka had to retain and replicate it, raw files were written to object storage, Spark batch jobs processed the data for the DSP and internal exchange reports, and Druid indexed the results.

We were effectively paying to store and process the same large volume of data at different stages. Among these, the Spark reporting jobs were one of the biggest cost contributors.

It became clear that the most effective way to reduce the cost was to reduce the data volume at the source itself. If we could avoid writing unnecessary data, we could reduce the cost across the entire pipeline rather than optimising each component separately.

Most of the data was not being used

I mapped the events in the stream against the requirements of the reports that consumed them. One thing stood out immediately: around 99% of the data volume came from bid requests that were filtered out before reaching an auction.

At the same time, around 99% of the reporting use cases relied on post-filter events, such as bid responses, auction decisions, fills, impressions and clicks.

In simple terms, we were spending a significant amount of money processing data that most reports did not need.

Auction stage	Relative data volume	Required by reports
Bid requests filtered out before the auction	Highest	Rarely
Bid responses	Lower	Yes
Auction decisions	Lower	Yes
Fills and impressions	Lower still	Yes
Clicks and installs	Lowest	Yes

This finding shaped the rest of the project. Instead of trying to optimise the processing of the entire stream, we focused on reducing the volume of data that was not required by the reporting use cases.

Understanding the requirements before making changes

Before implementing sampling, I prepared a list of the event types and fields used by the reports. I reviewed these requirements with the report owners, including teams such as ProgOps, the DSP reporting team and internal BI.

Initially, I considered applying uniform sampling across the entire stream. However, after discussing the requirements with the report owners, it was clear that this would not work for every event type. Some fields needed to retain their full accuracy because business and operational decisions depended on them.

We therefore decided on the following approach:

Pre-filter events: Sample at 0.5%.
Post-filter events: Retain 100% of the data.

This allowed us to significantly reduce the volume of data while keeping all the information required by the reporting teams. We avoided compromising the accuracy of the metrics that mattered to the business.

Sampling at the auction level

One important design decision was how to perform the sampling.

Instead of sampling individual events independently, I used the auction ID as the sampling key. This ensured that all events belonging to a selected auction were retained together.

For example, if we sampled a bid response but dropped its corresponding request, it could create problems when joining different event types. Sampling at the auction level avoided this issue and preserved the relationships between related events.

For metrics that required counts from the sampled data, we applied the appropriate scaling factor of (1/p), where (p) was the sampling probability. Ratios could also remain comparable when both the numerator and denominator were calculated from the same sample.

This approach allowed us to reduce the data volume without breaking the joins and calculations required by the reporting jobs.

Implementing the changes

On the producer side, we created a separate Kafka topic for the sampled stream and updated the reporting jobs to consume data from this topic.

We also moved the sampled stream to the newer streaming ingestion path. This provided a tighter delivery SLO and reduced infrastructure costs. As a result, the stream volume dropped from approximately 10 GB per second to around 2 GB per second.

Once we had reduced the incoming data volume, I focused on the Spark clusters powering the DSP and exchange reports.

The existing clusters were running at roughly 50% CPU utilization and 70% memory utilization, even though we were paying for the full allocated worker capacity. This indicated that there was scope to improve how the resources were being used.

I evaluated three executor configurations: smaller executors, larger executors, and a balanced configuration in between. I experimented with the number of cores, memory allocation, and parallelism, and validated the configurations through regression runs.

The balanced configuration gave us the best results. CPU utilization increased by 41%, and we reduced the number of Spark worker nodes from 240 to 30, an 87.5% reduction.

We also found issues in the existing reports

During validation, we compared the sampled reports against the reports generated by the original pipeline. Not all discrepancies were caused by sampling.

While refactoring the jobs, we discovered that the BidWin and WinPrice metrics had been calculated incorrectly since 2019. The issue was caused by a join that resulted in row duplication and inflated the calculations.

The numbers had looked reasonable, so the issue had gone unnoticed. The cost of the existing pipeline also made it difficult to perform frequent comparisons against alternative calculations.

This was an important lesson for us. We could not assume that the old reports were necessarily correct just because they had been running for several years.

After this discovery, we started time-boxing discrepancy investigations and checking the existing calculation logic before assuming that the new pipeline was responsible.

Challenges during the project

There were a few challenges along the way.

Changing requirements: Midway through the project, additional dimensions were requested for a commercial deal. We applied the same principle we had agreed on earlier: if a dimension could not be sampled safely, we retained it at full fidelity.
Team reorganization: The team became smaller following a reorganization, which affected the available bandwidth.
Running two pipelines: During the migration, we had to run the old and new pipelines in parallel. This increased the infrastructure load, so we monitored resource usage carefully while validating the new implementation.

Results

The project delivered improvements across both data volume and infrastructure usage.

Reduced stream volume from approximately 10 GB/s to 2 GB/s.
Reduced Spark worker nodes from 240 to 30, an 87.5% reduction.
Increased Spark CPU utilization by 41% through executor tuning.
Preserved all fields required by the reporting use cases.
Identified and corrected existing issues in the BidWin and WinPrice calculations.

Reducing the data volume also made it easier to extend the reporting pipeline. The next version of the buyer report could support additional fields without adding the same level of infrastructure overhead.

How we measured this

Stream volume. The 10 GB/s and 2 GB/s figures are the average bytes-in rate on the exchange Kafka topic, read from our Grafana dashboards before sampling (December 2022) and after the sampled topic went live (March 2023).
Spark workers. 240 was the configured worker count for the DSP Report cluster before sampling, sized to process the full-volume stream. 30 is its count after sampling and executor tuning. Both numbers come from the DAG configuration. The reduction happened in stages: Spark tuning ran before, during, and after the sampling rollout. By the time of my tuning experiments on February 4–5, 2023, the cluster was at 180 workers. I tested configurations at a low-volume hour and a peak hour, and validated each one with regression runs against the full-volume output before moving production to it.
CPU. The 41% figure compares the DSP Report cluster’s CPU utilization before and after executor tuning, from the cluster’s monitoring metrics.
Cost. GCP billing for the DSP Report transform job confirmed the drop. For March 9 to April 6, 2023, it was about 41% lower than the prior period.
Accuracy. Before switching any report over, we ran the sampled pipeline for two weeks alongside the full-volume one and compared the report outputs.

What I learned

The biggest takeaway for me was that we should first understand how data is being used before trying to optimize the infrastructure processing it.

In this case, the initial problem appeared to be the cost of running Spark jobs at a high data volume. However, reducing the data at the source had a much bigger impact because the savings carried through the entire pipeline.

I would also recommend talking to report owners before changing what data you keep. It is important to understand which events and fields can be sampled and which ones need to remain at full fidelity.

Finally, before adding more Spark capacity, it is worth checking whether the existing executor configuration is suitable for the workload. And when replacing an old pipeline, we should validate its outputs rather than assume that the existing calculations are correct.