Quick version: Change Data Capture (CDC) sends only what changed in a database to another system, so you don't have to copy the whole table over and over.
Imagine ordering something online. You place an order, change your delivery address, and later check whether it has shipped. Each action updates information in a database.
Now think about the company's sales dashboard. How does it get those updates? It could copy the whole orders table every few minutes, but that seems like a lot of work when only a few orders have changed. Change Data Capture, or CDC, offers another way.
🔍 What Is Change Data Capture?
CDC is a way to identify changes in a database and send them to another system. The main changes are INSERT (adding a row), UPDATE (changing a row), and DELETE (removing a row).
Say order #2041 goes from 'pending' to 'shipped.' Instead of loading every order again, a CDC system can capture that update so a second system knows what happened. This is useful when a business keeps its everyday transactions in one database but runs reports somewhere else.
🕑 First, What Is a Batch Job?
A batch job is simply work that runs at a certain time or in groups, rather than processing each change the moment it happens. For example, a store might run a job every night at 2 a.m. to update its sales report.
But 'batch' tells us when the job runs, not how much data it copies. A batch job could use any of these methods:
Full refresh: Copy the entire table again. If the orders table has one million rows, the job reads all one million, even if only 100 changed.
Cursor-based incremental sync: Save a marker, often an
updated_attimestamp or increasing ID, and fetch rows newer than the last marker. If 100 rows were updated, the job may only read those 100.CDC: Read recorded database changes, usually from a transaction log or change stream. This can capture inserts, updates, and deletes, including changes that a simple timestamp query might miss.
So batch jobs and CDC are not opposites. A CDC tool can also run on a schedule. And people sometimes use 'incremental' as a broad term for any method that avoids copying everything. Here, 'cursor-based' specifically means checking rows against a saved marker.
Figure 1. Different ways of keeping data up to date.
❓ Why Does CDC Matter?
For small tables, a nightly full refresh might be perfectly fine. It is simple, and the report may not need to be up to the minute. As data grows, though, reading and moving unchanged records again and again can waste time and put extra work on the source database.
CDC helps keep dashboards, search results, and other systems fresher without repeatedly copying everything. It's especially helpful for data that changes often, such as orders or product inventory. It still isn't guaranteed to be instant, since the connector and destination can take time to process events.
⚙️ How Does Log-Based CDC Work?
Many databases keep a log of transactions. PostgreSQL uses a Write-Ahead Log (WAL), while MySQL has a binary log, or binlog. With the right setup, a CDC connector can read changes from these logs.
For example, when a worker changes order #2041 from 'pending' to 'shipped,' the database writes the update and records it in the log. A CDC connector reads the change and passes it to a destination, such as an analytics system, which can update the matching order.
Figure 2. A basic log-based CDC pipeline.
Often, a CDC tool begins with an initial snapshot to copy the rows that already exist. After that, it follows new changes. It also saves its position so it knows where to continue if the connection stops.
🧩 Three Common Ways to Capture Changes
1. Query-Based or Cursor-Based Sync
A script runs a query like 'show me rows updated since yesterday.' It's easy to build if your table has a reliable updated_at column. Think of a small club website updating its member list once a day. The catch is that hard deletes won't show up unless you track them separately, and several updates to one row may look like just one final state.
2. Trigger-Based CDC
A trigger is a rule inside the database that automatically runs when a row is inserted, updated, or deleted. For example, a school registration app could create a trigger that writes 'student 52 dropped course A' into a separate history table when an enrollment is removed. Another job can then read that history and update other systems.
Triggers are useful when you want a clear record of specific actions, especially in a smaller app. But every trigger adds work when data changes, and someone has to maintain the rules as tables change.
3. Log-Based CDC
This approach reads the database's own change records rather than checking rows over and over. A busy online store with thousands of orders might use it to keep its analytics system updated throughout the day. It can capture deletes and successive updates, but it needs the right permissions, log settings, and monitoring.
At a glance
| Method | How it works | Good for | Watch out for |
|---|---|---|---|
| Query/cursor-based | Query rows changed since a saved marker | Small apps with a reliable updated_at column | Hard deletes and repeated updates to one row |
| Trigger-based | Database rules write changes to a history table | A clear record of specific actions in smaller apps | Extra work on every change, plus rule upkeep |
| Log-based | Read the database's own transaction log | Busy systems that need deletes and successive updates | Permissions, log settings, and monitoring |
🛠️ What Tools Can Help?
You don't have to build a CDC pipeline from scratch. Debezium is an open-source tool for capturing database changes, and AWS Database Migration Service supports loading existing data followed by ongoing changes.
OLake Go is another example. For supported sources, it can read database changes and move data into Apache Iceberg tables or write plain Parquet files into object storage, such as Amazon S3. Iceberg helps manage tables in a data lake, while Parquet is a file format often used for analytics. Depending on the setup, OLake can do this without a separate Kafka pipeline.
⚠️ What Can Go Wrong?
CDC still needs maintenance. A connector might fall behind, logs might be removed before it catches up, or a restart could send an event more than once. Changing a column name or type can also cause trouble for the destination. Teams need monitoring and a plan for recovering safely.
💭 Final Thoughts
CDC sounds complicated at first, but the idea is pretty practical. Databases change all the time, and other systems need to keep up. Sometimes a full refresh is enough. Other times, a cursor-based job or CDC makes more sense. Understanding the difference helps you choose a solution without doing more work than you need to.
