Moving Big Data from AWS to GCP: Redshift, EMR and Glue

A worked migration of a Redshift, dbt, Glue, Kinesis and MWAA platform to Google Cloud: BigQuery on-demand versus slots with real cost math, the SQL that translates cleanly and still returns different results, Kinesis versus Pub/Sub semantics, and a parallel-run method that proves finance's numbers match.

Finance closes the books every month from dashboards that read a Redshift cluster. A dbt project builds those numbers overnight from Glue jobs, a Kinesis stream and a few dozen MWAA DAGs. Leadership has signed a Google Cloud commit, and the data platform is the biggest line on the AWS bill. The question is not which GCP service replaces Redshift. That answer takes a minute. The question is how you move it without a single revenue figure changing, and without a bill that is bigger than the one you left.

This post works through that move on one representative platform. It is an illustrative workload, not a customer, but every assumption is stated so you can swap in your own numbers. We make the decisions in the order they bite: the BigQuery pricing model, the storage billing model, the SQL that translates cleanly and still returns different answers, moving the bytes, the jobs and the stream, access rules, and the parallel run that proves the numbers match. Prices are list prices as of November 2026: AWS in us-east-1, Google Cloud in us-central1 (Iowa).

This guide covers big data. For EC2, EKS and GPUs, see moving compute from AWS to GCP. For Lambda, Fargate and EventBridge, see moving serverless from AWS to GCP. For S3, EBS and EFS, see moving storage from AWS to GCP. Going the other way? Read moving big data from GCP to AWS.

The platform we are moving

Compute on the warehouse is the anchor cost: 4 nodes x $3.26/hour x 730 hours is $9,519 a month, plus 28,000 GB of managed storage at $0.024/GB-month, another $672. If you hold reserved nodes, use your effective rate instead, and remember the reservation keeps billing after you leave. Plan the cutover around its end date if you can.

Where each piece lands

The mapping is the easy part, so we keep it short. What matters is the condition under which you would pick the other option:

Decision 1: on-demand or slots

BigQuery bills compute one of two ways. On-demand charges $6.25 per TiB of data read, with the first TiB each month free. Editions charge per slot-hour: $0.04 Standard, $0.06 Enterprise, $0.10 Enterprise Plus, pay as you go, with autoscaling in multiples of 50 slots and a one-minute minimum per scale-up. On-demand punishes scanning. Editions punish compute. The right answer depends on how many slot-hours each workload burns per TiB it reads, and you can't know that from Redshift.

Redshift's own scan statistics don't translate: it reads compressed blocks across slices, while BigQuery bills the uncompressed size of the columns a query touches. The only honest profile is one taken on BigQuery. Load a representative month, replay the dbt build and a sample of dashboard and ad hoc queries, then read the job metadata:

-- Slot-hours per TiB, by workload label, for the replay window SELECT (SELECT value FROM UNNEST(labels) WHERE key = 'workload') AS workload, SUM(total_bytes_billed) / POW(1024, 4) AS tib_billed, SUM(total_slot_ms) / 1000 / 3600 AS slot_hours, SAFE_DIVIDE(SUM(total_slot_ms) / 1000 / 3600, SUM(total_bytes_billed) / POW(1024, 4)) AS slot_hours_per_tib FROM `region-us-central1`.INFORMATION_SCHEMA.JOBS WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY) AND job_type = 'QUERY' AND state = 'DONE' GROUP BY workload; 

Label jobs before you replay (dbt supports query_comment and job labels; BI tools can set labels on their service account's jobs). Here is the profile for our representative workload, with one more number you have to estimate: the autoscale overhead, the ratio of slot-hours billed to slot-hours used. Long, steady jobs keep the autoscaler busy and waste little. Bursty small queries pay the one-minute minimum and the 50-slot rounding over and over. Measure yours from INFORMATION_SCHEMA.RESERVATIONS_TIMELINE once a test reservation exists.

The break-even is simple. At $6.25 per TiB against $0.06 per slot-hour, editions win when a workload burns fewer than 104 billed slot-hours per TiB (used slot-hours times overhead). dbt sits at 45.5, dashboards at 50, so both belong on a reservation. Ad hoc lands at 150: analysts write window functions over joins of small tables, which is compute-heavy and scan-light, and the bursts waste slots. Ad hoc stays on-demand.

Reservations are assigned per project, folder or organization, so the decision becomes a project layout: a transform project and a bi project assigned to an Enterprise reservation with a baseline of zero and a max of 1,000 slots, and an analytics project left on on-demand. Total compute: about $3,000 a month, against $5,525 if everything ran on-demand and $3,442 if everything ran on Enterprise.

Why not Standard edition, or a commitment?

Standard at $0.04 looks like a free 33% cut on the dbt reservation. It isn't, for this platform. Standard edition has no fine-grained security controls, so no row-level or column-level security, and it caps a reservation at 1,600 slots. Our Lake Formation row filters become BigQuery row access policies, so the projects that read governed tables need Enterprise. A team with no fine-grained access rules should look hard at Standard.

Enterprise slot commitments cost $0.048 per slot-hour for one year and $0.036 for three, but you pay for every hour of the term. 100 committed slots is $3,504 a month, double our whole dbt bill, because the build uses most of its slots for a couple of hours a night. A one-year commitment only beats pay as you go when the committed slots are busy more than about 62% of all hours (0.048 / (0.06 x 1.3)). Look at the timeline after a month of real traffic, and commit to the floor you never drop below, not the peak.

On-demand without a cap is how a first week goes wrong. Set a per-user daily custom quota on the on-demand project before analysts get access, and set maximum_bytes_billed in dbt profiles and BI connections, so a missing partition filter fails instead of billing.

Decision 2: logical or physical storage billing

BigQuery bills storage per dataset either on logical (uncompressed) bytes or on physical (compressed) bytes. In us-central1, logical costs $0.023/GiB-month active and $0.016 long-term (partitions untouched for 90 days). Physical costs $0.04 and $0.02, but with one catch: time travel and fail-safe bytes are billed separately at the active physical rate, while on logical billing they are included.

Physical billing wins whenever compression beats the price ratio (1.74x for active, 1.25x for long-term) and tables are append-mostly. A dbt project full of materialized='table' models is the opposite: every nightly rebuild leaves the old table in time travel and then fail-safe for 14 days. Cutting the time travel window to its 2-day minimum still leaves 9 days of copies. Put churn-heavy models in their own dataset on logical billing, or make them incremental, and the decision flips back.

SQL that translates cleanly and still returns different numbers

The BigQuery batch SQL translator converts Redshift SQL in bulk and flags what it can't handle. The dangerous cases are the ones it doesn't flag, because the query runs on both engines and returns a plausible number. These are the ones we would grep for before the first parallel run:

The trailing-space row deserves a second look because it fails on joins, not filters. If a CRM export pads merchant codes, Redshift joins 'ACME ' to 'ACME' and BigQuery drops the row. Revenue by merchant falls by exactly the padded merchants' share, which looks like a data-loading gap rather than a semantic one.

The timestamp that moves revenue between days

Redshift TIMESTAMP has no time zone and ignores any zone in input strings. If the order service wrote New York wall-clock time into order_ts, Redshift has been storing local time all along. BigQuery TIMESTAMP is an absolute instant, and a value loaded without a zone is taken as UTC. Load the column as-is and BigQuery stores 9 p.m. New York time as 9 p.m. UTC, four or five hours before the real instant. Nothing breaks until something does the correct thing: a model that calls DATE(order_ts, 'America/New_York'), a BI tool that converts to the viewer's time zone, or new events from the stream that carry true UTC. Then orders placed between midnight and 4 a.m. Eastern (5 a.m. in winter) land on the previous business day, and history and new data mean different things in the same column. Monthly totals barely move. Daily revenue does, and month-end shifts by a few hours of sales.

-- Interpret the stored wall-clock value in its real zone, once, in staging SELECT order_id, TIMESTAMP(DATETIME(order_ts_raw), 'America/New_York') AS order_ts, DATE(TIMESTAMP(DATETIME(order_ts_raw), 'America/New_York'), 'America/New_York') AS business_date FROM staging.orders_raw; 

Then partition on business_date, not on the UTC date, so a finance query for one business day prunes to one partition instead of two.

DISTKEY and SORTKEY become partitioning and clustering

Don't translate the keys mechanically. A Redshift DISTKEY exists to co-locate join rows on the same slice. BigQuery has no slices to co-locate on, so the distribution key has no direct heir. Pick partitioning from the date filter almost every query uses, and up to four clustering columns from the next most selective filters. Two limits shape the choice: 10,000 partitions per table (daily partitions last 27 years, hourly ones about 14 months), and clustering pruning that the dry-run estimate can't see, so on-demand estimates on clustered tables are an upper bound.

-- dbt on Redshift {{ config(materialized='incremental', unique_key='order_id', dist='customer_id', sort='order_ts') }} -- dbt on BigQuery {{ config( materialized='incremental', incremental_strategy='insert_overwrite', partition_by={'field': 'business_date', 'data_type': 'date'}, cluster_by=['merchant_id', 'region'], require_partition_filter=true ) }} 

Note the strategy change. A merge incremental with only a unique_key scans the whole target table to find matches. On a 20 TiB fact table that is $125 per run on-demand, or a large slot bill on a reservation, every hour. insert_overwrite replaces only the partitions the new batch touches. Use merge only with incremental_predicates that bound the target scan.

Moving the bytes

The BigQuery Data Transfer Service has a Redshift connector: migration agents run UNLOAD to an S3 staging bucket, and the service loads the files into BigQuery with the schema. It is the right tool for the long tail of dimension and small fact tables. It is the wrong tool for the big ones, for four reasons in Google's own documentation:

• It stages CSV. Text unloads collapse NULL and empty string, and files containing an ASCII 0 character can't be loaded at all. The documented workaround is your own Parquet unload.
• 15 TB per load job per table. Exported text is larger than Redshift's compressed size, so a table Redshift reports at 6 TB can cross the line. Google asks you to contact Customer Care above 15 TB.
• Re-runs append. Running a transfer twice for the same table duplicates rows. Pair it with a truncate step, or use it once for history.
• S3 to BigQuery goes over the public internet, and the cluster's security group must allow Google's IP ranges on port 5439.

For the big fact tables we unload ourselves, month by month, to Parquet, copy with Storage Transfer Service, and load from Cloud Storage:

-- Redshift: one month of orders, Parquet keeps NULLs and types UNLOAD ('SELECT * FROM sales.orders WHERE order_ts >= ''2026-09-01'' AND order_ts < ''2026-10-01''') TO 's3://acme-unload/orders/2026-09/' IAM_ROLE default FORMAT AS PARQUET MAXFILESIZE 1 GB; # GCP: load the copied month into a staging table bq load --source_format=PARQUET \ staging.orders_raw 'gs://acme-unload/orders/2026-09/*' 

Time it before you promise a date. The lake plus the unloaded warehouse is about 220 TB here. At a sustained 5 Gbps, one TB takes about 27 minutes and the whole set about 98 hours; at 1 Gbps it is three weeks. Test throughput on a real 1 TB prefix first. On egress fees: AWS waives data transfer out to the internet for customers moving all their data off AWS, through a request to AWS Support, and eligible customers outside the EU get 180 days to finish. Get the approval before the first large copy, not after.

Glue and EMR: sort jobs by what they actually do

Glue has no single home on GCP, so don't port it job by job. Read each job and classify it. On our 46 Glue jobs and crawlers, the split looks like this:

Half the Glue estate disappears into the dbt project, which is the cheapest outcome: one less runtime to operate. The nine real Spark jobs and the nightly EMR job move to serverless Spark with s3:// swapped for gs:// and EMRFS or S3A settings deleted, since the Cloud Storage connector is built in. Glue job bookmarks have no equivalent; replace them with a watermark table the job reads at start and writes on success, which is also easier to reset during the parallel run.

Kinesis to Pub/Sub: ordering, replay and the bill

Producers port in a day. Consumers are where the semantics differ:

Two consumer changes are mandatory. Any code that deduplicates on Kinesis sequence numbers needs a producer-assigned event ID in a message attribute, because Pub/Sub redelivery will hand it the same event twice and no sequence number to recognise it by. And any consumer that replays more than 31 days (a backfill job reading 90 days off the stream, say) needs a different source: a Cloud Storage subscription archiving raw events, or Kafka.

Then the bill, which surprises people in the other direction. Our stream carries 2,500 events/s x 1.2 KB, about 7.17 TiB a month. Kinesis: 8 shards x 730 hours x $0.015 is $87.60, plus 6.57 billion PUT payload units at $0.014 per million, $92, so $180 a month before Firehose. Pub/Sub bills throughput at $40 per TiB published and $40 per TiB per subscription delivered, and $50 per TiB for BigQuery subscriptions. Publish ($287), one pull subscription for the fraud service ($287) and a BigQuery subscription replacing Firehose ($359) total $932. At steady, high throughput, Pub/Sub costs more than provisioned shards. You are paying to stop resharding and to drop the Firehose and its Glue streaming job. If that trade isn't worth $750 a month to you, price Managed Service for Apache Kafka for the same throughput before you choose.

resource "google_pubsub_topic" "orders" { name = "orders" message_retention_duration = "604800s" # 7 days on the topic } resource "google_pubsub_subscription" "fraud" { name = "orders-fraud" topic = google_pubsub_topic.orders.id enable_message_ordering = true # producer sets ordering_key = merchant_id ack_deadline_seconds = 60 retain_acked_messages = true # needed to seek back after a bad deploy message_retention_duration = "604800s" } resource "google_pubsub_subscription" "orders_to_bq" { name = "orders-to-bigquery" topic = google_pubsub_topic.orders.id bigquery_config { table = "acme-data.raw.orders_events" use_table_schema = true } } 

MWAA to Composer, and the switch that moves readers

Composer 3 offers both Airflow 2 and Airflow 3 builds. Land the 38 DAGs on the same Airflow major version MWAA runs today, and upgrade Airflow as a separate change after cutover, so an upgrade regression can't be confused with a migration one. Every S3, Redshift, Glue, EMR and Athena operator becomes its GCS, BigQuery, Dataproc or Dataflow counterpart from the Google provider package, and stored AWS keys become the environment's service account.

During the parallel run both orchestrators run. MWAA keeps feeding Redshift and the live dashboards; Composer runs the same pipelines into BigQuery plus a reconciliation DAG. Each reader team switches by repointing its BI connection, never by changing pipelines, so rolling a team back is a connection change too.

Rebuilding Lake Formation grants

There is no converter from Lake Formation to BigQuery. LF-tags on columns map to policy tags in a Dataplex taxonomy, and data filters on rows map to row access policies. Export every grant, rebuild it in Terraform, and treat the list as a security review: grants nobody can explain are the ones to drop.

CREATE ROW ACCESS POLICY apac_finance ON finance.revenue_daily GRANT TO ('group:finance-apac@acme.example') FILTER USING (region = 'APAC'); 

Test it as the restricted group, not as yourself: a user with no matching policy sees zero rows rather than an error, which looks exactly like an empty table.

Proving the numbers: the parallel run

Row counts alone prove almost nothing: every silent difference above keeps row counts identical. We compare in three layers, each run daily by the reconciliation DAG on both engines and written to one BigQuery table.

1. Shape, per table per business date: row count, distinct key count, min and max timestamp, NULL count and empty-string count per nullable text column.
2. Content fingerprint, per table per business date: the sum of a hash over a canonical form of the key and the money columns.
3. Metrics, the 40 numbers finance actually signs off: revenue, refunds, net revenue, orders, and each ratio, by day, region and product line. Money must match to the cent. Ratios must match to the precision the dashboard shows.

The fingerprint only works if both engines build the same string. Two traps: BigQuery's CAST(NUMERIC AS STRING) drops trailing zeros (12.5) where Redshift keeps the scale (12.50), so fingerprint money as integer cents. And sum a short hex prefix of the hash, so the total can't overflow a 64-bit integer.

-- Redshift SELECT business_date, COUNT(*) AS n, SUM(STRTOL(LEFT(MD5(order_id::VARCHAR || '|' || (amount * 100)::BIGINT::VARCHAR), 7), 16)) AS fp FROM sales.orders GROUP BY 1; -- BigQuery SELECT business_date, COUNT(*) AS n, SUM(CAST(CONCAT('0x', SUBSTR(TO_HEX(MD5(CONCAT( CAST(order_id AS STRING), '|', CAST(CAST(amount * 100 AS INT64) AS STRING)))), 1, 7)) AS INT64)) AS fp FROM sales.orders GROUP BY 1; 

The pattern of the diff tells you which bug you have, which is the point of running three layers instead of one:

How long to run both

Duration is set by the calendar, not by a number of days. The run has to cover every cadence the numbers are produced on: the nightly build, the weekly reports, and at least one full month-end close, including the adjustments finance posts in the first business days of the next month, because that is when late data and manual corrections arrive. Then require ten consecutive business days with zero unexplained diffs on the 40 signed-off metrics. An explained diff (a bug fixed on the BigQuery side, or a Redshift bug finance accepts) resets nothing but goes in a log finance signs.

Price the overlap honestly. Every month of parallel running costs the full Redshift bill, about $10,200 here, on top of the new platform. That pressure is real, and it is still cheaper than restating a month of revenue. After the last team switches, pause the Redshift cluster instead of deleting it: compute stops billing, storage keeps the rollback option open until the next close passes cleanly, then take a final snapshot and delete.

The monthly bill, before and after

Glue, EMR, MWAA, Composer and serverless Spark are left out because their cost depends on job shapes we haven't fixed here; model them from your own job history. Two things in this table are worth more than the totals. Storage costs more on GCP, not less, because BigQuery bills uncompressed bytes unless you choose physical billing carefully. And the biggest compute saving doesn't come from BigQuery being cheaper per unit. It comes from not paying for a cluster sized for the nightly build to sit idle the other 20 hours.

The order we would do it in

1. Inventory and label. Every schema, job, DAG, stream consumer and dashboard, with owner and the metrics finance signs.
2. Land a month and profile it. Replay labelled workloads, measure slot-hours per TiB, choose the project layout and pricing model from data.
3. Grep for the silent six. Integer division, integer AVG, week truncation, padded keys, naive timestamps, text unloads.
4. Copy history in Parquet, then feed both platforms daily.
5. Rebuild grants and test them as the restricted groups.
6. Parallel run through a month-end close plus ten clean business days.
7. Switch readers team by team, pause Redshift, delete after the next clean close.

Common questions

Is BigQuery cheaper than Redshift?

For bursty workloads, usually, because you stop paying for an idle cluster. In our worked example warehouse compute drops from $9,519 to about $3,000 a month. Storage can cost more, and on-demand without quotas can cost far more. Profile slot-hours per TiB on BigQuery before deciding.

Should we use BigQuery on-demand or editions?

Per workload, not per company. At list price in us-central1, Enterprise beats on-demand when a workload burns fewer than about 104 billed slot-hours per TiB it reads. Batch transforms usually do; ad hoc analysis often doesn't. Split them into separate projects.

How do I migrate data from Redshift to BigQuery?

Use the BigQuery Data Transfer Service Redshift connector for small and medium tables. For large fact tables, unload to Parquet month by month, copy with Storage Transfer Service and load from Cloud Storage, which avoids the CSV staging, the 15 TB per-load limit and duplicate rows on re-runs.

Why do my numbers differ after moving from Redshift?

The usual causes are integer division and integer AVG (BigQuery returns FLOAT64), weeks starting on Sunday, trailing spaces that Redshift ignores in comparisons, naive timestamps loaded as UTC, and NULL versus empty string lost in text unloads.

What replaces Kinesis on Google Cloud?

Pub/Sub for most event streams, with ordering keys in place of partition keys and idempotent consumers. If consumers replay more than 31 days back, or depend on Kafka semantics, use Managed Service for Apache Kafka.

What is the GCP equivalent of AWS Glue?

There isn't one product. SQL-shaped Glue jobs become dbt or Dataform models in BigQuery, real Spark jobs move to serverless Spark, streaming jobs to Dataflow or a Pub/Sub subscription, and the Data Catalog to Dataplex and BigLake.

About MigrateX

MigrateX is our platform and service for moving software, data and infrastructure from one technology to another. Automation does the repetitive work, and experienced engineers make the decisions that need judgement. On an AWS to GCP data move, that splits like this:

1. Assessment. We inventory every warehouse schema, Athena table, EMR cluster, Glue job, stream consumer and DAG, with size, owners, and hidden AWS ties like s3:// paths and stored access keys, and we classify Glue jobs by what they actually do.
2. Plan. With your team we profile the workload on BigQuery, choose the pricing and storage billing model per project and dataset, design partitioning and access rules, and order the waves.
3. Migration. Automation translates Redshift SQL, flags the semantic differences that change results, rewrites Glue jobs as plain Spark, swaps Airflow operators and sets up transfers. Engineers handle data models, stream semantics, grants and cost.
4. Proof. Both platforms run side by side with layered reconciliation, and readers switch only when the signed-off metrics match.
5. Handover. We retire AWS jobs and clusters wave by wave, tune reservations once usage is steady, and leave your team owning a platform they understand.

What you get along the way:

• No big switch-over weekend. Reports keep running while readers move one team at a time.
• Numbers you can trust. Reconciliation at the row, fingerprint and metric level before anyone switches.
• A bill you predicted. Pricing and storage models chosen from a measured profile, with quotas set before the first query.
• More than one kind of move. The same approach covers compute, storage and serverless, and front ends like Angular, Ember and jQuery.

Planning a move from AWS to GCP? We'd love to hear about it. Tell us a little about your setup through Contact Us, and we'll start with a free chat about where the work is likely to be.

References

Google Cloud

• Google Cloud. Amazon Redshift to BigQuery migration: Overview. https://docs.cloud.google.com/bigquery/docs/migration/redshift-overview
• Google Cloud. Migrate schema and data from Amazon Redshift. https://docs.cloud.google.com/bigquery/docs/migration/redshift
• Google Cloud. Translate SQL queries with the batch SQL translator. https://docs.cloud.google.com/bigquery/docs/batch-sql-translator
• Google Cloud. BigQuery pricing. https://cloud.google.com/bigquery/pricing
• Google Cloud. Understand BigQuery editions. https://docs.cloud.google.com/bigquery/docs/editions-intro
• Google Cloud. Introduction to slots autoscaling. https://docs.cloud.google.com/bigquery/docs/slots-autoscaling-intro
• Google Cloud. Data retention with time travel and fail-safe. https://docs.cloud.google.com/bigquery/docs/time-travel
• Google Cloud. BigQuery quotas and limits. https://docs.cloud.google.com/bigquery/quotas
• Google Cloud. Introduction to clustered tables. <...