How I Migrate DigitalOcean Spaces to S3: 12 Million Objects, Two Hops, Zero Failures

On 5–6 October 2026 I had to migrate DigitalOcean Spaces to S3 for a fintech platform I run infra for. The source was a Space in Frankfurt (fra1) and the destination was S3 in Tokyo (ap-northeast-1): about 2.5 TiB across 12,071,264 objects. The final hop finished at 10:27 UTC on 6 October with 12,071,264 objects succeeded and 0 failed.

That result is clean. The route to it wasn't. This is the version of the story with the mistakes left in, because the mistakes are where the useful lessons are.

I'm Nazmul Ahasan, and I lead Platform Engineering at NEXT Ventures. I write about AWS, cost and reliability work at nazmul-ahasan.pro.bd.

The shape of the problem

Look at the average object size before anything else: about 220 KB. This isn't a "move a few big files" job. It's a massive-small-file workload, which means the limit is requests per second, not bandwidth. Every object costs you a GET, a PUT, and possibly a HEAD or two. Multiply that by twelve million.

The second fact is geography. Frankfurt to Tokyo is roughly 250–280 ms round trip. A copy tool running anywhere along that path pays that latency on every small object. You can throw concurrency at it, but you'll hit the source's rate limits long before you beat physics.

So I didn't do a direct copy. I did two hops:

  1. Hop 1, same metro: rclone on an EC2 instance in Frankfurt, copying DO Spaces to an S3 staging bucket in eu-central-1. Low latency on both sides.
  2. Hop 2, server-side: S3 Batch Operations copying staging to the production bucket in Tokyo. The bytes travel over the AWS backbone and never pass through a machine I manage.

If you take one thing from this post, take that one. Do the slow, chatty part of the transfer where latency is cheap, then let AWS move the bulk between its own regions.

Hop 1: rclone, DO Spaces to S3 Frankfurt

Config, and my first dumb mistake

Spaces keys come from the DigitalOcean control panel under API → Spaces Keys. The secret is shown exactly once. Copy it straight into your secret store.

The working config:

[dospaces] type = s3 provider = DigitalOcean endpoint = fra1.digitaloceanspaces.com region = fra1 access_key_id = <SPACES_KEY> secret_access_key = <SPACES_SECRET> [s3fra] type = s3 provider = AWS env_auth = true region = eu-central-1

My first version had the Space name baked into the endpoint host, my-space.fra1.digitaloceanspaces.com, because that's the URL you see in the DO console. rclone lsd dospaces: came back empty. No error, just nothing. The endpoint should be the regional host, and the bucket goes in the path: dospaces:my-source-space. Five minutes lost, and a reminder that a silent empty result is worse than an error.

A smaller red herring: on the AWS side the instance role was scoped to specific buckets, so rclone lsd s3fra: returned a 403 on ListAllMyBuckets. That's harmless. You don't need to list every bucket in the account. Go straight to s3fra:my-staging-bucket-fra and move on. Don't widen the policy to make a cosmetic error go away.

Tuning: SlowDown, then crawling

My first serious run fired a lot of concurrency at Spaces and got 503 SlowDown back. That's DO telling you to back off. Two things were burning requests: raw parallelism, and rclone doing a HEAD on objects it didn't need to inspect.

Once the 503s stopped it was still crawling at about 150 objects per second. The cause was modtime comparison. To compare modification times, rclone HEADs the object on both sides. Two extra round trips per object for twelve million objects is a lot of waiting. For a one-shot migration into an empty bucket, size is a perfectly good comparison.

I raised the request rate step by step, --tpslimit 300 → 600 → 800, and watched for throttling at each step. This is the command I finished with:

rclone copy dospaces:my-source-space s3fra:my-staging-bucket-fra \ --size-only \ --transfers=96 --checkers=32 \ --tpslimit=800 --tpslimit-burst=100 \ --s3-no-head-object --s3-no-head \ --s3-upload-concurrency=2 --s3-chunk-size=5M \ --s3-no-check-bucket \ --retries=10 --low-level-retries=20 \ --stats=1m --stats-one-line \ --log-file=rclone-hop1.log -v

Why the flags matter:

  • --size-only removes the modtime HEADs. It's the biggest single speedup.
  • --s3-no-head and --s3-no-head-object skip the HEAD before and after each upload. For small files, those HEADs are a big share of the total work.
  • --s3-upload-concurrency=2 and --s3-chunk-size=5M fit objects of about 220 KB. Big multipart settings do nothing here except use memory.
  • --s3-no-check-bucket stops rclone from checking or creating the bucket over and over.
  • --tpslimit=800 is the number that matters. At 800 TPS I saw zero throttling.

It sustained about 400 objects per second, peaking around 145 MiB/s. Hop 1 took about a day, tuning included.

The six hours I lost

This was my most expensive mistake, and it had nothing to do with clever flags.

The instance started as a c6i.xlarge with 8 GB of RAM. rclone builds directory listings in memory, and a full listing of 12 million objects holds about 7 GB. The first real run ran out of memory and froze the box. Nothing crashed loudly. The process just stopped making progress, and I didn't notice for about six hours.

I also tried --fast-list, which is supposed to cut list calls by fetching everything in one go. On this bucket it sat idle for around 30 minutes building the listing before it copied anything. On a box that's already short of memory, that's the wrong trade.

The fixes:

  • Resize to an r6i.4xlarge. Memory-optimised, because the listing is the workload.
  • Work per top-level folder instead of the whole bucket at once.
  • Monitor long runs. Tail --stats-one-line output, alert if the transferred count stops moving, watch memory. A watch loop on the log would have saved me most of a working day.

My opinion: any copy job expected to run longer than lunch needs a liveness check. "It's probably still going" doesn't count as monitoring.

Verifying hop 1

A full-bucket rclone check hit the same memory wall, so I checked per folder:

rclone check dospaces:my-source-space/Invoices s3fra:my-staging-bucket-fra/Invoices \ --one-way --size-only \ --missing-on-dst missing.txt --differ differ.txt

--one-way asks only one question: is everything at the source present at the destination? For a migration, that's the right question.

The gaps were real, and they were all explainable. The application was still live. New uploads under the current 2026/10 prefix kept arriving after the bulk pass had listed them. Some files had been regenerated in place with a different size. Both kinds got targeted re-copies:

rclone copy dospaces:my-source-space s3fra:my-staging-bucket-fra \ --files-from missing.txt --no-traverse --size-only

Feeding the list in with --files-from plus --no-traverse means rclone copies exactly those keys without listing the bucket again. For a sense of scale, the top-ups were about 1,830 certificates, 5,382 invoices, 282 agreements, 46 photos, and 5 dispute files with size differences. This bucket holds user documents like invoices and certificates. "Mostly there" isn't acceptable for data like that.

Hop 2: S3 Batch Operations to migrate DigitalOcean Spaces to S3 Tokyo

Once staging matched the source, the rest was meant to be the easy part. It was, eventually.

The IAM role

Batch Operations assumes a role that trusts batchoperations.s3.amazonaws.com:

{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "batchoperations.s3.amazonaws.com" }, "Action": "sts:AssumeRole" }] }

Permissions, split by purpose:

{ "Version": "2012-10-17", "Statement": [ { "Sid": "SourceRead", "Effect": "Allow", "Action": ["s3:GetObject", "s3:GetObjectTagging", "s3:GetObjectVersion", "s3:ListBucket", "s3:GetBucketLocation"], "Resource": ["arn:aws:s3:::my-staging-bucket-fra", "arn:aws:s3:::my-staging-bucket-fra/*"] }, { "Sid": "DestWrite", "Effect": "Allow", "Action": ["s3:PutObject", "s3:PutObjectTagging", "s3:PutObjectAcl", "s3:GetBucketLocation", "s3:ListBucket"], "Resource": ["arn:aws:s3:::my-prod-bucket-tokyo", "arn:aws:s3:::my-prod-bucket-tokyo/*"] }, { "Sid": "ManifestAndReports", "Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject"], "Resource": ["arn:aws:s3:::my-prod-bucket-tokyo/batch-manifests/*", "arn:aws:s3:::my-prod-bucket-tokyo/batch-reports/*"] } ] }

Four failures before one success

I tried to be clever with the generated manifest first. The console and API can build the manifest for you from filters, so you don't write a CSV yourself. That's how the day went:

  1. Instant failure from the EligibleForReplication filter. That filter only works on versioned buckets, and staging wasn't versioned. I removed it.
  2. Missing PutInventoryConfiguration. The auto-generated manifest uses inventory machinery under the hood, and the role needs that permission.
  3. 301 PermanentRedirect, 100% failure. An auto-manifest job runs in the source region. Writing cross-region to Tokyo from a Frankfurt job redirected every task.
  4. CrossRegionOperationProhibited. When I tried to fix the region, Batch Operations made it clear it can't generate a manifest across regions.

The lesson: if source and destination are in different regions, build the manifest yourself and run the job in the destination region.

Building the manifest by hand

s5cmd is far faster than the AWS CLI at listing millions of keys. I piped its output through sed to strip it down to keys, then through a short Python step that URL-encodes each key:

s5cmd ls 's3://my-staging-bucket-fra/*' \ | sed -E 's/^.{19}[[:space:]]+[0-9]+[[:space:]]+//' \ | python3 -c ' import sys, urllib.parse for line in sys.stdin: k = line.rstrip("\n") if k: print("my-staging-bucket-fra," + urllib.parse.quote(k, safe="")) ' > manifest.csv wc -l manifest.csv # 12071264 aws s3 cp manifest.csv s3://my-prod-bucket-tokyo/batch-manifests/manifest.csv aws s3api head-object --bucket my-prod-bucket-tokyo \ --key batch-manifests/manifest.csv --query ETag --output text

Use safe=''. Encode everything, slashes included, and Batch decodes it correctly. Keys with spaces, plus signs, or non-ASCII characters are where hand-built manifests usually break. The file came to about 1.23 GB with exactly 12,071,264 lines, which matched the object count. It went into the destination-region bucket next to the job.

The job 

aws s3control create-job --region ap-northeast-1 \ --account-id "$ACCOUNT_ID" \ --operation '{"S3PutObjectCopy":{ "TargetResource":"arn:aws:s3:::my-prod-bucket-tokyo", "MetadataDirective":"COPY", "StorageClass":"STANDARD"}}' \ --manifest '{"Spec":{"Format":"S3BatchOperations_CSV_20180820","Fields":["Bucket","Key"]}, "Location":{"ObjectArn":"arn:aws:s3:::my-prod-bucket-tokyo/batch-manifests/manifest.csv", "ETag":"<manifest-etag>"}}' \ --report '{"Bucket":"arn:aws:s3:::my-prod-bucket-tokyo","Prefix":"batch-reports", "Format":"Report_CSV_20180820","Enabled":true,"ReportScope":"FailedTasksOnly"}' \ --priority 10 --role-arn "$BATCH_ROLE_ARN" \ --confirmation-required --client-request-token "$(uuidgen)"

MetadataDirective: COPY keeps content types and user metadata. FailedTasksOnly keeps the report small. If it's empty, that's your proof.

The job went through the lifecycle you'd hope for. Preparing read 12,071,264 rows with 0 errors. It then sat Suspended, waiting for my confirmation. I read the summary, then released it:

aws s3control update-job-status --region ap-northeast-1 \ --account-id "$ACCOUNT_ID" --job-id "$JOB_ID" --requested-job-status Ready

Active, then Complete: 12,071,264 succeeded, 0 failed. Always keep --confirmation-required on. Validating the manifest before twelve million PUTs is cheap insurance.

The mistake I'm a bit embarrassed by

Before I settled on Batch, I briefly ran an rclone S3-to-S3 copy on the EC2 box and thought it was a backbone transfer. It isn't. rclone always streams the bytes through the machine it runs on. Server-side copy within a single provider is a narrow case, and it doesn't cover cross-region at this scale with this tooling. If you want the bytes to stay inside AWS, use Batch Operations or replication, not a relay instance.

What hop 2 cost

All of these are approximate:

  • Batch Operations job and per-object fees: ~$12
  • ~12 million PUT requests in Tokyo: ~$56
  • ~2.5 TB inter-region transfer: ~$50

That's about $120 for hop 2. DigitalOcean egress and the EC2 time for hop 1 come on top. Staging in Frankfurt isn't free, but it's cheap next to an engineer babysitting a latency-bound copy for a week.

Cutover and cleanup

The manifest is a snapshot. Anything written to Spaces after you list it isn't in it. So the cutover went like this:

  1. Switch application writes to the Tokyo bucket, or briefly pause uploads.
  2. Run a final delta sync of the recent prefixes directly from DO to Tokyo. It's small enough that latency doesn't matter. Use --checksum, because this is the last chance to catch an in-place change: 
 rclone copy dospaces:my-source-space/2026/10 s3tokyo:my-prod-bucket-tokyo/2026/10 --checksum -v
  1. Clean up: terminate the EC2 instance, put a lifecycle expiration rule on the staging bucket, delete the manifest and report prefixes, and remove the Batch role.
  2. Rotate every credential used during the migration. Spaces keys, any access keys, all of it. Migration credentials are long-lived by accident and broad by design, so don't leave them lying around.

What I'd do differently

  • Start on a memory-optimised instance and work per folder from minute one.
  • Monitor from the first run. Six hours of a frozen box is the cost I resent most.
  • Use --size-only and the no-HEAD flags immediately for small files into an empty destination.
  • Skip the generated manifest for cross-region jobs. Build the CSV and run in the destination region.
  • Remember that rclone is a relay, not a backbone.

Two hops, about a day and a half including mistakes, and zero failed objects. If you're planning a similar move and want to compare notes, you'll find me at nazmul-ahasan.pro.bd.