The short version
The highest monthly infrastructure bill we ever paid was $1,472. Today we run at about $267 a month, 82% less. I am the Technical Lead for Backend & Platform at Byteria, where we build Alphyn, a mobile app security platform for RASP and device attestation. I did the infrastructure work alone over four weeks, in 75 pull requests.
Some context first, because it explains the rest. Customers run Alphyn on-premises, in their own infrastructure. Our cloud environment, on managed Kubernetes, is where we run demos and proofs of concept. It was built when it also carried customer workloads, and it had kept that size.
I measured what each workload really used, sized everything to that, and removed the components we could live without.
| Category | Before (October invoice) | After (monthly run rate, mid-November) |
|---|---|---|
| Kubernetes nodes | $987 | $129 |
| Managed databases | $275 | $55 |
| Block storage | $126 | $12 |
| Standalone VMs | $56 | $43 |
| Load balancer | $18 | $18 |
| Object storage and snapshots | $10 | $10 |
| Total | $1,472 | $267 |
All figures are USD per month, as billed. "Before" is our October invoice, the highest we ever paid. The work began midway through that month, when we were running at a rate of $1,646 a month. "After" is the monthly cost of what is running today, including about $28 of on-demand CI nodes. November is a transition month and will close between $370 and $400. December will be the first full invoice at the new rate, and I will update this post when it arrives.
What we paid for it: production now runs on a single node, the database is half its former size, and there is no staging environment. More on that further down.
Where the money was going
Two thirds of the bill was Kubernetes nodes: $987 of $1,472. By mid-October the cluster had eight of them, four general-purpose nodes with 32 GB of memory each and four 8 GB nodes reserved for Elasticsearch. The run rate at that point was $1,646 a month.
Billing breakdown for 1 September to 1 October, 31 days, before the work started: $1,237.50 before tax, $1,485.00 with tax. Kubernetes is the $828.70 line.
Daily spend in September: about $41 a day before tax.
I built this cluster myself. It grew one node at a time as we added products, environments and CI capacity, and each addition made sense on the day it was made. Last month I went back and measured the whole thing.
When I looked, the nodes were mostly idle but fully reserved. Kubernetes places pods by what they request, not by what they use, so a pod that asks for 1 GiB and uses 50 MiB still takes 1 GiB of room.
On the last 32 GB node, measured over its final week, memory use averaged 49% and CPU 11%. Requests on the same node claimed 83% of memory and 67% of CPU. The scheduler treated that machine as nearly full while half of it sat idle.
So I went through the cluster one workload at a time with two questions: how far is its request from its measured peak, and can something we already run do its job?
What I changed
Measured against the October peak of $1,646, the changes add up like this:
The node pool is the largest line, and it took several of the steps below to get there. Each one freed enough reserved memory to switch off another node.
CI got its own pool that scales to zero
Our self-hosted CI runners sat on always-on nodes and reserved 10 GiB of memory each, so that capacity was paid for around the clock whether or not a build was running.
I moved the runners to a dedicated pool of 8 vCPU / 16 GB nodes that scales between zero and seven, tainted so nothing else lands there. That removed $230 of fixed cost and added $28 to $84 of usage cost, depending on how busy the month is.
CI also got more headroom. With the runners isolated I could raise their CPU limit from 2 to 7 cores and allow six builds at once in place of two. Once the new pool settled, the 95th-percentile wait in the queue fell from 31 minutes to under 6. The cost is a cold start: from an empty pool, a job begins 2 to 3 minutes after it is queued. For a typical job the two effects cancel out, and the gain is at busy times.
Requests sized from measured peaks
The rule: request = at least the 7-day peak + 25%. For workloads that autoscale on memory, the peak may not exceed 65% of the request, so the autoscaler has room to act. Limits stayed where they were.
| Workload | Memory request | Measured peak |
|---|---|---|
| Container registry | 1,344 Mi to 224 Mi | 138 Mi |
| Runtime security agent | 384 Mi to 272 Mi | 210 Mi |
| Backup node agent | 128 Mi to 72 Mi | 56 Mi |
| Session and rate-limit store (Valkey) | 256 Mi to 64 Mi | 15 Mi |
Sizing went both ways: where the measured peak was above the request, the request went up.
A lighter observability stack
- Logs: Elasticsearch and Kibana gave way to a single-node VictoriaLogs. Elasticsearch requested 2 GiB and Kibana another 1 GiB. VictoriaLogs requests 512 Mi and keeps 21 days.
- Metrics: Prometheus gave way to a single-node VictoriaMetrics with 10 days of retention. I ran both side by side first: all 13 scrape jobs matched, series counts were 104,137 against 103,503, and all 233 rules evaluated without errors.
- Registry: image layers moved from a 200 GB block volume to object storage, which bills for what is stored.
The services were rewritten in Go
This part was a team effort. I led it as tech lead, wrote the first Go version of five of the six services and ran the production cutover. Teammates wrote the sixth, our main API. Since the summer we had been rewriting our Java and NestJS services in Go, for cost and for speed. Production cut over on 10 November, and it is what made the last step possible. Our on-premises images are almost identical to what we run in the cloud, so this change and the move to PostgreSQL below ship to customers too, and they get the smaller footprint on their own hardware.
The main API image is now 14 MB. The requests were still sized for the old runtimes, so the request per pod went from 1,280 Mi to 128 Mi. The speed we gained is in startup, builds and tests.
PostgreSQL took over from Elasticsearch and RabbitMQ
The application used Elasticsearch to store logs and RabbitMQ to pass messages. The new code writes logs to PostgreSQL tables and uses an outbox table as its queue. Log rows expire after 90 or 365 days, processed messages after 7.
That removed the four-node Elasticsearch pool ($230) and the RabbitMQ cluster. I did it in stages: staging first, then production Elasticsearch from three nodes to one after a fresh snapshot (the fourth node in the pool had carried staging's), then removal once nothing read from or wrote to it. Before deleting, I restored two indices under new names and compared document counts.
Nine database clusters became two, and most of the disks went
- The main production PostgreSQL went from 4 GB / 2 vCPU with 80 GiB of disk to 2 GB / 1 vCPU with 30 GiB ($97 to $37).
- The production Valkey, which holds session revocations, rate limits and job locks, moved from a managed instance into the cluster as a single-replica StatefulSet.
- Several small application databases moved onto one 1 GB cluster ($18) by logical replication. I compared all 102 tables by row count and MD5 before switching, and the switch itself was an ordinary rolling restart.
- The rest belonged to environments we retired, staging among them.
Block storage went from 1,085 GB to 98 GB along the way. Most of that followed from the steps above: the registry volume, the Elasticsearch data disks and the old log store. The largest single item was a 300 GB artifact repository, retired after its contents were backed up and verified file by file with SHA-1.
Staging moved to laptops
Staging's own database, Valkey and search node cost $152 a month, and it shared a 32 GB node with other environments that were retired the same day. I shut it down and we replaced it with a Docker Compose environment that starts with one command: PostgreSQL behind PgBouncer, Valkey, MinIO for object storage, Mailpit for email, the schema migrator, every service, the TLS sidecar and an nginx edge.
One 16 GB node
After all of this, requests on the last 32 GB node added up to 15.0 GiB of memory and 4.44 cores. A 16 GB node offers about 13 GiB and 3.9 cores. I trimmed the remaining requests by the same rule and swapped the $230 node for a $101 one.
The pool autoscales between one and two nodes. I also marked 14 workloads as safe to evict, so that a second node can drain and disappear once a burst is over.
Timeline
The drop came in two waves. The first, in the second half of October, halved the run rate. The second, in the first days of November, took it to its present level.
| Date | What had changed | Monthly run rate, CI excluded |
|---|---|---|
| 13 Oct | Peak: eight nodes | $1,646 |
| 17 Oct | CI on its own pool | $1,436 |
| 27 Oct | Staging and other environments retired, Elasticsearch down to one node | $808 |
| 3 Nov | Lighter observability, registry on object storage, smaller database | $464 |
| 11 Nov | Go in production, Elasticsearch and RabbitMQ gone, one 16 GB node | $230 |
| 17 Nov | Today (plus about $28 of CI usage) | $238 |
Daily spend from 1 to 16 November, before tax: from about $25 a day to about $7.
Month to date on 17 November: $203.99 before tax, $244.79 with tax.
What we gave up
What makes these trade-offs acceptable is the role of this environment. Customers' own deployments do not depend on it at runtime, and our SDKs are designed to keep working when it cannot be reached.
Everything in production now runs on one 16 GB machine. If that node dies, everything is down until a replacement joins. The planned swap on 10 November shows what that looks like: the new node was ready 1 minute 47 seconds after it was created, stateless pods were ready 5 to 17 seconds after being scheduled, and our application's one pod with a disk attached took 157 seconds.
That node is also full on paper. Requests claim 94% of memory and 90% of CPU, while real use averages 56% and 25%. The first pod that does not fit brings up a second node at $101 a month, so $267 is the minimum and a busy month will cost more.
The production database now has half the memory and CPU it had, and carries the logs and the queue as well. It runs as a single node, protected by automated backups with point-in-time recovery. In the first three days after the cutover we logged queries that took several seconds. We fixed those in the days that followed, and queries that take longer than a second are now rare.
There is no staging any more. The local environment verifies application behaviour and schema migrations, but not the deployment itself: rolling updates, resource limits, ingress rules and certificates get their first real test in production. Since staging went away, no production deployment has been reverted. That record is only a few weeks old.
And one thing did go wrong. During the node swap, the single-replica Valkey needed two and a half minutes to move, because its disk had to detach from the old node and attach to the new one. Our dashboard API refuses requests when it cannot check session revocations and rate limits there. That is a deliberate fail-closed choice, and it meant the API was unavailable for 2 minutes 50 seconds. The main API returned no 5xx to clients during the swap, or anywhere else in the three weeks of logs I checked. I had not planned for either part: the volume being the slow piece of the move, or the store's downtime becoming the API's downtime.
The way back
If any of this stops being acceptable, this is what undoing it costs.
- More capacity is automatic: the autoscaler adds a second node for $101 a month.
- A database standby is a setting on the managed cluster, not a migration, and costs about $36 a month.
- Staging comes back for what its own resources cost before, $152 a month, plus whatever node capacity it needs.
That is where things stand one week after the last change.







