Dark abstract signal wave narrowing and expanding in neon colors, representing Amazon ECS blue/green and canary deployments

Deploying containers safely at scale can mean assembling your own system: traffic-splitting logic, alarm evaluation, rollback automation, approval gates. It’s infrastructure that exists solely to protect the deployment itself. AWS has built all of those capabilities directly into Amazon Elastic Container Service (ECS), so you don’t have to.

“Deploying containers safely at scale can mean assembling your own system: traffic-splitting logic, alarm evaluation, rollback automation, approval gates. It’s infrastructure that exists solely to protect the deployment itself.”

You can access all these tools through a single API endpoint. AWS built them to work out of the box when you use Amazon ECS as your container orchestration platform with the Amazon ECS deployment controller. As a result, you don’t have to build your own abstractions or learn a new framework to get the deployment behavior you want. Instead, you pick and choose the features you want through a service deployment configuration and update it as your needs change.

How the building blocks map to ECS

If you’ve run progressive delivery on Kubernetes with Argo Rollouts or Flagger, you know the shape of the stack: a controller you install and keep current, analysis wired to your metrics backend, and a traffic provider for weighted canaries. Argo Rollouts approximates canary weights with replica counts and uses an ingress controller, service mesh, or Gateway API for precise percentages. Flagger uses a service mesh or ingress controller for weighted canaries, and plain Kubernetes Services for blue/green. Both are mature open-source projects and a great fit when you want to own and tune every layer. ECS takes a different approach and builds the same capabilities into the service. Here’s a comparison of the different deployment controllers:

What you needWith Argo Rollouts or Flagger (Kubernetes)Built into ECS
Deployment strategiesRolling updates come with Kubernetes Deployments; blue/green and canary come from a controller you install and keep currentRolling, blue/green, linear, and canary, set in the service’s deployment configuration
Traffic shiftingArgo Rollouts: replica-count weighting by default, or an ingress controller, service mesh, or Gateway API for precise percentages.
Flagger: a service mesh or ingress controller for weighted canaries
ECS shifts traffic through your ALB, NLB, ECS Service Connect, or VPC Lattice, or calls your own logic through lifecycle hooks
Health and metric analysisAnalysis templates or built-in checks against your metrics provider (Prometheus, Datadog, CloudWatch, and others)Deployment circuit breaker plus CloudWatch alarm monitoring
Automatic rollbackThe controller aborts and shifts traffic back to the stable version when analysis failsECS rolls back on circuit breaker or alarm, and you can pull the cord yourself with the StopServiceDeployment API
Custom checks and approval gatesWebhooks, custom analysis steps, or manual promotionLifecycle hooks with Lambda or pause targets, resumed with the ContinueServiceDeployment API

When are rolling deployments not enough?

Rolling deployments are where ECS started, and they’re still the right default for most customers. In a rolling deployment, ECS replaces tasks running the old version with tasks running the new one, a batch at a time. New tasks start receiving production traffic as soon as they pass health checks, and ECS stops old tasks as the new ones take over. Two settings, minimumHealthyPercent and maximumPercent, control how many tasks can be replaced at once and how much extra capacity the deployment can use. Rolling deployments work with both the deployment circuit breaker and CloudWatch alarm-based automatic rollback, which we’ll cover below.

So, where’s the gap? Let’s say you have an API service running on ECS behind an ALB. You ship a large refactor. A bug surfaces, but it takes time because of metric emission and alarm-processing delays. By the time your alarm fires, a large share of your tasks (maybe all of them) are already running the new version and serving production traffic. ECS rolls back for you, but in a rolling deployment, rollback means another rollout of the previous task definition: ECS must launch new tasks, pass health checks, and replace the bad ones. Your application’s availability will be exposed for some time.

“There’s a limit to how fast a rolling deployment can roll back, which increases customer impact (measured by failed requests per unit of time) from the new, bad application version.”

So, there’s a limit to how fast a rolling deployment can roll back, which increases customer impact (measured by failed requests per unit of time) from the new, bad application version. Blue/green deployments can shorten the duration of the impact. With ECS’s built-in blue/green strategy, the old version stays running alongside the new one for a configurable bake period (up to 24 hours). All production traffic routes to the new version, but the old version’s tasks remain registered and ready to take traffic with a flip of a switch. If something goes wrong, rollback is a simple traffic shift, and not a complete redeploy.

Diagram showing ECS routing all production traffic to the green service revision while the blue revision stays running.
Figure 1: During the bake time, ECS routes all production traffic to the green service revision while the blue revision stays running, so rollback is just a traffic shift.

The tradeoff is that you’re doubling compute capacity during the deployment window. For services where a bad release serving 100% of traffic for even a few minutes is costly, that tradeoff is worth it.

How does ECS know something went wrong?

How would Amazon ECS know when to flip traffic back? There are several ways to perform a service deployment rollback, and they’re all configurable through an ECS service’s deployment configuration:

  • Task launch and health state monitoring during a deployment through circuit breaker
  • Amazon CloudWatch alarm(s) monitoring and auto-rollback
  • Controls to roll back an ongoing service deployment through the StopServiceDeployment API
  • Service deployment lifecycle hooks (Lambda or pause target type)

For API server failures, you can set up Amazon CloudWatch alarms to monitor server response codes and trigger a rollback when a high volume of 4xx or 5xx codes is detected. But what if your application monitoring constructs are outside of Amazon CloudWatch and stored on another service like Datadog, Splunk, or Grafana? With service deployment lifecycle hooks, you can implement your own automation that polls these data sources for alarm data by implementing that logic in a Lambda function and configuring a Lambda target type. The primary consideration is configuring the lifecycle hook for the POST_PRODUCTION_TRAFFIC_SHIFT stage.

Diagram showing the lifecycle stages for blue/green, linear, and canary service deployments, with lifecycle hook integration points.
Figure 2: Lifecycle stages for blue/green, linear, and canary service deployments, with lifecycle hook integration points.

If you can’t use Lambda functions, then implement an Amazon EventBridge trigger with a pause hook target type instead. Amazon ECS will send an Amazon EventBridge event when the service deployment reaches the lifecycle stage configured for the lifecycle hook. The trigger should fire a process you own that runs the same polling loop to monitor an alarm or metric. Once the monitoring condition succeeds, call the ContinueServiceDeployment API to allow the service deployment to progress.

In addition to alarms and lifecycle hooks, the service deployment circuit breaker monitor can stop a bad deployment from progressing. This deployment monitor looks for failed or unhealthy tasks during a service deployment and compares the sum against a threshold. With configurable circuit breaker settings, you can customize the counter algorithm to ignore intermittent task launch or health failures. You can also customize the failure threshold based on your application’s needs.

Advanced deployment strategies

The example above is just one way to use built-in ECS features to customize your service deployment. If a one-step traffic shift isn’t suitable for your application, linear and canary deployments are also alternatives.

Linear and canary deployments are like blue/green deployments in that they use well-defined traffic-shift steps to isolate application traffic between versions. Linear deployments move traffic in equal increments based on your configuration. Canaries are a little different in that there is a two-step migration in application traffic as opposed to n number of incremental shifts.

Not advanced enough? There’s more!

All the deployment tools we talked about today apply to all deployment strategies that ECS supports. Similarly, all deployment strategies are compatible with ECS Service Connect and VPC Lattice network integrations. The only caveat is that lifecycle hooks can be applied to only two stages for a rolling deployment, since that lifecycle is inherently simpler than blue/green, linear, or canary deployments.

“All deployment strategies also work without a load balancer, ECS Service Connect, or VPC Lattice integration! That’s right, even blue/green, linear, and canary deployments.”

All deployment strategies also work without a load balancer, ECS Service Connect, or VPC Lattice integration! That’s right, even blue/green, linear, and canary deployments. We developed these service deployment lifecycle hooks with customization in mind, so we created specific integration points to shift your application traffic. You can control the behavior by writing your own logic, and Amazon ECS invokes a lifecycle hook at the production traffic-shift stage. Just note that this stage takes a Lambda target, since pause hooks can’t be configured there.

Which strategy should you pick?

Here’s the short version of everything above:

Your situationReach forWhat you getWhat it costs
Most everyday changes, or cost-sensitive servicesRolling with circuit breaker and alarmsAutomatic rollback on failed tasks or alarms, and you can tune minimumHealthyPercent and maximumPercent so you’re not paying for a second copy of your serviceRollback is a redeploy, so the bad version serves traffic until it’s replaced
A bad release is expensive, and you need an instant way backBlue/greenOld version stays warm through the bake period; rollback is a traffic shiftDouble compute capacity during the deployment window
You want production signal from a small slice before full exposureCanaryA small share of traffic hits the new version first, then the rest moves in a second stepSame capacity tradeoff as blue/green, and a slower rollout
You want exposure to grow steadily and catch load-dependent issuesLinearTraffic moves in equal increments, with a chance to roll back at each stepSame capacity tradeoff, and the longest rollout
Your monitoring lives outside CloudWatch (Datadog, Splunk, Grafana)Blue/green, linear, or canary, plus lifecycle hooks at the post-production traffic shift stage (rolling supports hooks too, at fewer stages)Your own checks gate the deployment through Lambda or pause targetsCode you write and own
You don’t use a load balancer, Service Connect, or VPC LatticeBlue/green, linear, or canary, plus a Lambda lifecycle hook at the production traffic shift stageECS calls your logic to shift your own trafficCode you write and own

Pick a strategy

Whichever deployment strategy you choose, the guardrails (alarms, circuit breaker, lifecycle hooks, and rollback) come with the service. Configure them once, and spend your engineering time on your product, not your pipeline.

Pick one service, turn on blue/green with a CloudWatch alarm, and see for yourself. The Amazon ECS lifecycle hooks guide is a good next step once you want custom checks.

The post How Amazon ECS handles blue/green, canary, and linear deployments appeared first on The New Stack.