Azure SNAT exhaustion: detect and alert with Dynatrace

Picture a nightly batch that calls a partner API thousands of times. Three nights out of seven a handful of calls time out, the job retries, everything eventually succeeds, and nobody finds a single error in the code. If those machines reach the internet through a public load balancer, the culprit has a name: Azure SNAT exhaustion. Here is how to recognise it, measure it in Dynatrace SaaS, and decide when it deserves a page.

A word of candour first: span.observer did not replay these queries on a live subscription. The Azure facts come from Microsoft’s pages on outbound connections and Load Balancer diagnostics. The DQL is assembled from the Dynatrace documentation and flagged as not executed.

The scope is deliberately narrow: the collection described belongs to Azure Cloud Platform Monitoring, presented as a Dynatrace SaaS service. Nothing was checked for Dynatrace Managed, so do not carry these settings over without its own documentation. If your subscription is not connected yet, start with the Dynatrace Azure integration: Azure Monitor and logs.

Why Azure SNAT exhaustion looks like an application bug

A machine behind a public load balancer has no internet-routable address of its own. Azure rewrites the source address of its outgoing packets to the frontend’s public IP. That is SNAT, and every distinct connection to a given destination IP and port ties up one port. Each public frontend IP brings 64,000 usable ports.

The catch is in the sharing. Those ports are not a common pool: Azure hands them out to backend instances in advance. With no outbound rule, the bigger the pool, the thinner each slice, capped at 1,024 ports per instance. Microsoft’s table below shows the tiers.

The consequence: one chatty machine can run dry while the frontend still holds tens of thousands of ports. Its new connections fail, then work again once a port is freed (after 4 to 120 minutes idle, tunable with an outbound rule). That on-and-off behaviour explains the outage that heals itself and that application logs cannot account for.

Two signals, two levels of urgency

Before writing a query, decide what you need to know. Azure exposes three SNAT metrics on the public load balancer, at a one-minute grain.

• SnatConnectionCount counts new SNAT connections, split by ConnectionState (pending, successful, failed).
• AllocatedSnatPorts is the slice of ports reserved for each instance.
• UsedSnatPorts is what each instance really consumes.

They serve different purposes. Failed SnatConnectionCount proves the outage is happening: anything above zero on the failed state is a live incident. The used-to-allocated ratio warns you ahead of time. Microsoft suggests low severity at 75% and high severity at 90% or 100%. Keep a simple decision rule: the 75% warning goes to a ticket or a channel, while failures and the upper threshold page someone. Also note that multi-dimensional metrics are not supported on a Basic load balancer.

Getting the metrics into Dynatrace SaaS

Dynatrace’s poller queries Azure Monitor every 5 minutes, with a 5-minute lag. It needs a service principal with the Monitoring Reader role and the Microsoft.Insights provider registered on each subscription; without it, topology arrives but no metrics. Remember that collection lag: the alert is not instantaneous.

Three details then cost time if you discover them late.

1. The metric set. The three Standard Load Balancer SNAT metrics are not in the « Recommended » set but in « Recommended+Custom » (microsoft_network.loadbalancers_essentialplus). Skip adding it to the Azure connection and you will hunt for data that never comes.
2. Key naming. The documented rule is cloud.azure.<namespace>.<MetricName>, resource type in lowercase. That gives cloud.azure.microsoft_network.loadbalancers.UsedSnatPorts, but the page never spells it out, so verify it.
3. Dimension case. The poller writes them in lowercase: backendipaddress, protocoltype, connectionstate.

To confirm the key, this discovery query, adapted from a documentation example, lists what your tenant actually receives.

metrics | filter startsWith(metric.key, "cloud.azure.microsoft_network.loadbalancers") | filter dt.da.source == "azure-metric-poller" | fields metric.key, azure.subscription | dedup metric.key 

If the older integration (Dynatrace Classic) is still running, its keys start with ext:. Run both only for the length of a migration, or you will get duplicates and doubled alerts.

The calculation that matters: usage per instance

An average across the whole load balancer hides the problem, because nine quiet machines mask the tenth one that is out of ports. The query below therefore divides used ports by allocated ports, instance by instance. It is built from the pieces the documentation describes and has not been run: try it in a Notebook first.

To turn it into an alert, the documentation points to Settings (available from version 1.344; the Anomaly Detection app is deprecated) and accepts a timeseries query as the source. Set a static threshold on snat_usage_pct: 75 for warning, 90 for critical, taken from Microsoft’s example. The second alert, on failed SnatConnectionCount, first needs a check on how the connectionstate value is spelled in your data. The full mechanics are in creating a Dynatrace alert step by step.

How to check that it works

1. Run the discovery query: three SNAT keys should come back. If not, the « Recommended+Custom » set is not enabled.
2. Run the ratio query: one series per backend address, with values between 0 and 100.
3. Compare one point with Used SNAT Ports in the Azure portal (Average aggregation, one-minute grain), allowing for the collection lag.

What to do when the alert fires

Rebooting machines buys a little time and fixes nothing. The real levers are structural, and Microsoft ranks them like this:

• Azure NAT Gateway: Microsoft’s preferred outbound method. It avoids the same exhaustion limits and takes precedence over the load balancer on the subnet involved.
• Manual allocation through an outbound rule: ports per instance = frontend IPs × 64,000 ÷ pool instances. Useful if you stay on the load balancer.
• Reused connections: an application-side connection pool lowers consumption, and its absence raises the risk.
• Private Link to Storage, SQL or Cosmos DB: that traffic no longer goes through SNAT.

Best practices

• Alert per instance, never on the load balancer average.
• Avoid default allocation in production, which Microsoft advises against.
• Leave the warning at 75%: that margin is what spares the on-call engineer.
• Keep connection failure as the primary symptom, following alerting on symptoms, not causes, and the port ratio as the early signal.

Frequently asked questions

How many SNAT ports does an Azure Load Balancer provide?

According to Microsoft, 64,000 per frontend public IP, less a block of eight ports for each port used in a load-balancing or inbound NAT rule. With default allocation an instance gets 1,024 at most.

Which metric reveals Azure SNAT exhaustion?

A failed-state SnatConnectionCount above zero confirms exhaustion. The ratio of UsedSnatPorts to AllocatedSnatPorts announces it before failures begin.

Should a NAT Gateway replace the Load Balancer?

No, only its outbound function. The NAT Gateway handles outgoing traffic, while the load balancer keeps the inbound side.

Conclusion

To stop losing nights to a phantom outage, track port usage per instance, warn at 75%, escalate at 90% or on failures, then cure the cause with a NAT Gateway or reused connections. Still to confirm on your side: the exact metric key, the case of dimension values and the alert’s behaviour on real data.

span.observer is an independent blog, not affiliated with Dynatrace LLC (nor Microsoft). Trademarks belong to their owners. Services change quickly: check metric names and parameters in the official documentation before applying them.

Originally published on span.observer. French version: https://span.observer/fr/epuisement-snat-azure-dynatrace/.