The Safety Alert Waited Behind 40 Likes: A Post-Mortem of a Naive .NET Notification Fan-Out

This is a made-up incident, but every piece of it is something I've seen in a real code review. A user's photo goes viral. Forty "liked your photo" pushes land on their phone in under a minute. Somewhere in the middle of that storm, a high-priority safety alert for the same user is generated. It arrives fourth from last.

Nothing crashed. No exception was logged. The notification service did exactly what it was written to do: loop over events, call the push vendor, move on.

So let's write the post-mortem the way a team would, blameless, with a timeline, root causes, and the fixes. The code below is the shape of the fix in .NET 10, with the heavy parts trimmed and marked.

Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. The complete, working, tested project behind this post (facebook-notification-fanout-dotnet, Clean Architecture, 100+ tests, plus an in-depth PDF) is inside. Join Tech Skill Builder

The original design (what we shipped)

// The whole "notification service", more or less. foreach (var evt in incomingEvents) { foreach (var device in await devices.ForUserAsync(evt.RecipientUserId)) await pushVendor.SendAsync(device.Token, evt.Title, evt.Body); } 

One code path for every kind of notification. No priority, no batching, no memory of what was already sent.

Timeline

Root cause 1: every notification had the same priority

A safety alert and a like went through the same FIFO loop. Whatever arrived first went out first.

Fix: priority is part of the event, not a UI concern. Four lanes are enough to start:

public enum NotificationPriority { P0 = 0, P1 = 1, P2 = 2, P3 = 3 } // P0: safety / security. P3: digestible social noise. 

And the queue hands out the most urgent ready work first. That's the whole point of a priority queue in the style of Meta's FOQS: consumers pull, the queue orders.

public interface IPriorityQueue { Task<string> EnqueueAsync(string topic, NotificationPriority priority, string payloadJson, DateTimeOffset? deliverAfter = null, TimeSpan? leaseDuration = null, string? metadata = null, CancellationToken ct = default); Task<IReadOnlyList<QueueItem>> DequeueAsync(string topic, int count, string leaseOwner, CancellationToken ct = default); Task AckAsync(string itemId, CancellationToken ct = default); Task NackAsync(string itemId, TimeSpan? redeliverAfter = null, string? metadata = null, CancellationToken ct = default); } // Ordering rule: priority, then deliver_after, then a stable id. // The in-memory implementation and lease expiry are in the complete project. 

Root cause 2: forty likes became forty pushes

Nobody needs forty buzzes for forty likes on one photo. They need one: "40 people liked your photo."

Fix: batch the delay-tolerant kinds on the way in, before they hit the queue. Batching after dequeue saves you nothing on queue pressure.

public static bool IsBatchable(NotificationEvent evt) => evt.Priority is NotificationPriority.P2 or NotificationPriority.P3 && evt.Kind is NotificationKind.Like or NotificationKind.Comment or NotificationKind.FriendSuggestion or NotificationKind.Digest; public bool TryOffer(NotificationEvent evt, out BatchedPayload? flushedImmediate) { flushedImmediate = null; if (!IsBatchable(evt)) return false; // P0 never gets here var collapse = evt.CollapseKey ?? $"{evt.Kind}:{evt.RecipientUserId}"; // bucket per user + collapse key, opened on a TimeProvider clock, // flushed on max size or when the window expires // ... full bucket + flush logic in the complete project return true; } 

Two details matter more than the code:

• The collapse key must describe the thing, not the event. If you use the event id as the collapse key, every like is its own bucket and you've built a very elaborate foreach.
• Use TimeProvider, not DateTimeOffset.UtcNow. The batching window is the thing you most want to test, and you can't test it if you have to sleep.

Root cause 3: "important" had no special path

The user had muted likes at night and set a push cap per hour. Good settings. But the code applied them to everything, and a muted kind or a capped hour would have swallowed the safety alert too.

Fix: preferences are evaluated in a router with an explicit P0 rule, and the rule is written down where reviewers can see it:

public PreferenceOutcome Evaluate(NotificationEvent evt) { var prefs = _prefs.Get(evt.RecipientUserId); if (prefs is null) return PreferenceOutcome.Allow; if (prefs.MutedKinds.Contains(evt.Kind) && evt.Priority != NotificationPriority.P0) return PreferenceOutcome.DropMuted; if (evt.Priority == NotificationPriority.P0) return PreferenceOutcome.Allow; // bypasses DND and caps // DND window -> DeferDnd, hourly cap -> DeferFrequencyCap // ... full rules in the complete project return PreferenceOutcome.Allow; } 

Yes, that's privilege for P0. The alternative is a second "urgent" pipeline nobody load-tests. One pipeline with a visible bypass, plus a metric that screams when P0 volume spikes, is easier to trust.

Root cause 4: the vendor was inside the loop

When the push vendor slowed down, the loop slowed down. Retries happened inline, so every retry for a like delayed every notification behind it.

Fix: channel delivery is an adapter behind an interface, called by a worker that acks or nacks queue items with a lease:

public interface IChannelAdapter { DeliveryChannel Channel { get; } // Apns, Fcm, Email, Sms Task<AdapterSendResult> SendAsync(AdapterSendRequest request, CancellationToken ct = default); } // Worker (BackgroundService), per dequeued item: // success -> AckAsync // failure -> NackAsync with a growing redeliver delay // too many retries -> ack it, write a Failed receipt, keep the lane moving // ... worker loop, receipts and fake APNs/FCM adapters in the complete project 

If a worker dies mid-send, the lease expires and the item becomes ready again. That's at-least-once delivery, so the ingest side needs an idempotency key check, or retries turn into duplicate pushes. Fake adapters that can be told to fail the next N sends let you test all of this in CI without a vendor account.

The acceptance test that would have caught it

If I could add only one test to a notification service, it would be this scene:

1. Seed one user with two devices.
2. Ingest 40 P3 likes with the same collapse key.
3. Ingest one P0 safety alert.
4. Drain the worker before the batching window ends.

Expected: the P0 alert is delivered right away and isn't batched. No like pushes yet. Then advance the fake clock past the window, drain again, and exactly one batched push shows up carrying all 40 events.

If the alert is missing from the first drain, P0 went into the batcher or the queue sorted by insertion order. If you see 40 pushes, the collapse key is wrong.

Action items

1. Priority on every event, with P0 defined in writing.
2. A pull priority queue with leases, ack and nack.
3. Batching for delay-tolerant kinds on the ingest path, keyed by a meaningful collapse key.
4. A preference router with an explicit P0 bypass and a metric on it.
5. Channel adapters out of the hot path, with retries as nacks, not inline loops.
6. Idempotency on ingest, because at-least-once is the honest guarantee.
7. The like-storm-plus-alert scene in CI.

One question this post leaves open on purpose: if the user is also inside a do-not-disturb window and over their hourly cap, and a like storm and a safety alert arrive in the same second, which pieces of your pipeline should even see the likes, and when should they finally go out?

Get the complete project

The full .NET 10 solution (facebook-notification-fanout-dotnet) is in Tech Skill Builder: Domain, Application, Infrastructure and Api projects, the FOQS-style in-memory queue with leases, the social batcher, the preference router, the fan-out worker, fake APNs/FCM adapters, receipts, a CLI with the like-storm scene and others, 100+ unit and integration tests, and an in-depth PDF that walks through the design before the code.

Tech Skill Builder is only $5/month for a very limited time (normally $10). Grab it now before it goes back to $10.
Join Tech Skill Builder