I found it by reading a send log, not a page. Dozens of "The fire is lit" emails had gone out at 8pm, and an hour later the daily counter on our Resend account read 97 of 100. Three sign-in codes were left for the rest of the day, during the one hour people were actually trying to log in.
Fyreflock is a random video and text chat app. It runs a feature I'd nicknamed the campfire: the matching queue only opens in set evening windows, and if you'd asked to be told when it opens, we'd email you. Turns out that email and the one carrying your sign-in code were drawing from the same tap, and nobody had told the tap it had two jobs.
One account, two very different emails
Resend's free plan gives you 100 sends a day. For a year, every email Fyreflock sent went through the same account on that plan: sign-in codes, the campfire nudge, award notices, the eventual launch broadcast. The code that watched for the campfire opening looked like this:
// apps/server/src/reminders.ts, before the fix export async function checkCampfireOpening(ctx: AppCtx): Promise<ReminderResult> { const isOpen = fire.enabled && fire.open; if (!isOpen || wasOpen) return { opened: false, sent: 0 }; if (!ctx.mailer.sendAnnouncement) return { opened: true, sent: 0 }; const due = all(ctx.db, `SELECT id, email, unsub_token FROM waitlist WHERE remind = 1 AND (last_reminded_at IS NULL OR last_reminded_at < ?) LIMIT 500`, now - MIN_GAP_MS, ); for (const row of due) { await ctx.mailer.sendAnnouncement(row.email, 'The fire is lit 🔥', reminderText(...)); // ... } } This runs once a minute, watching for the closed-to-open edge. The day it fired, 102 people were on the waitlist with remind = 1. LIMIT 500 was never going to bind anything, so the whole batch went out in one pass, against the same 100-a-day account that sendCode uses for every sign-in. Do that at 8pm, the hour the queue opens and the hour people try to sign in, and you've built a feature whose entire job is to compete with login for the same daily allowance.
sequenceDiagram participant Clock as reminder loop (every 60s) participant DB as waitlist table participant Mail as shared Resend account (100/day) participant Login as someone signing in Clock->>DB: campfire just opened, who's owed a nudge? DB-->>Clock: 102 rows, remind = 1 loop up to 500 rows, no pacing Clock->>Mail: sendAnnouncement("the fire is lit") end Note over Mail: 97 of 100 spent in under a minute Login->>Mail: sendCode(code) Mail-->>Login: 429, quota's gone for todayThe fix that shipped that evening
I wasn't going to redesign the mail layer at 9pm with the queue still open. The honest fix was small: turn the nudge off by default, and bound it when it's on.
if (getSetting(ctx.db, 'reminders_enabled') !== '1') return { opened: true, sent: 0 }; if (!ctx.mailer.sendAnnouncement) return { opened: true, sent: 0 }; const batch = Math.max(1, Number.parseInt(getSetting(ctx.db, 'reminder_batch_max'), 10) || 25); const due = all(ctx.db, `SELECT id, email, unsub_token FROM waitlist WHERE remind = 1 AND (last_reminded_at IS NULL OR last_reminded_at < ?) ORDER BY created_at ASC LIMIT ?`, now - MIN_GAP_MS, batch, ); reminders_enabled defaults to '0'. Someone has to look at the day's headroom and switch it on. reminder_batch_max caps it at 25 even then, because 500 was never a number any Resend plan we're on could honor.
The subtle part is where that switch gets read. It has to come after the open/closed edge gets recorded, not before, or flipping it mid-window finds a stale wasOpen = 0 and fires immediately at whoever just flipped it, which is the exact kind of surprise burst the switch exists to stop. Off has to mean silent, not blind: the edge-tracking keeps running while it's off, so turning it on waits for the next opening instead of the one already in progress. I wrote a test for exactly that, because it reads as obviously correct until you trace what happens if you flip the switch at 20:01 instead of 19:59.
One other test was wrong before it was right. I seeded the batch-cap case by posting to /api/waitlist in a loop, and the assertion kept failing: looked like the cap logic was broken. It wasn't. That endpoint rate-limits signups to 5 an hour per IP, so my own test harness was the thing capping the rows at 5, not the code I was testing. Writing rows straight into the table fixed the test and taught me to read my own rate limits before blaming the feature.
The fix that actually ends it
Default-off and a cap of 25 make the lockout impossible rather than merely unlikely. That was a deliberate choice, written into the commit at the time, to not redesign the send path on an evening I was also trying to put a fire out. Six weeks later I came back and gave sign-in codes an account that nothing else touches.
// apps/server/src/mailer.ts export class SplitMailer implements Mailer { constructor( private readonly codes: Mailer, bulk: Pick<Required<Mailer>, 'sendAnnouncement'> | null, ) { if (bulk) this.sendAnnouncement = (email, subject, text, opts) => bulk.sendAnnouncement(email, subject, text, opts); } sendCode(email: string, code: string) { return this.codes.sendCode(email, code); // never shared with announcements } } sendCode goes through the old fallback chain, untouched. sendAnnouncement goes to a pool of separate free Resend accounts, and the two paths never cross: if every bulk account is spent, the broadcast fails outright instead of falling back to the codes account, which is exactly the fall-through that caused the original lockout. With no bulk accounts configured, sendAnnouncement is undefined, and every caller already checks for that before using it.
Picking which bulk account handles a given person was the part I almost got wrong in the obvious direction. Rotating senders per email looks like the move to dodge spam filters. It's closer to the opposite:
sendAnnouncement = async (email, subject, text, opts = {}) => { const home = parseInt(sha256(email.trim().toLowerCase()).slice(0, 8), 16); // ring.map rotates the account list starting at `home`, so the same person // always lands on the same account unless it's exhausted or refused }; Spreading one person's mail across several sending domains over time is what filters call snowshoe spam, and it also means that person never builds the open/reply history on any one domain that actually earns the inbox. A hash keeps everyone on their own account by default. It's stateless, costs a few microseconds, and needs no rotation pointer to persist, and it only moves someone to the next account in the ring if theirs is full or refused for the day.
flowchart LR A[sendCode] --> B[codes account\nalways, never shared] C[sendAnnouncement] --> D{hash of recipient} D -->|picks home account| E[ResendPool] E -->|exhausted or refused?| F[next account in the ring] F -->|still exhausted?| F E -->|all exhausted| G[fails loud: BulkQuotaExhausted]A brand-new subdomain sending 95 emails on day one reads to Gmail like a stranger shouting, so each bulk account also ramps its own cap up from its first send: 20/day for 3 days, 40 through the first week, 70 through the second, then the full cap. That cap sits below Resend's real limit on purpose, so our count and theirs can drift without turning into a refusal:
That count lives in a mail_quota table, one row per account per UTC day, because a restart mid-evening would otherwise forget what had already gone out and walk an account straight into Resend's wall a second time. Resend's own refusal still wins over our count whenever it disagrees: the moment an account answers with a quota error, or with anything that won't clear today (a bad key, a domain Resend hasn't cleared for sending), it gets marked exhausted and skipped for the rest of the day, no matter what our counter says. The two-requests-a-second rate limit is the one exception, since that clears in about a second and retrying it once is cheaper than retiring a perfectly healthy account over a batch that went out slightly too fast.
Both switches (the campfire nudge and the re-engagement email I added at the same time) ship off, and the day's usage per account now lives on an admin dashboard card instead of a database row I'd have had to edit by hand to check.
What else came out of the same fix
Tightening one path exposed a second bug living next to it: an admin "notify everyone" button that marked every waitlist row as notified even when there was no bulk account configured to send through. It spent each person's one lifetime launch email on silence and reported success while doing it. Harmless as long as the codes account could still double as a fallback announcer, and live the moment that fallback was removed on purpose. It now refuses with a 503 instead of lying about what it sent.
What I'd do differently
I'd set myself a deadline for the real fix the same evening I shipped the quick one. "Off by default plus a cap" was the right call for that evening, and it also quietly became the permanent state for six weeks, because a feature that's off by default stops feeling urgent the moment it stops breaking things. Nothing forced me back to it; I just happened to need bulk mail for something else and ran into the same account limit again.
So: would you rather ship the conservative stopgap and trust yourself to come back to it, or hold the fix until the real architecture is ready, even if that means the feature stays off for longer than anyone wants? I went with the first one and got lucky that nothing else leaned on that account in the meantime. I'm less sure I'd make the same call on a path with more than one feature behind it.
