Fifteen outside services can be down, and our code has five different answers

Nakodo is an outreach app for brands. Counting generously, fifteen things outside our own code have to be working for all of it to work, and on any given day one of them is not.

I went looking for the policy. There is no document, no circuit-breaker library, no resilience.ts. What there is instead is 139 catch blocks and 43 .catch() calls in non-test code, each one written in a hurry by me, at the moment I was holding the problem. This post is the audit of all of them, grouped by what they actually do.

The short version: there are five answers in the codebase, two of them are exact opposites, and one dependency has no answer at all.

You can see one of the five in the served HTML without signing up for anything. This is the sign-up form:

$ curl -s "https://nakodo.app/login?mode=signup" | grep -o "Checking you're not a bot" Checking you're not a bot 

That string is the label on a disabled button. The form will not submit until Cloudflare has answered, and if Cloudflare never answers, it never submits. Hold that thought, because the rate limiter two files away does the exact opposite.

The five answers

Five policies: let it through, turn it away, come back later, say so, and pretend it did not happen. Postgres gets a sixth, which is to give up honestly.

The two opposites

Both of these are a try with one catch. The difference is the return value.

export async function rateLimit(rule: RateLimitRule, ...keys: (string | null | undefined)[]): Promise<void> { const l = limiter(rule); if (!l) return; let results: Awaited<ReturnType<Ratelimit["limit"]>>[]; try { results = await Promise.all(keys.filter((k): k is string => !!k).map((k) => l.limit(hash(k.toLowerCase())))); } catch (e) { console.error(`Rate limit check failed (${rule}); letting the request through`, e); return; } ... } 

export async function passedTurnstile(token: FormDataEntryValue | null): Promise<boolean> { ... } catch (e) { console.error("Turnstile check failed", e); return false; } ... } 

The rate limiter protects us from volume. Eighteen rules, from five password attempts in fifteen minutes to 300 category lookups a minute as somebody types. Every one of them is a guess at a number that is comfortably above normal use, so the cost of not enforcing one for a few minutes is that a handful of people could have gone faster than I would like. The cost of enforcing it when Redis is unreachable is that sign-in stops working for everybody. There is a whole post on that limiter, and the fail-open decision is the only interesting thing in it.

Turnstile protects us from automation, and automation is what shows up when a check is weak. A sign-up that succeeds without the check is an account, and accounts cost us money the moment they launch a campaign. So an unreachable Cloudflare means the form says to try again.

It is the same shape of outage with the opposite answer, and the reason is not technical. It is which direction the loss runs.

A fail-open written as a race

This is my favourite line in the audit, because the policy is not in a catch at all:

// A slow lookup gets the benefit of the doubt. function acceptsMail(domain: string): Promise<boolean> { let check = mx.get(domain); if (!check) { if (mx.size > 5000) mx.clear(); check = Promise.race([ dns.resolveMx(domain).then( (r) => r.length > 0, (e: NodeJS.ErrnoException) => !["ENOTFOUND", "ENODATA", "ESERVFAIL"].includes(e.code ?? ""), ), new Promise<boolean>((r) => setTimeout(() => r(true), 5000)), ]); } ... } 

Two fail-opens in six lines. The rejection handler says that a DNS error is only a "no" if it is one of three codes that genuinely mean this domain has no mail; anything else, including a timeout from the resolver, is a "yes". And the race against a five second timer resolves true, so a slow nameserver is a pass rather than a wait.

The question this answers is "does this domain accept mail at all". Getting it wrong in the strict direction means silently dropping a real address, which nobody would ever notice. Getting it wrong in the loose direction means one email bounces, which we record anyway.

The inbound spam check does the same thing with a sentence I am glad I wrote down at the time:

 } catch (e) { console.error("inbound spam check: AI check failed, forwarding", e); return false; } 

A broken check forwards the mail. The alternative is a check that quietly eats a customer's reply.

Come back later

Anything running in the background gets a third answer, because it has something the request path does not: time.

 } catch (e) { if (opts.rethrow?.(e)) throw e; const until = e instanceof DeferJob ? e.until : opts.deferUntil?.(e); if (until) { await deferJobs([job.id], until); return { outcome: "deferred", error: e }; } const outcome = await failJob(job, e); // Its own failure (say an email that can't go out while Resend is down) // mustn't take the rest of the batch with it. if (outcome === "failed") { await opts.onGiveUp?.(job).catch((err) => console.error(`Giving up on the ${job.type} job failed`, job.id, err)); } return { outcome, error: e }; } 

Two different failures, two different words for them. A deferral is "not now", and it names a time: the next quota reset, the next hour, the end of a wait budget. A failure is "that did not work", and it counts against three attempts before the job is given up on.

The distinction matters because a deferral is not an error anywhere in the system. It does not count attempts, it does not log, it does not alert. That is right for a daily quota and it was wrong for a prepaid source running out of credit, which looked exactly like a daily quota for nine days.

Say so

The two paths where a person is sitting there waiting both tell the truth.

 } catch (e) { throw e instanceof GeocodeError ? e : new GeocodeError(errorMessage(e)); } 

GeocodeError is one of the error classes that inherits the right to be shown to a user, so a place lookup that fails says it failed, in the form, next to the box. There is nothing clever to do instead. The person typing "Leeds" knows what they meant and we do not.

The ten AI tasks are split by the same test, and the comments in that file are all about who is waiting:

 // A support answer is read by someone watching a widget, so it gets one try // and a short leash: past this, a person is the better answer anyway. support: { timeoutMs: 20_000, maxRetries: 0 }, 

Twelve AbortSignal.timeout calls across the app, one of them the shared fetch timeout, and the values are not a scale of how slow the service is. They are a scale of how long somebody will sit there.

Pretend it did not happen

Three services are allowed to fail invisibly, and the test is whether the user's own work depends on the result.

export function track(userId: string, event: string, properties: Record<string, unknown> = {}): void { if (!POSTHOG_TOKEN || !isLiveDeployment()) return; const send = () => capture(userId, event, properties, new Date()); try { after(send); } catch { // Outside a request (a script): send it now. void send(); } } 

Analytics goes out in after(), so it is not in the response at all, and a refused event is one console.error. Recording what an AI call cost us is the same: await recordUsage(task, usage).catch((e) => console.error(...)), with a comment saying the call is paid for whether or not we manage to write it down. A logo lookup is the same again.

The honest reading is that these are the places where the failure is ours to notice, and the only reason it is acceptable is that the log is read.

The one with no answer

Grep the codebase for a retry around a query, a fallback read, or a cached last-known-good page. There is nothing. The database is the app.

// Errors are plain: no illustration, no joke. export default function AppError({ error, retry }: { error: Error & { digest?: string }; retry: () => void }) { ... <EmptyState title="This page didn't load." actions={<Button variant="outline" onClick={() => retry()}>Try again</Button>}> Your campaigns and results are saved. If it keeps happening, try again in a few minutes. {error.digest && <span className="mt-2 block font-mono text-xs">Reference {error.digest}</span>} </EmptyState> ... } 

One sentence of that is load-bearing: "Your campaigns and results are saved." It is the only thing a person actually wants to know when a page fails, and it is true for exactly the reason the page failed, which is that everything lives in Postgres and nothing important lives in the request.

Server Actions get the same treatment from the other end: expected failures come back as values and only bugs reach the boundary, which is a separate post.

What the audit changed

Two things I would not have found by reading one file at a time.

A failed session claims read is indistinguishable from being signed out. getUser returns null if the claims are missing for any reason, and requireUser redirects to the login page. For a genuine auth outage that means the app bounces people to a sign-in form that will also not work, rather than saying what is happening. Nobody has seen it; the shape of it is in the code.

And the fail-open paths all point the same way: the rate limiter lets requests through, DNS assumes mail is accepted, the spam check forwards, a dead website is treated as an empty one. Every one of them increases what gets through. The only two fail-closed paths in the app are the two where getting it wrong creates an account or moves money. That is a defensible rule and it was never written down, so here it is in one line: when an outside service cannot answer, do the thing that costs us rather than the thing that costs the customer, unless the cost is an account or a charge.

You can read the promises those services sit behind on the privacy page, and the plan limits the rate limiter is protecting on the pricing page. The interesting half of a dependency list is not who you call. It is what you have decided to do about each of them at two in the morning.