Shipping a three-repo security platform on free tiers: what broke and why

I built Sentinel, a small self-hostable security monitoring platform, as three separate repos:

• sentinel-api: Node 22, TypeScript, Express 5, PostgreSQL, Prisma 7. It ingests security events and scan results, raises alerts from threshold rules, tracks incidents and streams alerts live over SSE.
• sentinel-dashboard: Next.js. Live alert feed, events explorer, incident timeline, scan reports, admin screens.
• sentinel-scanner: a Python CLI that scans a site you own, checks dependencies, analyses logs and writes incident post-mortems, then sends results to the API.

Everything worked on my laptop with Docker Compose. Putting it online for free (Neon for Postgres, Render for the API, Vercel for the dashboard) is where the interesting bugs were. This is the account of those, in the order I hit them.

The design decision that shaped the deployment

The dashboard keeps the short-lived access token in memory only. The long-lived refresh token lives in an httpOnly; Secure; SameSite=Strict cookie set by the API. That protects against token theft by XSS and against CSRF.

SameSite=Strict has a consequence. my-app.vercel.app and my-api.onrender.com are different sites (both are on the public suffix list), so a browser would never send the refresh cookie from one to the other. On localhost, ports don't count, so everything worked. Hosted, I would have been logged out on every reload.

The tempting fix is SameSite=None. I didn't, because it removes the protection the cookie was chosen for. Instead the dashboard runs as a same-origin proxy: the browser only talks to the dashboard, and Next.js rewrites /api/v1/* to the API. No CORS, no cross-site cookie.

Bug 1: "Cannot reach the Sentinel API", and no network request

After deploying, sign-in showed "Cannot reach the Sentinel API. Check your connection." The Network tab was empty, and the same login worked from PowerShell, curl and the browser console.

I chased the wrong things first: CORS, the proxy, cold starts, browser extensions. The clues that finally mattered:

1. The error appeared instantly, with no request.
2. The message is only shown when fetch() itself throws; every real server response maps to a specific message.
3. Only the silent session-refresh call worked, and it was the only one using fetch directly.

The cause was one line: new URL(${API_BASE}${path}). In proxy mode API_BASE is the relative string /api/v1, and new URL throws Invalid URL on a relative string with no base. Every API call threw before it was sent. The fix is the origin as a base: new URL(path, location.origin), which is ignored when the base is already absolute.

Two lessons:

• My proxy change had been "verified", but against a stub upstream and a built server, which never exercised the browser's real call path. A regression test that uses a relative base failed with Invalid URL before the fix and passed after, which is the test I should have written first.
• A catch-all error message hides the real error. "Cannot reach the API" was true for none of the actual causes. Distinguish "the request failed" from "the code threw before sending".

Bug 2: the password I set did not work

I seeded the admin on Neon from my own machine, then couldn't sign in. The seed script is an upsert with update: {}, so re-running it never overwrites an existing admin's password. That is deliberate (a re-run must not reset a real admin), but it means "just seed again" is not a reset.

I wrote a small one-off script that hashes with the API's own hashPassword, updates the user, and revokes their refresh tokens so old sessions don't survive. Running it from PowerShell also taught me to use Read-Host for secrets (it keeps them out of shell history) and that Invoke-RestMethod against the API directly is the fastest way to tell "the API is wrong" from "the dashboard is wrong".

Bug 3: every user looked like Cloudflare

After signing in, the audit log showed client IPs in 172.68.x, 104.22.x, 162.158.x, 141.101.x: all Cloudflare edge addresses, not mine. Express's trust proxy setting says how many proxy hops to trust when reading X-Forwarded-For, and I had set it one hop short for the chain browser, Vercel, Cloudflare, Render's load balancer, app.

This is not cosmetic. Rate limits and the audit log both key on the client IP, so with the wrong value every visitor behind the same edge node shares one login-rate-limit bucket.

The guide I had written said "adjust it if the audit log shows a Render or Vercel address", so I used the audit log as the instrument. I sent a login with fake credentials from my machine (public IP known) and read which address the API recorded for it at each setting:

At 4 the dashboard path was right. Then I tried the other path: calling the API's URL directly with a forged X-Forwarded-For: 198.51.100.77 and fake credentials. The audit log recorded 198.51.100.77. Express's own docs describe exactly this: a numeric trust proxy is only safe when there are no "multiple, different-length paths to the Express application". Mine had two (through the dashboard proxy, or direct, as the scanner does), and one number cannot be right for both. A forged IP means the IP-based login rate limit can be dodged by rotating the header.

How people usually handle it

I read around before choosing a fix. The recurring advice:

• Never use trust proxy: true on a public server, because it takes the left-most (client-supplied) entry. express-rate-limit even logs a warning for it.
• Trust by address, not by count, where you can. That works for Cloudflare (it publishes its ranges) but not for Vercel's outbound addresses, which are not fixed.
• Make the edge overwrite the header. Vercel does overwrite X-Forwarded-For with the real visitor IP at its edge, and its docs describe a custom header carrying the real client IP as the pattern for a proxy in front of Vercel.
• Don't rely on IP alone for login limits: add a per-account limit too, with the caveat that lockouts can be abused to lock real users out.

What I did

The proxy vouches for the visitor. The dashboard's middleware adds X-Sentinel-Client-IP (read from x-vercel-forwarded-for) plus X-Sentinel-Proxy-Secret to /api/v1/* requests and strips any such headers a visitor sent. The API only believes the IP header when the secret matches (constant-time compare). TRUST_PROXY then only has to count the hops every route shares. It is a small amount of code in each repo, with unit tests, and I checked the dashboard half through a production build and a rewrite against a stub that echoed the headers it received.

Result, on the live deployment (TRUST_PROXY=3, the same secret on Render and Vercel, fake credentials from a machine with a known public IP): six probes, and every audit-log entry showed my real IP.

The first and last rows matter most. At TRUST_PROXY=3 the dashboard route would record Vercel's outbound address without the vouching header (that is what I measured earlier), so a correct IP there proves the header is doing the work.

The second layer: a per-account limit, without handing attackers a lockout button

With a trustworthy IP, the existing limit (10 attempts per IP and email per 15 minutes) is solid against one machine, but an attacker with many real IPs gets a fresh budget on each. OWASP's advice is to keep the failure counter on the account, not the IP. It also warns that a counter that blocks an account lets anyone who knows the email lock the owner out, and suggests exponential delays or a CAPTCHA instead of a flat lock. Auth0's default is the narrow one: block only the offending IP for that account, and make account-wide lockout opt-in.

What I built is in between, and the last piece is my own idea rather than something I found written down, so I tested it hardest:

• A counter keyed on the email alone, across all IPs: 30 failed logins per 15 minutes.
• Only failures count. A successful login gives its slot back, so normal use never consumes the budget.
• Once the budget is spent, an attempt is still allowed if that IP has already logged in successfully to that account in the last 30 days (read from the audit log, and only queried when someone is over the limit). Everyone else gets a 429. So an attacker spraying from unfamiliar IPs is stopped, and the owner at their usual desk is not.

On the live deployment, twelve failed logins for a made-up email showed the new third counter (30-in-15min) going 29, 28, 27 and so on, while the per-IP-and-email counter (10) reached zero first and produced the 429 on the 11th attempt, at which point the account counter stopped moving. The cross-IP block and the known-IP exemption can't be exercised from one machine, so those are covered by integration tests only.

It isn't a perfect answer. An owner on a new network during an attack waits out the window, and a CAPTCHA or an unlock-by-email flow would be the real fix. I wrote that into the threat model as a known limitation, not a footnote.

Smaller things worth knowing

• Neon: use the direct connection string, not the pooled one. Prisma migrations take advisory locks that connection poolers can break. The pooled host has -pooler in its name. If the string has channel_binding=require, remove it.
• Don't use Render's free Postgres for anything you want to keep; it expires.
• Free-tier cold starts: Render's free web service sleeps after 15 minutes idle and takes about a minute to wake, so I ping the health endpoint from an uptime monitor.
• Health checks and rate limits: Render health-checks every few seconds. I checked that this wasn't consuming visitors' rate-limit allowance (it uses a separate bucket).

Screenshots without leaking anything

For the README screenshots I seeded demo data tagged metadata.demo, using documentation-only IP ranges (RFC 5737) and reserved *.demo.example domains, with a reset command that removes only the demo rows. For the CLI I captured real runs as SVG using Rich's Console.save_svg, which GitHub renders crisply. The first capture of the post-mortem command printed my local Windows path, including my username into the image, so I scanned every SVG for personal paths before committing. Worth doing for any terminal screenshot.

Cleaning up history

Early commits carried AI co-author trailers that I wanted gone. Removing them meant rewriting history on a protected main in all three repos: I rewrote fresh clones with git filter-branch --msg-filter, verified the trees were byte-identical and the commit counts unchanged, pushed backup branches, then force-pushed with --force-with-lease pinned to the SHA I had verified. Branch protection had to be relaxed temporarily by me; a tool should not quietly change your security settings.

What I'd tell myself at the start

1. Test the deployed shape, not just the app. My proxy mode passed unit tests and a stub; it failed on the first real click.
2. Write the deploy guide as you deploy. Mine said the hosted steps had "not been run end to end", and the first real run found two problems.
3. Make error messages specific. Three hours went into a message that was wrong.
4. Know your proxy chain. trust proxy is a number you can get wrong silently.
5. Treat secrets as shell-only. Environment variables for one-off scripts, prompted not typed, never in chat or git.

All three repos are on GitHub if you want to read the code or the deployment guide: sentinel-api/docs/DEPLOY.md.