At 3:47 AM, your Slack bot pings: "OpenAI bill: $1,847 this month." Last you checked, the team's AI enrichment budget was $200. Somewhere between a retry storm and a developer's local experiment, every process hammered the same unrestricted provider key. The job summarization feature that was supposed to cost $50 just burned nine months of runway.

This is the failure mode I'm engineering around: a small team integrating AI-assisted summaries into Freehire, an open-source job search engine, without giving every engineer a loaded financial gun. The solution is a single enforceable boundary—no enrichment request reaches the upstream provider before its maximum permitted cost has been reserved.

I'm evaluating Freehire as an external integrator proposing a separate enrichment gateway, not describing verified Freehire internals. The architecture goal is budget enforcement at the request level, not application-layer authentication.

Authorization Sits Beside the Ledger

Keep job ingestion, search, and any existing authentication separate from this gateway. An enrichment worker sends messages using a seat-specific credential. The gateway authenticates that credential, reserves budget in a transaction, and forwards an allowlisted request using the upstream key.

For five to twenty engineers, a single-host service with SQLite is defensible. An immediate transaction serializes budget decisions across local worker processes. An in-memory counter would not survive restarts or concurrent writers.

This implementation charges reservations upfront rather than refunding after completion. That sacrifices utilization but avoids a dangerous ambiguity: a connection timeout does not prove the upstream request was never billed.

Single Executable Gateway

Install fastapi, uvicorn, and httpx. Configure UPSTREAM_KEY, MODEL, SEAT_KEYS_JSON, and CEILING_CENTS_PER_MTOK. The ceiling must cover the highest input or output price for the selected model—a $20-per-million-token ceiling is 2000 cents.

SEAT_KEYS_JSON maps gateway secrets to seat names: {"random-secret":"alice"}. Run with uvicorn gateway:app --host 127.0.0.1 --port 8080 --no-access-log behind authenticated TLS termination.

import datetime as dt import json import os import sqlite3 import httpx from fastapi import FastAPI, Header, HTTPException from pydantic import BaseModel, Field app = FastAPI() BASE_URL = "https://b-lost.com/v1" MODEL = os.environ["MODEL"] KEY = os.environ["UPSTREAM_KEY"] SEATS = json.loads(os.environ["SEAT_KEYS_JSON"]) PRICE = int(os.environ["CEILING_CENTS_PER_MTOK"]) assert PRICE > 0 DB = os.getenv("BUDGET_DB", "budget.sqlite") WEBHOOK = os.getenv("BUDGET_WEBHOOK") def connect(): return sqlite3.connect(DB, timeout=30) with connect() as db: db.execute("""CREATE TABLE IF NOT EXISTS reservations ( month TEXT, day TEXT, seat TEXT, tokens INTEGER, cents INTEGER )""") db.execute("""CREATE INDEX IF NOT EXISTS reservations_month ON reservations(month)""") class Request(BaseModel): messages: list[dict[str, str]] max_tokens: int = Field(default=512, ge=1, le=2048) def reserve(seat, tokens): now = dt.datetime.now(dt.timezone.utc) day, month = now.strftime("%Y-%m-%d"), now.strftime("%Y-%m") cents = (tokens * PRICE + 999999) // 1000000 with connect() as db: db.execute("BEGIN IMMEDIATE") seat_tokens, seat_cents = db.execute( """SELECT COALESCE(SUM(tokens),0), COALESCE(SUM(cents),0) FROM reservations WHERE month=? AND seat=?""", (month, seat), ).fetchone() team_cents = db.execute( "SELECT COALESCE(SUM(cents),0) FROM reservations WHERE month=?", (month,), ).fetchone()[0] daily = db.execute( "SELECT COALESCE(SUM(cents),0) FROM reservations WHERE day=?", (day,), ).fetchone()[0] if (seat_tokens + tokens > 200000 or seat_cents + cents > 2000 or team_cents + cents > 10000): raise HTTPException(429, "Monthly reservation limit reached") db.execute("INSERT INTO reservations VALUES (?,?,?,?,?)", (month, day, seat, tokens, cents)) return daily < 5000 <= daily + cents @app.post("/v1/chat/completions") def complete(body: Request, authorization: str = Header(default="")): seat = SEATS.get(authorization.removeprefix("Bearer ")) if not seat: raise HTTPException(401, "Unknown gateway credential") if not 1 <= len(body.messages) <= 32: raise HTTPException(400, "Invalid message count") size = 0 for message in body.messages: if (set(message) != {"role", "content"} or message["role"] not in {"system", "user", "assistant"}): raise HTTPException(400, "Only plain-text messages are supported") size += len(message["content"].encode("utf-8")) if size > 24000: raise HTTPException(413, "Prompt exceeds gateway limit") # Includes an explicit, model-dependent message-framing allowance.  tokens = size + 128 * len(body.messages) + body.max_tokens alert = reserve(seat, tokens) if alert and WEBHOOK: try: httpx.post(WEBHOOK, json={"text": "Daily reservations reached $50"}, timeout=3).raise_for_status() except httpx.HTTPError: print("Budget webhook failed; inspect daily reservations", flush=True) try: response = httpx.post( f"{BASE_URL}/chat/completions", headers={"Authorization": f"Bearer {KEY}"}, json={"model": MODEL, "messages": body.messages, "max_tokens": body.max_tokens, "stream": False}, timeout=60, ) response.raise_for_status() return response.json() except (httpx.HTTPError, ValueError): raise HTTPException(502, "Upstream completion failed")

B-Lost's relay endpoint is the configured upstream in this deployment. That choice establishes neither a performance claim nor a retention guarantee.

What These Limits Guarantee

Each seat gets 200,000 tokens and $20 monthly reservation cap. The team gets $100 monthly. Crossing $50 daily attempts a webhook notification.

The input estimate uses UTF-8 bytes plus framing headroom. This is a model-adapter assumption, not a tokenizer contract. Before deployment, validate the bound against the selected model's tokenizer and message serialization. Models with hidden reasoning tokens, different output semantics, tools, or ancillary fees need a different adapter—this endpoint intentionally excludes those capabilities.

Monetary ceiling accuracy depends on correct pricing and upstream enforcement of max_tokens. Integer-cent rounding and conservative reservations reject requests earlier than billing would. That is intentional.

Where Production Diverges from the Happy Path

A timeout retains its reservation. An HTTP rejection does too. Automatic refunds would improve utilization but require evidence distinguishing unbilled failures from completed requests whose responses vanished. Blind client retries consume another reservation.

SQLite belongs on persistent local storage, not ephemeral container filesystems or network volumes. Database failure must fail closed. Keep the upstream key inaccessible to workers—otherwise the gateway is optional and caps are meaningless.

The webhook is best-effort, not durable delivery. A process crash after committing the threshold-crossing reservation loses the notification. Use a transactional outbox if alerts must survive that failure; budget enforcement itself remains committed.

The ledger stores no prompts or responses, but that alone is not zero-retention processing. Disable body capture in ingress, tracing, and error reporting. Confirm retention terms for both relay and downstream provider before forwarding private code or candidate data. A proxy cannot manufacture that guarantee.

The Retry Budget Dilemma

Here's the operational trade-off I have not solved cleanly: a transient upstream 502 consumes a reservation, the worker retries with exponential backoff, and five attempts burn five reservations with zero completions. Refunding on specific error codes assumes those errors are never billed—an assumption I cannot verify without provider documentation that does not exist for every model or relay.

The alternative is client-side idempotency tokens and deduplication in the gateway, but that requires request identity semantics the worker does not currently expose. How are you handling retry budget exhaustion in gateways with pessimistic reservation? Are you refunding on connection timeouts, betting the upstream never charged you?