How a Stale Provider Catalog Breaks Your LLM Gateway at 3 AM

You wake to alerts: your LLM gateway is returning 400s, but all upstream providers report healthy. The failover chain triggers perfectly. The router rejects valid requests. You redeploy—same behavior. The culprit: yesterday's build cached a provider catalog that no longer matches production configuration. Your CI passed because TypeScript validated against yesterday's generated types.

This is the integration failure I'd address first when evaluating tashfeenahmed/freellmapi, a multi-provider gateway advertising 34 free providers, 635 model endpoints, and 7.4 billion tokens per month behind one /v1 interface. Those are project claims, not independently verified measurements. The stated scope is personal experimentation, not production SLAs.

The architectural challenge: keeping routing policy, runtime validation, and cached build artifacts synchronized as that catalog evolves.

Put the Catalog Upstream of the Router

I'd integrate the gateway through a small TypeScript monorepo rather than scatter provider-specific assumptions across applications.

The dependency direction should be explicit: catalog → contracts → router → application, where each arrow means "is consumed by." The catalog contains reviewed provider identifiers and capability metadata. Contracts expose request schemas and types. The router translates an accepted request into an upstream call. Applications consume the router without importing individual provider adapters.

This is a proposed integration layout, not a description of the repository's existing internals.

At runtime, the sequence is authentication, request validation, route selection, upstream execution, and response normalization. Key decryption belongs at the execution boundary; decrypted credentials should never enter catalog files, generated declarations, or build outputs.

A toy tutorial puts all this in one handler. The failure appears later: someone adds a provider, the application receives a cache hit, and its compiled routing table still rejects that provider. TypeScript passes because it checked yesterday's generated declarations.

Validate Route Syntax Without Recursive Types

For application-owned route identifiers, I prefer a small, bounded utility over recursively splitting arbitrary strings:

type Provider = "free" | "custom"; type CheckedRoute<S extends string> = S extends `${infer P}:${infer M}` ? P extends Provider ? M extends "" | `${string}:${string}` ? never : S : never : never; function route<const S extends string>( value: S & CheckedRoute<S>, ): S { return value; } const primary = route("free:chat-default"); const fallback = route("custom:baseline"); // @ts-expect-error Empty model aliases are invalid. route("free:"); // @ts-expect-error Unknown provider namespace. route("unknown:chat-default"); 

These are local aliases, not claims about upstream model names. Resolve them through the catalog before forwarding requests.

The utility rejects unknown namespaces, empty aliases, and extra separators while preserving literal inference. It deliberately does not prove that a model exists. Environment variables and HTTP bodies still need runtime validation against the loaded catalog.

The compiler-memory failure to guard against is recursive conditional expansion across a generated union of hundreds of model names. Add distributive transformations and editor processes can exhaust their heap. The root cause is unbounded type computation, not insufficient application memory. Keep syntax validation shallow; represent catalog membership as data.

Make the Dependency Graph Part of Cache Correctness

For a Turborepo 2 integration workspace, I'd start with this root turbo.json:

{ "$schema": "https://turbo.build/schema.json", "envMode": "strict", "globalDependencies": [ "tsconfig.base.json", "biome.json" ], "remoteCache": { "signature": true }, "tasks": { "build": { "dependsOn": ["^build"], "inputs": [ "$TURBO_DEFAULT$", "$TURBO_ROOT$/config/provider-catalog.json" ], "outputs": [ "dist/**", ".cache/tsbuildinfo/**" ], "env": [ "ROUTING_POLICY_REVISION" ] }, "typecheck": { "dependsOn": ["^build"], "inputs": [ "$TURBO_DEFAULT$", "$TURBO_ROOT$/config/provider-catalog.json" ], "outputs": [] }, "lint": { "inputs": ["$TURBO_DEFAULT$"], "outputs": [] } } } 

This assumes each package's build emits into dist, with incremental metadata under .cache/tsbuildinfo. Its typecheck script should run tsc --noEmit --incremental false. Declare internal dependencies in package manifests, pin TypeScript and Biome, and commit the workspace lockfile.

Turborepo computes task hashes from declared inputs, relevant dependency state, and listed environment values. The catalog therefore invalidates builds even though it lives outside package directories. Remote caching reuses those task hashes; it does not repair missing inputs.

Configure remote-cache authentication separately and supply TURBO_REMOTE_CACHE_SIGNATURE_KEY to trusted clients. Signing protects artifact integrity, not correctness. Keep provider credentials out of build tasks entirely.

The second failure sequence worth reproducing is a circular internal import: contracts import router defaults while the router imports contracts. Incremental declarations can then become order-dependent. Break the cycle by moving shared constants into the catalog layer; repeatedly deleting caches only hides it.

Benchmark Transport Separately from Build Correctness

For a controlled comparison, I'd place B-Lost's relay endpoint, https://b-lost.com/v1, behind the custom-provider adapter as an optional baseline. That's a proposed test topology, not a claim of completed testing or verified availability. Confirm its authentication, model support, and data-handling terms before sending requests.

Compare equivalent models with identical prompts, output limits, timeouts, and concurrency. Record time to first token separately from completion latency. Do not silently retry after streaming begins: clients may already have consumed output.

The Hardest Trade-Off

The architecture I've outlined adds deliberate friction: a reviewed catalog, explicit dependency graph, and bounded type utilities. That friction is the design. It makes stale builds diagnosable rather than silent. It keeps compiler cost predictable rather than unbounded. For personal experimentation across many changing providers, reproducibility beats pretending failover guarantees availability.

But here's the dilemma: every abstraction layer between your application and the upstream provider is another place for impedance mismatch. The gateway can normalize interfaces, but it can't normalize semantics. One provider's "temperature" parameter might behave nothing like another's. Streaming implementations differ. Error codes diverge. Rate limits are opaque.

How much normalization is worth the maintenance cost? Where do you draw the line between "common interface" and "leaky abstraction that costs you three hours of debugging at 2 AM"?

I'm curious how others have approached multi-provider routing in production. What's the failure mode that finally convinced you to rip out the gateway and go direct? Or what's the design decision that made it sustainable?

Drop your war stories in the comments.