You fire 50 LLM calls at once. A batch job, a fan out over documents, a busy hour. The provider only lets 10 through at a time, so the rest come back as 429 Too Many Requests.
The usual fix is to retry 429s with backoff. That helps, but notice what it means: every retry is a request you sent after you already got rate limited. You are reacting to a limit you could have stayed under.
This post shows the difference with real numbers and how to rate limit LLM calls in TypeScript with VernLLM, before the provider says no.
Retrying 429s vs not causing them
I ran a 50 call burst against a scripted provider that accepts 10 requests in flight and rejects the rest with 429. Both runs used maxRetries: 2.
| Retries only | rateLimit: { maxConcurrent: 10 } | |
|---|---|---|
| Calls answered | 20 | 50 |
| Calls failed | 30 | 0 |
| Provider 429s | 110 | 0 |
| Requests sent | 130 | 50 |
Retries alone sent 130 requests, collected 110 rejections, and still lost 30 calls. With a local limit, every call waited its turn and went out once.
Rate limiting LLM calls with VernLLM
npm i vern-llm openai import OpenAI from 'openai'; import { VernLLM } from 'vern-llm'; import { fromOpenAI } from 'vern-llm/adapters'; const llm = new VernLLM({ client: fromOpenAI(new OpenAI({ apiKey: process.env.OPENAI_API_KEY })), model: 'gpt-6.2-sol', rateLimit: { requestsPerMinute: 500, tokensPerMinute: 200_000, maxConcurrent: 20, }, }); const results = await Promise.all(tickets.map((t) => llm.call({ userContent: t.text }))); rateLimit has three buckets, each optional:
- requestsPerMinute caps how many requests leave per minute. It refills continuously, so it does not reset all at once every 60 seconds.
- tokensPerMinute caps token throughput. It checks an estimate before the request goes out, then corrects it with the real usage the provider reports.
- maxConcurrent caps requests in flight, and frees a slot the moment one finishes.
A call goes out only when every configured bucket has room. Otherwise it waits.
A fair queue
Calls that cannot go out yet wait in strict first in, first out order. A large call is never starved by smaller ones behind it.
You decide how long a call may wait and how deep the queue may get:
rateLimit: { requestsPerMinute: 60, maxQueueMs: 10_000, // default 30000 maxQueueSize: 50, // default unbounded }, When a call gives up waiting, it fails with a rate_limited error and a code you can branch on (rate_limit_queue_timeout or rate_limit_queue_full). These never reached the provider, so VernLLM does not retry them. A call too big to ever fit under tokensPerMinute fails right away with rate_limit_capacity_exceeded instead of blocking the queue forever.
Retries count too
Capacity is taken per attempt, not per call. A retry is a real request against the same provider limit, so it waits in the same queue as everything else. That is what keeps retries from turning one 429 into a storm of them.
Adaptive limits: AIMD
You often do not know the provider's real limit, or it changes with your tier and their load. aimd lets the requests ceiling adjust itself: it grows a little on every success and shrinks on a real 429.
rateLimit: { requestsPerMinute: 500, aimd: { increaseBy: 10, decreaseFactor: 0.5, minCapacity: 50, maxCapacity: 1000 }, }, It shrinks at most once per 60 seconds, so a burst of 429s from calls already in flight counts as one overload. With proactiveFloor, adapters that can read rate limit headers shrink the ceiling when the provider reports low remaining capacity, before a 429 happens.
A 429 that still gets through keeps its own code, provider_rate_limited, so you can tell a real provider limit apart from VernLLM's own queue in your logs.
One limit per provider
With fallback, each target gets its own limiter. A burst against a struggling primary cannot use up the slots a healthy backup needs:
const llm = new VernLLM({ client: fromOpenAI(openai), model: 'gpt-6.2-sol', rateLimit: { maxConcurrent: 50 }, fallback: { client: fromGemini(gemini), model: 'gemini-4-neon', rateLimit: { maxConcurrent: 10 }, }, }); Scaling past one process
The built in limiter lives in one process. If you run several instances, pass your own RateLimiterAdapter, backed by Redis or anything else, so every instance shares one budget against the provider.
Wrap up
Retrying 429s is damage control. A local limit keeps you under the provider's ceiling in the first place: queue in order, count every retry, adapt when the provider pushes back.
npm i vern-llm Docs: Rate limiting

