If you ship a free, no-signup AI web tool, a naive rate limit like "10 requests per minute per IP" almost always fails in both directions.

Real users on shared mobile carrier NATs or office Wi-Fi hit false-positive blocks after two clicks, while simple rotating-IP scripts burn through your token budget in minutes.

Here is a lightweight, practical rate-limiting pattern that protects LLM endpoints without ruining the first-time experience for genuine visitors.

1. Separate Burst Protection From Token Cost Budgets

Counting only requests treats a 40-character prompt the same as a 4,000-character prompt. Instead, enforce two lightweight checks in Redis (or an in-memory store for single-instance apps):

  • Short burst limit: Prevents accidental double-clicks and tight loops (for example, max 5 requests per 30 seconds).
  • Rolling cost/character budget: Tracks input + output budget per hour or day so normal short turns don't exhaust a user's quota too quickly. 
type LimitCheck = { allowed: boolean; retryAfterSec: number; }; type RedisCounter = { incr: (k: string) => Promise<number>; incrby: (k: string, n: number) => Promise<number>; expire: (k: string, s: number) => Promise<void>; ttl: (k: string) => Promise<number>; }; async function checkDualLimit( key: string, charCost: number, redis: RedisCounter ): Promise<LimitCheck> { const burstKey = `rl:burst:${key}`; const budgetKey = `rl:chars:${key}`; const burstCount = await redis.incr(burstKey); if (burstCount === 1) await redis.expire(burstKey, 30); if (burstCount > 5) { const ttl = Math.max(1, await redis.ttl(burstKey)); return { allowed: false, retryAfterSec: ttl }; } const totalChars = await redis.incrby(budgetKey, charCost); if (totalChars === charCost) await redis.expire(budgetKey, 3600); if (totalChars > 25000) { const ttl = Math.max(1, await redis.ttl(budgetKey)); return { allowed: false, retryAfterSec: ttl }; } return { allowed: true, retryAfterSec: 0 }; }

2. Don't Key Exclusively by Raw IPv4/IPv6

On mobile networks and dual-stack connections, a single user's IPv6 address can rotate across requests, while thousands of legitimate visitors can share one IPv4 exit node.

Two small adjustments reduce both abuse and false positives:

  • Normalize IPv6 to a /64 prefix instead of tracking the full /128 host address.
  • Pair IP subnet with an HttpOnly anonymous session token issued on first page load. Keep tighter per-IP limits as a backstop, and allow a slightly smoother per-session allowance so legitimate shared-network users aren't punished for someone else's traffic.

3. Run Cheap Pre-Flight Guards Before Touching the Model

Every rejected invalid request before your LLM call saves money and keeps your queue fast for real users:

  1. Trim & enforce hard length bounds early: Reject empty inputs and anything above your maxInputChars limit right in the route handler.
  2. Cache identical recent prompts: Hash (normalizedPrompt + mode + locale) for 60 seconds. If a user accidentally hits Submit twice—or refreshes immediately—serve the cached response without counting it as a second model call.
  3. Verify browser Origin / Sec-Fetch-Site headers on browser-only endpoints before hitting the rate-limiter or LLM client.

4. Return Structured 429 Payloads With Retry-After

A plain "Too Many Requests" string leaves users guessing whether the app is broken or permanently blocked. Return a standard Retry-After header plus structured JSON:

return Response.json( { error: "rate_limited", message: "You've sent several requests quickly. Please wait a moment and try again.", retryAfterSec: result.retryAfterSec, }, { status: 429, headers: { "Retry-After": String(result.retryAfterSec), }, } );

On the frontend, disable the submit button with a live countdown (Try again in 14s) instead of letting frustrated users hammer the button and extend their own cooldown.

Key Takeaway

Good rate limiting on public AI endpoints isn't about locking everything down with heavy CAPTCHAs on the first visit. Combining burst + character budgets, /64 IPv6 normalization, short prompt-hash deduplication, and clear 429 countdowns stops most accidental and automated waste while keeping the experience smooth for real visitors.

How are you protecting free LLM endpoints in production—token buckets, anonymous session cookies, or Edge middleware?