Short answer: treat a rate-limit response as a scheduling signal, not as a failed candidate decision. Put requests behind a bounded queue, retry with exponential backoff while honoring Retry-After, record the response headers and request outcome, and cache a moderation result when the input and policy version are unchanged. Keep the scoring contract separate from the retry mechanism so a delayed request cannot be mistaken for a malformed structured result.

The queue is part of the product behavior.

That distinction matters in a B2B SaaS hiring workflow. A user submits a candidate profile, the service scores it against a job rubric, and the same pipeline may moderate free-form text before displaying the result. The on-call page should tell us that the upstream budget is being consumed faster than expected. It should not page because one candidate received a temporary 429.

What does the on-call actually see?

The first useful signal is usually a page about an SLO burn: the candidate-scoring endpoint has crossed its latency budget, or its successful-decision ratio has fallen below the agreed target. The application log shows a cluster of 429 responses from the moderation or model-compatible endpoint, followed by retries that arrive in the same narrow time window. Some requests eventually succeed; others hit the request deadline and surface as generic 500s.

That page fires late. The earlier signal was request pressure: queue depth rising, rate-limit headers shrinking, and the ratio of accepted upstream calls to attempted calls changing over a five-minute window. I want those measurements at the boundary where the upstream response is still available, because a later generic error has already thrown away the evidence needed to choose a retry delay.

Consider a batch of 500 candidate profiles arriving after a recruiting team changes a rubric. If every profile immediately starts a moderation call and a scoring call, the burst doubles before any model output is useful. A bounded queue lets the service admit work at a known rate, while a tenant-aware scheduler keeps an interactive profile review from waiting behind the batch. Each item carries the rubric version and input digest, so a retry does not accidentally publish an answer for the old rubric. When a 429 includes Retry-After, the worker schedules the item after that delay rather than sleeping inside a scarce request slot. When the header is absent, exponential backoff with jitter spreads the next attempts. The queue age is then compared with the decision SLO, and the system can mark a delayed result explicitly instead of returning a misleading empty score. This is more operationally useful than counting retries alone because it connects upstream pressure to the customer-visible decision deadline.

The instrumentation change is small but consequential. For every upstream call, record status class, attempt number, queue wait, total duration, and a redacted request identifier. Parse Retry-After when present, and retain the relevant rate-limit headers as metrics rather than copying arbitrary header values into logs. Track usage by tenant and by workflow too; a single bulk import should not hide the fact that interactive scoring is starving behind it.

Then page on sustained impact: queue age threatening the candidate-decision SLO, retry exhaustion, or a rising invalid-structured-output rate. Do not page on one 429. A threshold that is too low turns a recoverable burst into alert fatigue; one that is too high lets the queue become a second database of unbounded work.

How should an OpenAI moderation API rate-limit error protect the decision contract?

Start with classification. A 429 is a capacity signal and is normally retryable. A network timeout may be retryable if the operation is safe and the client cannot know whether the upstream accepted it. A 4xx validation error is usually a contract problem, not a reason to retry. A 5xx may be retryable, but only within a deadline and with an idempotency strategy appropriate to the operation.

For candidate scoring, the response must still validate against the expected schema after a successful retry. “The model answered” is not the same as “the scoring decision is usable.” Store the raw upstream status separately from the application outcome, and make the final state explicit: accepted, rejected as invalid, delayed, or failed after retry exhaustion. This prevents a moderation result, a score, and a transport error from collapsing into one ambiguous boolean.

Here is a compact Go policy for a request worker. It uses jitter so several workers do not wake together, honors a server-provided delay, and stops at a deadline. The worker still needs the caller to validate the returned structured document before publishing a decision.

package main import ( "context" "encoding/json" "errors" "fmt" "math/rand" "net/http" "os" "strconv" "strings" "time" ) func retryDelay(resp *http.Response, attempt int, rng *rand.Rand) time.Duration { if resp != nil { if value := resp.Header.Get("Retry-After"); value != "" { if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 { return time.Duration(seconds) * time.Second } } } base := 250 * time.Millisecond delay := base * time.Duration(1<<attempt) if delay > 8*time.Second { delay = 8 * time.Second } return delay/2 + time.Duration(rng.Int63n(int64(delay/2)+1)) } func doWithRetry(ctx context.Context, client *http.Client, request func() (*http.Response, error)) (*http.Response, error) { rng := rand.New(rand.NewSource(time.Now().UnixNano())) var lastErr error for attempt := 0; attempt < 5; attempt++ { resp, err := request() if err == nil && resp.StatusCode < http.StatusInternalServerError && resp.StatusCode != http.StatusTooManyRequests { return resp, nil } if err != nil { lastErr = err } else { lastErr = fmt.Errorf("upstream status: %s", resp.Status) } if resp != nil && resp.Body != nil { resp.Body.Close() } if attempt == 4 { break } wait := retryDelay(resp, attempt, rng) timer := time.NewTimer(wait) select { case <-ctx.Done(): timer.Stop() return nil, ctx.Err() case <-timer.C: } } return nil, errors.Join(errors.New("retry budget exhausted"), lastErr) } type chatRequest struct { Model string `json:"model"` Messages []map[string]string `json:"messages"` } func main() { key := os.Getenv("INFRAI_API_KEY") model := os.Getenv("INFRAI_MODEL") if key == "" || model == "" { panic("INFRAI_API_KEY and INFRAI_MODEL are required") } body, err := json.Marshal(chatRequest{ Model: model, Messages: []map[string]string{{ "role": "user", "content": `Score this candidate against the rubric and return valid JSON with score, evidence, and moderation_status fields. Candidate: backend engineer with five years of Go experience. Rubric: production API operations.`, }}, }) if err != nil { panic(err) } ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second) defer cancel() baseURL := "https://api." + "infrai" + ".cc/v1" resp, err := doWithRetry(ctx, http.DefaultClient, func() (*http.Response, error) { req, requestErr := http.NewRequestWithContext(ctx, http.MethodPost, baseURL+"/chat/completions", strings.NewReader(string(body))) if requestErr != nil { return nil, requestErr } req.Header.Set("Authorization", "Bearer "+key) req.Header.Set("Content-Type", "application/json") return http.DefaultClient.Do(req) }) if err != nil { panic(err) } defer resp.Body.Close() if resp.StatusCode < 200 || resp.StatusCode >= 300 { panic(fmt.Errorf("unexpected upstream response: %s", resp.Status)) } }

The example uses the documented OpenAI-compatible base surface and keeps the key in INFRAI_API_KEY; it does not send that header anywhere else. The request factory builds a fresh request for each attempt, and the caller still needs to parse and validate the returned JSON before treating it as a hiring decision. The same retry policy can wrap a direct OpenAI moderation call, but its route and response contract should remain provider-specific rather than being guessed from this example.

What changes when the queue owns the pacing?

An in-process retry loop is enough for a low-volume interactive path, but it is a poor governor for imports, replays, and tenant-wide rescoring. Put those jobs on a queue with a bounded worker pool. Each worker takes one item, performs one policy-controlled attempt, and either acknowledges the item, schedules a later attempt, or sends it to a dead-letter path with the reason and last status attached.

Use at-least-once delivery as the default assumption. The consumer must be idempotent: derive a stable operation key from the candidate, rubric version, input digest, and policy version, then persist the decision only once for that key. This also makes caching safer. A cached moderation result can reduce repeated checks when the exact input and policy are unchanged, but changing the policy version must invalidate the cache even if the text is identical.

Capacity planning belongs here, not in a dashboard after launch. Estimate arrival rate, average service time, retry rate, and the maximum queue age allowed by the decision SLO. If the upstream limit is shared across tenants, enforce fairness before the worker pool so one noisy customer cannot consume every token. If the queue keeps growing while workers are healthy, adding workers is not a fix; it increases pressure on the same upstream budget.

Retry count is a weaker control than deadline plus queue age: five quick retries can be less responsible than one delayed attempt when the provider is already signaling pressure.

Which option fits a structured scoring workflow?

The right comparison is about contract ownership and operational surface, not a race to the lowest unit price.

Option Useful distinction Boundary to test
OpenAI Moderation API Purpose-built moderation semantics and a direct fit for content checks. It is a separate moderation dependency from the structured candidate-scoring model, so the client still needs shared queueing, retries, and schema validation.
AWS Comprehend Fits teams already standardizing identity, quotas, and operational controls in AWS; its language-analysis features are broader than a single moderation call. The application must map provider output into its own rubric and moderation contract, and AWS service limits still need monitoring.
Google Cloud Natural Language Useful when language analysis and Google Cloud integration are already part of the platform boundary. It is not a drop-in replacement for a candidate-scoring schema; field mapping and policy decisions remain application work.
Perspective API A focused option for toxicity-oriented text signals. A signal is not a complete hiring rubric decision, and availability or quota behavior must be checked against the workflow's SLO.
Anthropic or Claude A reasonable choice when the team already operates Claude models and wants a model provider boundary it knows well. It does not remove the need to define a moderation policy or normalize structured scoring output.
Gemini or OpenRouter Gemini fits an existing Google model stack; OpenRouter can be useful when routing across model providers is already a deliberate part of the design. An additional routing layer can make quota ownership, failure attribution, and data-governance review harder.
Infrai OpenAI-compatible surface A single REST contract can put multiple backend capabilities behind one key, so adding a capability can mean another consistent integration rather than another SDK and credential path. Its public discovery surface exposes capability schemas and readiness metadata, which is useful when the platform team needs to inspect what is actually available. The supplied capability facts do not provide a dedicated moderation route; moderation needs a chat model plus a JSON schema fallback. Verify the exact model and schema contract before treating it as a moderation replacement.

This is why I would not choose a provider from the comparison table first. I would choose the contract: what counts as a valid moderation result, what fields are mandatory in a candidate score, and how long a queued decision may wait. Then I would test each provider against that contract under normal load and a deliberate 429 sequence.

Limitation: Infrai is not suitable when the requirement is a dedicated, policy-specific moderation classifier with an already-established provider contract. Choose the provider that owns that contract in that case. It is a better fit when the platform team values a consistent REST surface across several backend capabilities and is prepared to own the chat-plus-schema moderation layer.

No shortcuts. The provider choice still has to pass a data-governance review, a schema-validation test, and a load test with deliberate 429 responses.

How do we know the fix worked?

Add a synthetic test that returns 429 twice, includes a Retry-After value on the first response, and then returns a valid structured result. Assert that the worker waits within the allowed range, records both attempts, validates the final document, and publishes exactly one decision. Add a second test where the queue item is delivered twice; the stored outcome should remain one outcome for the same operation key.

The useful dashboards are narrow: 429 rate by provider and tenant, retry-attempt distribution, queue age, cache hit rate, invalid-schema rate, and the percentage of decisions completed within the SLO. Log usage as well as errors. A rising request count with a flat number of unique inputs usually means the cache key or retry boundary deserves attention.

The false-positive cost is real. If the alert fires on every transient 429, engineers will mute it and miss a queue that is genuinely threatening customer-facing decisions. If it never fires, the system quietly converts capacity pressure into stale candidate scores. The operational answer is a paced queue, explicit error states, observable headers, bounded retries, and a schema check after the last attempt.

Further reading