Batch inference costs 50% less than real-time inference at Anthropic, OpenAI and Google, in exchange for results that can take up to 24 hours. For a workload of one million document classifications a month on a mid-tier model, that is roughly $7,000 a month in real time versus $3,500 in batch, about $42,000 a year saved (illustrative calculation below).
The catch is that only work nobody is waiting on can move to batch: enrichment, classification, evaluations, reports, backfills. Anything a user sees live, such as chat, search answers or agents acting in a session, stays real-time. For those, prompt caching is the main cost lever, and at Anthropic its discount stacks with batch.
The practical move for most companies is to split their AI traffic in two. Run interactive requests in real time with caching, and route everything else through a batch queue. The rest of this article shows how to size each part.

How batch and real-time inference differ

The price difference reflects one tradeoff: the provider decides when your request runs.
Real-time (standard) inference returns a response in seconds, usually streamed. You pay full list price, and your throughput is capped by per-minute rate limits.
Batch inference accepts a file or list of many requests, processes them when capacity is free, and returns all results together. Anthropic's Message Batches API, for example, accepts up to 100,000 requests or 256 MB per batch. Most batches finish within an hour, but any request not done within 24 hours expires, and results stay downloadable for 29 days (Anthropic docs).
Middle tiers now exist between the two. OpenAI's flex processing charges Batch API rates for ordinary synchronous calls that are slower and can be refused when capacity is short; refused requests return an error and are not charged (OpenAI docs). Google's Gemini API offers a Flex tier at 50% off with a 1–15 minute target, and a Priority tier that costs 75–100% more for the fastest, non-sheddable service (Google docs).
So the real choice is not two modes but a range. Pay more for guaranteed speed, standard price for normal interactive use, or half price for work that can wait minutes to hours.

Provider pricing: batch discounts and caching (sourced)

All three major API providers price batch at half of standard, so the decision is about workload fit, not vendor choice.
Anthropic: Message Batches cost 50% of standard prices on input and output tokens. Prompt caching discounts stack with batch pricing, though cache hits in batch are best-effort, typically 30–98% depending on traffic (Anthropic batch docs). Example batch prices: Claude Sonnet 5.5 at $1 input and $5 output per million tokens; Claude Opus 5.5 at $2 and $10. Standard prices are double.
OpenAI: launched its Batch API in April 2024 at a 50% discount with a completion window of up to 24 hours (Simon Willison). Flex processing gives Batch API rates on synchronous calls, and prompt caching can add further discounts (OpenAI docs).
Google Gemini: Batch mode is 50% off with a target turnaround of up to 24 hours and input files up to 2 GB. Flex is also 50% off. Context caching gives a 90% discount on cached tokens plus a storage charge (Google docs, updated October 2026).

How prompt caching changes the numbers

When many requests share a long prefix, such as a system prompt, policy document or product catalog, the provider can reuse it. Reading cached tokens costs a small fraction of normal input price: 10% of base input on standard Anthropic models, with lower multipliers reported for some newer models, and a premium of 1.25 to 2 times base input to write the cache. This model uses the conservative 10% read price. Verify current multipliers before budgeting.
List prices change frequently and enterprise contracts often differ. Treat every figure here as a November 2026 snapshot.
*Cost model: three workloads (illustrative)
**Batch halves the bill for work that can wait; caching does most of the saving for work that cannot. All three scenarios use Claude Sonnet 5.5 list prices (standard $2 input and $10 output per million tokens; batch $1 and $5). Volumes are invented examples, and cache-write charges are left out because they are small when one prefix serves many requests.
**The formula:
*\text{Monthly cost} = \frac{\text{requests} \times \text{input tokens}}{10^6} \times p_{in} + \frac{\text{requests} \times \text{output tokens}}{10^6} \times p_{out}

Scenario 1: document classification and enrichment

1 million documents a month, 2,000 input tokens and 300 output tokens each. Nobody waits on individual results.
Input: 2,000 million tokens. Output: 300 million tokens.
Real-time: 2,000 × $2 + 300 × $10 = $4,000 + $3,000 = $7,000 a month
Batch: 2,000 × $1 + 300 × $5 = $2,000 + $1,500 = $3,500 a month
Saving: $3,500 a month, $42,000 a year.

Scenario 2: customer support assistant

500,000 conversations a month, 4,000 input tokens each (of which 3,000 are a shared system prompt and knowledge snippet), and 400 output tokens. Customers wait live, so batch is not an option.
Input: 2,000 million tokens, of which 1,500 million are cacheable. Output: 200 million tokens.
Real-time, no caching: 2,000 × $2 + 200 × $10 = $4,000 + $2,000 = $6,000 a month
Real-time with caching: 1,500 × $0.20 + 500 × $2 + 200 × $10 = $300 + $1,000 + $2,000 = $3,300 a month
Saving: $2,700 a month (45%), with no change to user experience.

Scenario 3: nightly evaluation and reporting pipeline

200,000 requests a month, run overnight, 10,000 input tokens each (8,000 shared context) and 1,000 output tokens. Results are needed by morning.
Input: 2,000 million tokens, of which 1,600 million are shared. Output: 200 million tokens.
Real-time: 2,000 × $2 + 200 × $10 = $6,000 a month
Batch: 2,000 × $1 + 200 × $5 = $3,000 a month
Batch plus caching, assuming a 50% cache hit rate on the shared context (Anthropic describes batch cache hits as best-effort): 800 million cached × $0.10 + 1,200 million uncached × $1 + 200 × $5 = $80 + $1,200 + $1,000 = $2,280 a month

Saving against real-time: $3,720 a month (62%).

Notice what dominates once input is discounted: output tokens. In Scenario 3, output is 44% of the batch-plus-cache bill. Shorter, structured outputs are often the next lever.

Self-hosted inference: utilization decides the math

If you run an open-weight model on rented GPUs, you pay for GPU hours whether or not requests arrive. Batch workloads matter here for a different reason: they keep GPUs busy.
Sourced GPU prices. An October 2026 survey of H100 rental prices found specialist clouds listing roughly $2 to $4.50 per GPU-hour on demand, and hyperscalers roughly $7 to $12 (IntuitionLabs). Several providers raised list prices in 2026.
Illustrative calculation. Assume a $3 GPU-hour and a model that sustains 1,000 output tokens per second per GPU at full load. That throughput is an assumption for illustration; real figures vary widely by model size, hardware and serving software.
At full load: 1,000 × 3,600 = 3.6 million tokens an hour, so $3 ÷ 3.6 = $0.83 per million tokens
At 70% utilization: $0.83 ÷ 0.70 = $1.19 per million
At 30% utilization: $0.83 ÷ 0.30 = $2.78 per million
Interactive traffic is spiky, so real-time clusters are sized for peaks and sit partly idle. Batch jobs fill those idle hours, which is why self-hosting teams run backfills and evaluations overnight on the same hardware.
These per-token numbers are not directly comparable to API prices. They use a different model, and they exclude the engineers who keep the serving stack running, monitoring, and redundancy. Self-hosting usually makes economic sense only at sustained high volume, or when data cannot leave your environment.

Which workloads belong where

Ask one question of every AI call: is a person or a live process waiting for this answer right now?

Good batch candidates

Classifying, tagging or extracting fields from documents, tickets and records
Enriching CRM or product catalogs
Generating embeddings or summaries for search indexes
Running evaluation suites and regression tests on prompts or models
Overnight reports, digests and analytics narratives
Backfilling historical data after a prompt or model change
Generating synthetic data or test cases

Must stay real-time

Chat assistants and customer support
Search answers and in-app copilots
Agents taking actions inside a user's session
Fraud, moderation or routing decisions that block a transaction

Middle ground: flex tiers

Work that can wait minutes but not hours, such as a background summary a user checks later, fits flex tiers. They give batch-level prices on ordinary synchronous calls, but requests can be refused or delayed, so your code needs a fallback to standard.

The hybrid pattern

Most production systems end up with three lanes. Interactive traffic runs real-time with prompt caching. Deferrable traffic goes into a queue that submits batches on a schedule. Anything urgent that a flex call cannot serve falls back to standard. The engineering is modest, and the savings apply to every token in the deferrable lane.

Engineering costs and risks beyond the token bill

The batch discount is only real if the pipeline around it is reliable; budget engineering time for these pieces.
Queue and scheduler. Requests must be collected, grouped within batch size limits, and submitted on a schedule.
Result handling. Results arrive together, so each one must be matched back to its source record by ID and written to the right place.
Expiry and retries. Requests not completed within the provider's window expire. You need logic to detect and resubmit them, and an alert if a whole batch fails.
Partial failures. Some requests in a batch can fail while others succeed. Track status per request, not per batch.
Result retention. Anthropic keeps batch results for 29 days. Download and store them before then.
Freshness. A result produced overnight reflects yesterday's data. Check that downstream users accept that.
Cost visibility. Tag requests by lane and team so finance sees what batch is actually saving.
Two further risks apply to every mode. Provider prices and model versions change often, so a budget built on one month's list prices can drift. And a cheaper token bill does not help if output quality drops; rerun your evaluation set whenever you change model, mode or prompt.

Where CodingCops fits

Disclosure: the author works with CodingCops. This section uses only the company's published descriptions and includes no CodingCops pricing, client results or savings claims.
CodingCops is a Chicago-headquartered software company founded in 2010 (Built In Chicago). Its company profile lists AI and machine learning, cloud and DevOps, custom software development, and dedicated engineering teams among its services (Outsource Accelerator).
The work described in this article, separating interactive and deferrable traffic, building a batch queue with retries, and tracking cost per lane, sits at the intersection of AI engineering and DevOps. Teams without that capacity in-house can add it through a dedicated team or staff augmentation, which CodingCops offers for specific projects, short-term initiatives or ongoing development (codingcops.com).
[Marketing to verify before publishing] Confirm the description wording. Do not add token-cost savings or client results without evidence and approval.

FAQs

How much cheaper is batch LLM inference?

Anthropic, OpenAI and Google all price batch at 50% of standard rates, as of November 2026.

How long does a batch job take?

Up to 24 hours at all three providers. Anthropic says most batches finish within an hour; requests not done within 24 hours expire.

Can I combine batch with prompt caching?

At Anthropic, yes: the discounts stack, though cache hits in batch are best-effort. Check each provider's current docs.

What is flex processing?

A synchronous tier priced like batch. Calls are slower and may be refused when capacity is short, so you need a fallback. OpenAI and Google both offer one.

Is self-hosting cheaper than batch APIs?

Only at sustained high volume with high GPU utilization, and after counting the engineers who run it. Idle GPUs make self-hosted tokens expensive quickly.

What share of AI workloads can move to batch?

It depends entirely on the product. Audit your own traffic: anything nobody waits on is a candidate.