Batch API or real-time calls: how to choose the right channel for different workloads

First, distinguish the two time scales

LLM calls vary enormously in their sensitivity to time. A user waiting for a reply in a chat box tolerates seconds; a team's daily report generation task may tolerate hours. The batch channel is designed for the latter: it queues requests, processes them in bulk, and trades a longer completion time for a lower per-token price.

Anthropic's [batch processing docs](https://platform.claude.com/docs/en/build-with-claude/batch-processing) and [pricing page](https://claude.com/pricing) both list batch-tier prices separately; OpenAI also has corresponding batch and batch processing discounts. What they share is a discount off the standard price; what differs is the submission method and completion window.

Decision framework: four questions

1. Is someone synchronously waiting for this result? Yes → real-time. No → continue.
2. Can the result be a few hours late? Yes → batch candidate. Must be minute-level → real-time.
3. Is the request set something that can be enumerated offline? Yes (a batch of documents, a batch of tickets, a batch of evaluation samples) → batch is a very good fit.
4. Is the cost of rerunning on failure high? High → real-time for immediate feedback; low → batch retries in bulk are more cost-effective.

Typical batch scenario checklist

• Content classification and tagging: a unified tagging system for hundreds of thousands of products, reviews, and tickets.
• Offline data cleaning: extracting unstructured text into structured fields in bulk.
• Regression evaluation: running a fixed evaluation set for comparison before changing prompts.
• Bulk summarization / translation: offline organization of document libraries, customer service records, and meeting minutes.
• Synthetic data generation: generating samples in bulk for training or evaluation.

The pitfalls of batch: partial success and retries

A batch job is not 'all or nothing'. In a job of several hundred requests, a few usually fail due to parameter errors, overly long inputs, content policy, and so on. Engineering must ensure:

• Record results per item; do not treat the success or failure of the whole batch as a single boolean.
• Distinguish retryable from non-retryable: 429 / 5xx are retryable; retrying a 400 parameter error is useless.
• Idempotency keys: resubmitting the same business data must not cause duplicate billing or duplicate writes.
• Cost rehearsal: estimate an upper bound by tokens before submission to avoid an unexpected bill from a single job.

Manage both channels with one gateway

Splitting real-time and batch into two independent sets of accounts, keys, and bills is the most common architectural debt: when prices change you have to update two places, and troubleshooting failures means digging through two backends.

Admesh's approach is one Base URL, one set of keys, and one bill: on the business side you use task type to decide which channel to use, while the model catalog and unit prices always come from the same backend configuration, so margin and usage are visible in one place.

View rate limit and concurrency documentation →