Your Marketing AI Calls Run in Real Time. Most of Them Should Run Overnight in Batch.

Most marketing teams call their LLM provider the same way every time: one request, one response, wait for it, pay full price. That is the right pattern for a chat window. It is the wrong pattern for the work that actually fills your calendar: classifying ten thousand form fills, enriching an account list, scoring a backlog of call transcripts, or rewriting a product catalog. If nobody is waiting on the answer, you should not be calling the real-time endpoint. Batch APIs exist for exactly this, and almost no marketing ops team uses them.

Why Real-Time Calls Fail at Volume

Fire 20,000 requests through a loop and three things go wrong. You hit rate limits, so your script spends half its life in retry logic. One flaky response at request 14,000 forces a restart that nobody planned for. And you pay the premium rate for latency you never needed.

A batch job inverts all of that. You upload a file of requests, the provider queues them against spare capacity, and you collect a results file when it finishes. Rate limits for batch queues are separate from your real-time quota, so a big enrichment run no longer starves the production chatbot on your site. Failures are per request, not per job, so one malformed row does not poison the other 19,999.

The tradeoff is time. Results can take hours, not seconds. That is fine for any task where the output lands in a table, a CRM field, or a review queue instead of in front of a waiting human.

Sort Your Workloads Before You Batch Anything

The mistake is treating batch as a cost trick and sending everything through it. Sort by who is waiting.

The test is simple. If a human or a customer is blocked on the response, use the real-time endpoint. If the response gets written to a database and read tomorrow, it belongs in a batch. In most marketing stacks, that second category is the larger one by volume.

How to Build a Batch Pipeline That Does Not Break

The mechanics are consistent across providers. Each request in your input file gets a custom ID. You submit the file, poll the job status or take a webhook, then download the output file and join results back to your source rows by that ID. Output order is not guaranteed, so the ID is the only thing you can trust.

Three habits separate a reliable pipeline from a fragile one:

• Use stable IDs. Derive the custom ID from your own record key, such as the CRM contact ID plus a prompt version. Reruns then overwrite instead of duplicating.
• Pair batch with structured outputs. A schema on every request means the results file loads straight into your warehouse without a parsing step you have to babysit.
• Combine batch with prompt caching. If every request shares a long system prompt or a rubric, caching that prefix stacks with the batch discount. A scoring job with a 4,000 token rubric is the ideal case.

Always run a 50 row pilot batch first. Check the failure rate, spot check twenty outputs, and only then submit the full file. A bad prompt run through batch burns the whole job before you see a single result, which is the one real downside of working asynchronously.

What to Do This Week

Pull your last 30 days of LLM API logs and filter for calls where the response was written to storage instead of shown to a user. Those calls are your batch candidates. Then act on them.

Teams that do this stop treating their AI stack as a chat tool and start treating it as a data pipeline, which is what it has been all along. The interactive endpoint is for people. Everything else should run overnight.