---
title: "Your Marketing AI Jobs Don't Need Real-Time Answers. Move Them to the Batch API."
description: Lead scoring, enrichment, tagging, and audits have no human waiting on the answer, yet most teams run them on the real-time endpoint. Here is how to move bulk LLM work to a batch API and halve the bill.
author: LETSGROW Dev Team
date: 2026-10-16
category: AI Tools
tags: ["Batch API", "LLM", "Automation", "Data Enrichment", "AI Infrastructure"]
url: "https://letsgrow.dev/blog/batch-api-marketing-ai-bulk-jobs"
---
If a task does not need an answer in the next five seconds, it should not run through your real-time LLM endpoint. Most marketing AI work does not need an answer in five seconds, and most teams run it that way anyway.

Enriching 20,000 CRM records, tagging a year of blog posts, scoring every inbound lead from last quarter, rewriting 5,000 product descriptions: these are queue jobs. Every major model provider now offers a batch API built for exactly this shape of work, and it is the most underused piece of infrastructure in the marketing AI stack.

::stat-block
- value: 50%
  label: Typical batch discount on token pricing at the major providers
- value: Up to 24h
  label: Completion window, though most jobs finish far sooner
- value: 50,000+
  label: Requests a single batch job can hold at the larger providers
::

## Sort Your Workloads by Latency Tolerance

Start by splitting every LLM call in your stack into two buckets. Interactive calls have a human waiting: a chat widget, a live sales assistant, an editor suggesting copy. Everything else is deferred work, and deferred work belongs in a batch.

The deferred bucket is bigger than people expect. Nightly lead scoring. Weekly content audits. Bulk classification of support tickets or call transcripts. Metadata generation for a CMS migration. Backfilling embeddings or summaries after a schema change. Running your prompt eval suite before a model upgrade. None of these has a user staring at a spinner.

Anything in that second bucket that currently loops over records with a real-time client is paying a premium for speed nobody uses, and it is competing with your interactive traffic for rate limits.

## Know What You Give Up

Batch is not free. You trade latency and a tight feedback loop for price and throughput.

::compare-table
| Factor | Real-time API | Batch API |
| --- | --- | --- |
| Latency | Seconds | Minutes to hours, up to 24h |
| Price | Standard | Roughly half |
| Rate limits | Shared with live traffic | Separate, much higher queue |
| Error handling | Retry inline | Per-request results, retry the failures |
| Streaming | Yes | No |
| Best for | Chat, live assist, editors | Enrichment, scoring, audits, backfills |
::

Two details matter more than the price. First, batch jobs usually have their own rate limit pool, so a 50,000 row job will not starve your production chatbot. Second, results come back per request, with a custom ID you assign, in no guaranteed order. You must join on that ID, never on position.

## Build the Pipeline Once

A batch pipeline has four moving parts, and each is small.

Write a request builder that turns one row into one request with a stable custom ID, such as the CRM record ID. Keep the prompt identical across rows so prompt caching can stack with the batch discount, and put the per-row data last. Add a submitter that writes the request file, creates the job, and stores the job ID with a status. Add a poller on a schedule that checks status and, on completion, downloads results. Finally add a reconciler that joins results to source rows by custom ID, validates each output against your schema, and writes only the valid ones back.

The reconciler is where jobs succeed or quietly fail. Some requests in any large batch will error, expire, or return malformed output. Treat that as normal. Collect the failed IDs, resubmit them as a smaller second batch, and cap retries at two. If you already enforce schema-validated JSON on outputs, this step is a ten line check.

Test on 50 rows first. Run the small batch, read the outputs by hand, and fix the prompt before you submit 50,000. A bad prompt in a real-time loop wastes minutes. A bad prompt in a full batch wastes a day and your quota.

## Ship It With a Checklist

::checklist
title: Batch API rollout
items:
  - Every LLM call in the stack is labeled interactive or deferred
  - Deferred calls move to the batch endpoint, starting with the largest job
  - Each request carries a stable custom ID tied to a source record
  - Results are joined by ID and validated against a schema before write-back
  - Failed and expired requests are resubmitted once or twice, then flagged
  - A 50 row pilot is reviewed by a human before every full run
  - Jobs are scheduled to finish before the team needs the output
::

## The Takeaway

Pick your single biggest recurring LLM loop this week, the one that runs over thousands of records, and move it to a batch endpoint. You will cut its cost roughly in half, free up your live rate limits, and gain a clean per-request audit trail. Then label the rest of your stack. The default for anything without a waiting human should be batch, and real-time should be the exception you justify.