Your Marketing AI Is a Black Box. Trace Every LLM Call With OpenTelemetry.
Your marketing team probably runs a dozen LLM calls in production: product descriptions, lead enrichment, ad variants, support triage. Ask which one got slower last week, which one got more expensive, or which prompt version produced the bad batch of emails, and you will get a shrug. That is not an AI problem. It is an observability problem, and it is solvable in an afternoon with tools you already own.
The position is simple: every LLM call in your marketing stack should emit a trace, and you should use the OpenTelemetry GenAI semantic conventions to do it. Not a vendor dashboard. Not a spreadsheet of API invoices. Traces, in a format any backend can read.
Why Invoices and Provider Dashboards Are Not Enough
Provider dashboards tell you total tokens by API key. That answers "how much did we spend," never "which workflow spent it." When three automations share one key, the dashboard is a single number with no owner.
Worse, dashboards cannot connect cost to outcome. You cannot see that the lead-scoring prompt retried four times on malformed output, or that a prompt edit on Tuesday doubled average output length. Those are the failures that quietly burn money and degrade quality for weeks.
Traces fix this because each call becomes a span with attributes you control: which workflow, which prompt version, which model, how many tokens, how long, and whether it succeeded.
The Minimum Viable Span
OpenTelemetry defines standard attribute names for generative AI calls, so your data stays portable across Datadog, Grafana, Honeycomb, or a self-hosted collector. Do not invent your own field names. Capture these on every call:
gen_ai.request.modelandgen_ai.response.model, because aliases like "latest" driftgen_ai.usage.input_tokensandgen_ai.usage.output_tokensgen_ai.operation.name, such as chat or embeddingserror.typewhen a call fails or returns invalid output- Two custom attributes:
app.workflowandapp.prompt_version
Those last two are the ones that make traces useful to marketers. Without a workflow name, you cannot assign cost to a team. Without a prompt version, you cannot tie a quality drop to a change.
Here is the wrapper pattern in Node. It is about fifteen lines:
import { trace, SpanStatusCode } from '@opentelemetry/api';
const tracer = trace.getTracer('marketing-ai');
export async function tracedChat({ workflow, promptVersion, model, run }) {
return tracer.startActiveSpan(`chat ${model}`, async (span) => {
span.setAttributes({
'gen_ai.operation.name': 'chat',
'gen_ai.request.model': model,
'app.workflow': workflow,
'app.prompt_version': promptVersion,
});
try {
const res = await run();
span.setAttributes({
'gen_ai.usage.input_tokens': res.usage.input_tokens,
'gen_ai.usage.output_tokens': res.usage.output_tokens,
});
return res;
} catch (e) {
span.setStatus({ code: SpanStatusCode.ERROR });
span.setAttribute('error.type', e.name);
throw e;
} finally { span.end(); }
});
}
Route every LLM call through this one function. If someone bypasses it, that is a code review comment, not a philosophy debate.
Decide What You Will Not Log
Prompt and completion bodies are optional in the spec for a reason. Marketing prompts carry customer names, emails, and CRM notes. Log token counts and metadata by default. Capture full bodies only for a sampled slice, in a store with short retention, after redaction. If you have not stripped PII before the call, do that first and treat tracing as the second line of defense.
What to Alert On
Once spans flow, three alerts earn their keep immediately. Cost per workflow per day, compared against a trailing seven-day median. Output tokens per call by prompt version, which catches the verbose-prompt regression. Error and retry rate, which catches schema failures and rate limits before customers do.
Provider Dashboard vs. OpenTelemetry Traces
| Question | Provider Dashboard | OTel GenAI Traces |
|---|---|---|
| Which workflow costs the most? | Not available | Group by app.workflow |
| Did last Tuesday's prompt edit change output length? | Not available | Compare by app.prompt_version |
| Which calls are retrying? | Aggregate error count only | Per-span error.type and retry count |
| Can we switch backends? | Locked to vendor | Standard attributes, any backend |
| Is PII exposed? | Depends on vendor | You control what is recorded |
One caution on sampling: trace every call's metadata, but sample bodies. Metadata is cheap, a few hundred bytes per span, and it is what powers every alert. Bodies are expensive and risky, so keep one percent of successes and every failure for a short window. That gives you enough raw material to debug a bad batch without building a second copy of your customer database.
Your Rollout Plan
Start with the one workflow that spends the most. Wrap its calls, ship spans to whatever backend your engineering team already runs, and build one dashboard: cost and p95 latency by prompt version. Run it for two weeks before touching anything else. You will find at least one prompt that is far more expensive than anyone believed.
Then extend the wrapper to every workflow, add the three alerts, and make "prompt_version" a required field in code review. Observability is not overhead on AI work. It is the difference between running AI in production and hoping it behaves.
Tags
LETSGROW Dev Team
Marketing Technology Experts
Ready to Apply This Insight?
Schedule a strategy call to map these ideas to your architecture, data, and operating model.
Schedule Strategy Call