Most marketing teams are using a frontier model as a very expensive sorting machine. Every inbound form gets sent to a large LLM to decide if it is sales-ready, support, partner, or spam. Every support ticket gets tagged by topic. Every search query from Search Console gets bucketed by intent. These are classification jobs with a fixed set of labels, and they run thousands of times a day. Paying general-purpose reasoning prices for them is a habit, not a decision.
Our position: once a classification prompt is stable, distill it. Use the big model as a teacher to label your historical data, train a small model on those labels, and route only the uncertain cases back to the teacher. You keep most of the accuracy, cut per-call cost by one to two orders of magnitude, drop latency from seconds to milliseconds, and get a model you can version, test, and run anywhere.
Why Classification Is the Wrong Job for a Frontier Model
Frontier models earn their price on open-ended work: drafting, summarizing messy context, multi-step reasoning, tool use. Classification is the opposite. The output space is five or ten labels. The input is usually short. The decision boundary is learnable from a few thousand examples.
Three problems show up when a large model does this job at volume.
Cost scales linearly with traffic. A lead routing prompt with a 1,500 token system prompt and few-shot examples, running on 20,000 form fills and tickets a month, is mostly paying to resend the same instructions. Caching helps, but the floor is still a frontier model call per item.
Latency hurts the moment it sits in a user path. A form that waits two seconds for an LLM to decide which thank-you page or calendar to show feels broken. A fine-tuned small model answers in under 50 milliseconds on a CPU.
Behavior drifts when the provider updates the model. Your labels shift slightly, your routing rules quietly change, and nobody notices until sales complains about lead quality. A small model you trained yourself only changes when you retrain it.
The Teacher, Student, Fallback Pattern
Distillation here is not exotic research. It is a three-part pipeline any marketing ops or data team can run.
The teacher is your current LLM prompt, frozen. Pin the model version, lock the prompt, and require structured output: a label from a fixed enum plus a confidence field. Run it over 3,000 to 10,000 historical items. Use a batch API for this pass, since it is a one-time job and batch pricing is typically half of standard rates.
The student is a small encoder model fine-tuned on those labels. Good starting points are ModernBERT base, DeBERTa v3 small, or a sentence-transformer with a classification head via the SetFit library, which works well even with a few hundred labeled examples per class. Training on a few thousand short texts takes minutes on a single GPU or a cloud notebook. The result is a model of roughly 100 to 400 million parameters that you can serve on a CPU container, a serverless function, or inside your warehouse.
The fallback is the part teams skip, and it is what makes this production safe. The student outputs a probability for each label. When the top probability falls below a threshold you set on validation data, typically 0.7 to 0.85, the item goes back to the teacher. In practice, 80 to 95 percent of traffic is handled by the student and only the hard cases cost frontier prices.
Frontier LLM vs Distilled Classifier for Lead and Ticket Routing
| Factor | Frontier LLM per call | Distilled student plus fallback |
|---|---|---|
| Cost per 10,000 items | Tens of dollars or more | Cents for the student, plus fallback share |
| Latency | 1 to 4 seconds | Under 50 ms on CPU |
| Behavior changes | Whenever the provider updates | Only when you retrain |
| Data leaves your environment | Yes, every call | Only for fallback cases |
| Handles new label types | Edit the prompt | Relabel and retrain |
| Best for | Low volume, shifting categories | High volume, stable labels |
Doing It Without Fooling Yourself
The risk in distillation is that the student copies the teacher's mistakes and you measure success against the same flawed labels. Four practices prevent that.
Build a human-labeled gold set first. Before training anything, have the people who own the outcome, a sales ops lead for routing or a support lead for tickets, hand label 200 to 300 items. This set is never used for training. It is the only honest benchmark you have, and both teacher and student get scored against it.
Score per class, not overall. A routing model that is 94 percent accurate overall can still be terrible on the class that matters most, such as enterprise demo requests, which might be 3 percent of volume. Track precision and recall for each label and set a floor for the expensive ones.
Calibrate the fallback threshold on the gold set. Plot student accuracy against confidence. Pick the threshold where the student's accuracy on retained items matches or beats the teacher's. Everything below it goes to the teacher. Recheck this whenever you retrain.
Schedule retraining from fallback traffic. The items the student sends to the teacher are exactly the examples it struggles with. Log them with the teacher's labels, sample a few dozen per month for human review, and fold them into the next training run. Retrain monthly or when your fallback rate rises more than a few points, which is the clearest signal that your inputs have shifted.
Classifier Distillation Readiness Checklist
- The task has a fixed label set and the LLM prompt has been stable for at least a month
- Volume is high enough that per-call cost or latency is a real problem
- Teacher prompt is pinned to a model version and returns structured JSON with a confidence field
- A human-labeled gold set of 200+ items exists and is excluded from training
- 3,000+ historical items labeled by the teacher via batch API
- Student scored per class on the gold set, with floors for high-value labels
- Fallback threshold calibrated so retained items match teacher accuracy
- Fallback items logged and reviewed for the next retraining cycle
Where to Start This Week
Pick one classifier with the highest call volume. For most B2B teams, that is inbound lead or form triage. Pull the last six months of submissions, run your existing prompt over them in batch, and hand label 250 of them with the person who owns routing. Train a SetFit or ModernBERT student in a notebook and score both models against the gold set.
If the student plus fallback lands within a point or two of the teacher on the classes that matter, ship it in shadow mode for two weeks: it predicts on live traffic but the teacher still decides. Compare disagreements daily. Then flip the switch.
The win is bigger than the invoice. You end up with a classifier that answers instantly, behaves the same next quarter as it does today, and keeps most of your customer data inside your own infrastructure. Save the frontier model for the work that actually needs it.
Tags
LETSGROW Dev Team
Marketing Technology Experts
Ready to Apply This Insight?
Schedule a strategy call to map these ideas to your architecture, data, and operating model.
Schedule Strategy Call