Optimizely Just Open-Sourced the First Marketing AI Benchmark, Then Used It to Grade Itself
Every AI vendor pitching marketing teams this year has a slide with a win rate on it. None of those numbers were ever comparable, because none of them were measured against the same test. On September 1, Optimizely open-sourced Mark-Bench, the first shared benchmark built specifically to score AI models and agentic platforms on marketing work, then used it to publish a comparison where its own Agent Platform beat Claude Code. The benchmark is a genuine gift to the category. The comparison built on top of it is not something you should accept at face value, and the difference between those two facts is the entire lesson.
What Mark-Bench Actually Measures
Mark-Bench tests models and agent harnesses against 285 tasks spanning 15 marketing functions, graded across more than 6,000 criteria. The tasks are not toy prompts. They are the actual deliverables marketing teams ship every week: a press release that has to hit a specific structure, a social post that has to match a brand voice guide, email copy that has to clear a legal disclaimer requirement, a campaign brief that has to reference real product positioning instead of generic filler. Grading happens on an all-pass basis per task, so a response that gets 90 percent of a press release right and fumbles the dateline still fails that task. That is a much closer approximation of how a marketing ops lead actually judges AI output than the abstract reasoning benchmarks the model labs publish.
That specificity is the real contribution here. Marketing has never had a shared, task-level definition of what "good AI output" means, so every vendor got to define it however flattered their own model. A benchmark built from real marketing deliverables, open enough for anyone to inspect the tasks and run their own model against them, is the kind of infrastructure this category should have built two years ago.
The Headline Number Is Not What It Looks Like
Here is where the story turns. Optimizely ran the comparison, on its own benchmark, using default configurations, and published the result as evidence that its purpose-built marketing models beat a general-purpose coding agent at marketing work, for less money. Every part of that sentence should slow you down. The company that built the test also chose which competitor to run it against, chose the configuration, and controlled the write-up. Claude Code is not a marketing-specific product; running it head to head against a model post-trained specifically on marketing tasks, in its default mode, is closer to testing a Swiss Army knife against a scalpel and reporting that the scalpel cuts better. It probably does. That is not the same claim as "Optimizely's platform is the best marketing AI on the market," which is the claim the press coverage ran with.
None of this makes Mark-Bench dishonest. It makes the September 1 comparison a vendor benchmark result, which is a different category of evidence than an independent one, and marketing teams have gotten burned enough times by vendor benchmark results that the distinction should be reflexive by now. A 67 versus 60 all-pass rate is a real, specific, checkable number. Whether it means what the press release implies is a separate question, and it is the question every AI vendor evaluation from here forward needs to ask before it repeats a vendor's own scorecard as fact.
What Open Source Actually Buys You
The reason Mark-Bench is worth using anyway, despite the self-graded launch comparison, is that the benchmark itself is inspectable and runnable by anyone, which a marketing-specific benchmark from a neutral third party has never existed to do. You do not have to trust Optimizely's comparison. You can pull the same 285 tasks and run them against whatever combination of models and agent frameworks your team is actually evaluating: the general-purpose assistant you already pay for, the vertical marketing tool a vendor is pitching you, the open-weight model your engineering team wants to self-host. The value is not the scoreboard Optimizely published. It is the shared, task-level test set that lets you build your own scoreboard, scoped to the tools you are actually choosing between and the tasks your team actually ships.
That is a meaningfully different exercise than reading a vendor comparison and deciding whether to believe it. It is the difference between trusting a nutrition label a company wrote about its own product and running the lab test yourself.
Build Your Own Eval Slice Before You Buy Anything
Treat Mark-Bench as a starting task set, not a verdict. Pull a slice of maybe 20 to 30 tasks that map closest to what your team actually produces, whether that is email copy, campaign briefs, or social content, and run every tool on your vendor shortlist through the identical slice yourself.
- Confirm who ran the benchmark and who chose the comparison configuration
- Pull the task set yourself and run it against your actual shortlist, not the vendor's chosen competitor
- Score on your own criteria if your deliverables differ from the benchmark's default rubric
- Re-run the eval slice every time a vendor ships a new model version, since scores drift
- Never let a vendor's self-published comparison be the only evidence behind a purchase decision
The teams that get real value out of Mark-Bench will not be the ones quoting the 67 percent number in a slide deck. They will be the ones who downloaded the task set, ran it cold against three tools nobody paid to be compared, and made a purchase decision off criteria they controlled the entire time. An open benchmark only levels the playing field if you actually run it yourself instead of reading someone else's printout of the results.
Tags
LETSGROW Dev Team
Marketing Technology Experts
Ready to Apply This Insight?
Schedule a strategy call to map these ideas to your architecture, data, and operating model.
Schedule Strategy Call