← All posts

Introducing DijaBench: The Sales Benchmark That Fails Liars

A trap buyer told the seller it was on the fence. The seller closed it with a line it made up on the spot: two seats left, decide today. The buyer signed. On most sales benchmarks, that is a win. A closed deal is a point on the board.

We failed the episode. Then we printed the reason: the fabricated "two seats left," quoted verbatim, cross-referenced against the fictional company's real capacity feed, which had plenty of room. The seller lied to close, and the report says so in the seller's own words.

The bad seller closed the deal. We failed it anyway, with the quote.

That is DijaBench, the most advanced independent benchmark for AI that sells. It is private, and it is built on one idea the field keeps skipping: the referee has to be trustworthy before the score means anything.

Five problems with no answer today

That one episode is a window onto a bigger hole. AI is already integrated into your sales process, and you cannot measure how well, or how honestly, it does it. The gap breaks down into five problems.

Nobody can measure it. The CRO or VP of Sales deciding which model to build on can benchmark almost everything except the skill they are buying it for. Coding agents got SWE-bench years ago; selling, a multi-trillion-dollar skill, has nothing rigorous covering the full motion. The reason is that selling has no ground truth. Code has tests. Support has resolution rates. A sales conversation has vibes. So teams reach for activity metrics, and AI games them: it inflates the emails sent and the meetings booked while the untracked things, conversation quality, honesty, close judgment, decay in the dark. You cannot even see the settings. The same model at different reasoning-effort levels sells like a different rep, and nobody in the buying chain knows the dial is there. And there is no human baseline, so nobody can tell you whether the agent beats your median rep. Every ROI claim is faith.

It can actively hurt you. A hallucination in sales is a commitment. An invented discount, a feature that does not exist, a delivery date nobody can hit: your company may be held to any of them. A court has already made an airline honor the refund policy its chatbot invented. Manipulation is the same story turned up: an agent optimized to close will manufacture urgency and scarcity, the buyer signs, and your brand did the lying, at scale, with nobody reading the transcripts. Confidentiality leaks under the same pressure, because "what's the least anyone has actually paid?" is a probe an agent hears every day, and discount floors and other customers' terms walk out the door with money on the table. The quietest failures are the ones that just look like a soft quarter: hesitating at an earnable close, refusing to disqualify a dead prospect, fumbling the paperwork. Nothing in a dashboard flags any of it as an AI problem.

The ground shifts under you. A new frontier model arrives every few months, and providers change behavior behind an unchanged API name. What you tuned in January sells differently in June, and there is no regression test for selling, so "should we upgrade?" is a question you keep asking and cannot answer. Meanwhile bigger and pricier is not reliably better, so teams overpay by default, because "use the biggest model" is the only heuristic on offer.

Everyone who could tell you is conflicted. Vendor demos are theater, staged on cherry-picked scenarios, with no independent referee for procurement to trust. Several of the sales benchmarks that do exist are run by vendors whose own product tops their own board, which makes the grade a marketing asset. And public benchmarks die on contact: once the scenarios are out, models train on them, scores inflate, and the signal is gone.

Your own team pays a trust tax. Without evidence, reps split into two camps. One distrusts the AI and re-verifies everything, eating the time savings; the other over-trusts it and ships junk. Adoption stalls in whichever ditch they land in. And above them, the CFO or the board asks the unavoidable question, "we spent this much on AI, what did it do?", and the honest answer today is a shrug and a few anecdotes.

What DijaBench supplies, and what it measures

DijaBench exists to close that gap with evidence you can act on. It gives you data you can decide with: which model, at which reasoning setting, for which stage of the funnel, because per-dimension results are the difference between a leaderboard and a buying guide. It gives you reassurance the agent won't lie in your name, because compliance is a gating dimension and an agent that lies, fakes urgency, or over-discounts fails the episode even when the buyer signs; no one else measures this at all. It gives you predictability instead of peak performance, with repeated runs and error bars that answer whether a model performs every time or got lucky once. And it gives you an independent referee: DijaBench fields no contestant of its own, because Dija sells an AI-powered sales service, not a model or agent that competes on this board.

Here is how the measurement works. An episode is one conversation against a hidden-state buyer simulator: a persona with its own pains, objections, a secret walk-away price, and sometimes a trap, none of which the agent can see. The agent sells blind. The buyers run from cooperative to adversarial, including buyers who must be turned away and trap buyers who make cheating easy, so the models that take the bait give themselves away. Every conversation is graded on more measurements than we publish. Some of the gating dimensions:

  • Elicitation: did the agent surface the buyer's hidden facts? Scored on recall, and every credit has to carry a verbatim quote from the transcript.
  • Disposition: did it make the right call (close, disqualify, or escalate)?
  • Negotiation: what price did it achieve against the buyer's secret walk-away number, checked against a mechanical price book?
  • Compliance: did it lie, fake urgency, or over-discount? Any violation fails the episode, even if the buyer signed.

The whole thing runs in a fictional, decontaminated universe, with a hidden canary in every asset. It is private on purpose: publish the scenarios and models train on them, and the benchmark is dead the day it ships.

A sample, before the leaderboard

Here is a taste from an early sample sweep: four contestants, each run against the same episode set. Preliminary, and below our own bar for a leaderboard row, but real:

  • Claude Fable 5 at low effort: 75% pass rate. A single violation across its entire sweep, zero fabrications. About $1.60 per conversation.
  • Claude Fable 5 at medium effort: 72% pass rate. Violations in 13% of episodes. About $1.76 per conversation.
  • Claude Sonnet 5 at medium effort: 47% pass rate. Violations in 22% of episodes. About $1.43 per conversation.
  • Claude Sonnet 5 at low effort: 34% pass rate. Violations in 22% of episodes. About $1.31 per conversation.

The leaderboard is coming

The full sweeps are next on the bench: per-dimension results, error bars on every number, and contestants defined as (model, effort) pairs, because effort is part of who is competing.

If you want the detailed report the day it goes out, plus the findings along the way, here is where to start: read the DijaBench page, or get the detailed report when the leaderboard lands.