Small Decision Models May Deliver More

Why Test Small Decision Models?

Teams should test compact models instead of assuming the largest will perform best. For decision workloads, parameter count is a poor substitute for task-specific quality and deployment cost.

A decision model scores a fixed set of options and returns probabilities, rather than generating a written answer that another system must interpret. That makes it a candidate for classification and structured choices, where the required output is a label or selection.

A comparison of hosted Jev 1.13, open Clef models, and Kev 4B gives teams a concrete reason to evaluate smaller candidates. Jev led the comparison, but Kev 4B scored close to Clef-Flash 9B despite its smaller size. Those results support testing compact candidates, not assuming they will fit every production workload. Use your own labeled decisions and deployment constraints to decide which models belong on your shortlist.

What Did the 1,085-Decision Test Measure?

The comparison tested seven typed-decision models on 1,085 decisions. It graded 1,048 items for accuracy and reserved 37 to test consistency. The results compare these models on a defined set of tasks, not every production decision.

The five suites covered games with answers checked by search, logic and thought experiments, company policies across 20 industries, usability intents across 14 app types, and stress tests. The items had checkable answers, and the study made its item list and calls available for inspection. Teams can examine what was asked and how each answer was judged, rather than relying on a score from an unspecified collection of prompts.

The test also varied the presentation of choices. It ran choice items in their original order, with the order shuffled, and with options renamed. This probes whether a model changes its answer when labels or ordering change.

These suites do not establish performance on a particular company’s support tickets, routing rules, or risk decisions. Treat the benchmark as evidence about how the seven models behaved on these items, and as a reason to test compact candidates on your own labeled workload.

How Did Clef-Flash Compare With Smaller Open Models?

The 1,085-decision comparison shows a clear lead for hosted Jev 1.13, but a narrower gap between Kev 4B and Clef-Flash 9B. The results show that parameter count alone is not a reliable stand-in for decision quality, though they do not mean compact models always win.

Jev scored 76.8% across the 1,048 graded items. The strongest open model, Clef 27B, scored 69.9%. Clef-Flash 9B reached 65.2%, while Kev 4B scored 64.0%. The reported head-to-head test found no statistically significant difference between Kev 4B and Clef-Flash 9B ($p = 0.53$). The comparison treated them as tied; that is not proof they perform identically.

Kev 4B was reported at 3 GB, compared with Clef-Flash’s 9 billion parameters. Their measured accuracy differed by 1.2 percentage points on this benchmark. Clef-Flash scored higher in this pair, but teams should weigh that result against memory, hardware, and operating cost rather than assume it settles the choice.

Jev’s 6.9-point lead over Clef 27B, the best open model, is evidence for Jev on these tasks. It does not show that a larger model caused the lead or that the same ranking will hold for a team’s own decisions. The benchmark supports a narrower conclusion: model size alone does not explain the ranking, and Kev 4B’s proximity to Clef-Flash 9B makes compact candidates worth testing.

Why Is Parameter Count a Weak Guide to Decision Quality?

Parameter count cannot tell you whether a model fits a decision task. A fixed-option decision asks the model to interpret the state, the question, and the available choices, then score those choices. It does not reward a long, fluent answer. More parameters do not guarantee better performance on that mapping.

The 1,085-decision comparison makes the point with concrete results. Kev 4B scored 64.0 percent and Clef-Flash 9B scored 65.2 percent; their head-to-head test reported $p = 0.53$ and treated them as tied. Hosted Jev 1.13 scored 76.8 percent, ahead of every open model tested. These results reject a simple rule that more parameters always means better decision quality. They do not show why the models differ internally or whether a compact model will match a larger one on your workload.

The reported 95 percent accuracy intervals were 61 to 67 percent for Kev 4B and 62 to 68 percent for Clef-Flash 9B. The benchmark’s mix of games, logic questions, company policies, usability intents, and stress tests also shapes its ranking. A different mix could change the result. If representative evaluations repeatedly showed that size reliably predicted task accuracy after accounting for uncertainty, that would be evidence to treat parameter count as a stronger guide. For now, measure the decision behavior you need.

What Should Teams Measure Besides Parameter Count?

Rank candidates by workload-specific quality and deployment fit, not by parameter count or a single headline accuracy number. A model that scores well overall can still be a poor choice if it makes costly mistakes with unjustified confidence or cannot meet latency and memory limits.

Start with accuracy on labeled examples that represent the decisions your system will make. Break results out by important classes, since aggregate accuracy can hide weak performance in a high-impact category. If your system uses confidence thresholds or abstains, measure calibration and the rate of confident errors. Test consistency by reordering answer choices and making small wording changes. A model that flips its answer under either change may be harder to trust in production.

Measure latency on the hardware you intend to deploy, along with memory use, deployment requirements, and cost per decision. Hosted pricing and local inference costs are different quantities, so compare them against your actual serving setup.

A separate routing benchmark illustrates why accuracy alone is not enough. On 1,020 requests, Kev-4B reached 76 percent accuracy in English and 70 percent in French; hosted Jev reached 81 percent and 76 percent. Jev was wrong while confident on one in eight requests, compared with one in thirty for Kev-4B. For a router that acts only above a confidence threshold, that difference can matter more than the accuracy gap. These figures come from the 1,020-request routing test, not the 1,085-decision comparison discussed earlier.

No single metric settles the choice. Set minimum thresholds for quality, confidence behavior, latency, memory, and cost before comparing models. Choose among the candidates that meet them.

When Should Teams Choose a Larger Model?

Choose a larger model when it makes fewer costly mistakes on your workload and that quality gain justifies its deployment cost. The strongest case in the 1,085-decision comparison is Jev 1.13: it scored 76.8% and led every open model. Clef 27B, the best open model, scored 69.9%, above compact Kev 4B at 64.0% and Clef-Flash 9B at 65.2%.

For decisions where an error has a high business cost, that gap may justify a hosted service or a larger model. The benchmark’s aggregate accuracy does not show how errors are distributed across classes. A model can score well overall while failing on the class that carries the most risk. Measure that class directly before choosing.

A compact candidate is not a good choice if it misses your required quality, confidence, latency, or reliability thresholds. Parameter count cannot answer whether a model clears those thresholds; your evaluation can. Choose the larger model when representative, labeled cases show a meaningful improvement on consequential decisions and the cost of serving it is acceptable. Without that evidence, size alone is not a reason to pay more or take on heavier deployment requirements.

How Should You Shortlist Small Decision Models?

Start with a compact candidate, then move to a larger model only when the same workload-specific evaluation shows a worthwhile gain. The 1,085-decision comparison is a reason to test small decision models, not evidence that any candidate will handle your business decisions.

Use identical labeled cases and decision formats for every candidate. Before running them, set minimum thresholds for accuracy, high-impact errors, calibration or abstention, latency, memory, and cost per decision. Choose the least costly candidate that clears every required threshold.

Compare errors by consequence, not only in aggregate. A model with higher overall accuracy can still fail your shortlist if its mistakes cluster in a costly class. A compact model that misses your accuracy or reliability floor is not a bargain. The benchmark found Kev 4B close to Clef-Flash 9B, while Jev 1.13 led all open models; neither result predicts your own workload without testing it.

Include compact models in the evaluation instead of assuming the largest candidate will be best.