Two public workloads, three serving configurations, one reference server. Every figure is model-serving energy at batch size 1, first-party, and not yet reproduced by anyone outside this project.
Version 1.0 · 17 September 2026 · next refresh by 17 October 2026
The measurements
| Workload | Model | Output constraint | Accuracy | Energy / request |
|---|---|---|---|---|
| Banking77 | 27B | label grammar | 93.9% | 284.5 J |
| Banking77 | 2B | JSON schema | 91.5% | 73.1 J |
| Banking77 | 2B | label grammar | 92.0% | 24.3 J |
| ContractNLI | 27B | — | 78.5% | 649.1 J |
| ContractNLI | 2B | — | 84.7% | 9.0 J |
Swipe the table for more columns.
The single-sentence version: on a 77-class intent task, the same 2B model answered the same 3,072 cases for 24.3 J per request under a restricted label grammar and 73.1 J under a JSON schema — a 67% energy reduction with no change to the model, the prompt or the task. The 27B incumbent on that task used 284.5 J. On ContractNLI, 1,763 cases, the 2B used 9.0 J against the 27B’s 649.1 J.
Banking77 is 3,072 cases, ContractNLI is 1,763. The spread on Banking77 is the useful part: the same 2B model, on the same task, moved from 73.1 J to 24.3 J when the output constraint changed from a JSON schema to a restricted label grammar. Two thirds of the remaining energy was going into generating punctuation. The same kind of waste shows up with shared prompts: reusing a cached prefix cut energy per request 68 to 82% on 32,000-token prompts (prefix caching speed and energy).
Reading the accuracy column
Both benchmarks have published task-specific baselines that score higher than either model above, at a fraction of the size. Banking77 was introduced in a 2020 paper whose whole argument was that small sentence encoders beat fine-tuning BERT-Large on intent detection. ContractNLI’s origin paper reports Span NLI BERT-base at 110M parameters reaching 83.8%, and a 335M variant reaching 87.5%.
So the honest reading of the accuracy column is not that a small generative model kept up with a large one. It is that both generative models were the wrong tool for these two tasks, and the energy column is what that costs. Running a general-purpose model where a specialised one would do is a measurable expense, and it is exactly the kind of thing a measurement turns up and an accuracy leaderboard does not.
Published baselines above are task-fine-tuned encoders with span-level heads. Our runs use a different configuration, so these are not like-for-like comparisons. We cite them because they bound what is achievable on these tasks, not because they refute our numbers.
Disclosure
Model serving only. Retrieval, embedding and orchestration are outside the boundary and excluded rather than estimated.
Batch size 1. The format gap has not been measured under production batching, where per-token masking overhead may amortise.
One reference server. A replay from AMD onto a rented NVIDIA GPU matched 298 of 300 items; the two mismatches are reported, not rounded away. Memory needs by model size were measured separately, on a rented 96 GB card: see VRAM by model size, measured.
One family, two sizes. Nothing here establishes that the pattern holds across families, and we don’t claim it does.
Not disclosed in this version. It will be named in the next refresh rather than described vaguely.
Not disclosed in this version. Stating the field and leaving it empty is more useful than filling it with an approximation.
Controls
A third workload produced a result our pre-registered screening rejected on a data defect. We withdrew it rather than publish it. That stage exists so a bad run doesn’t become a claim, and it’s why this page has two workloads rather than three.
Across 2,400 cases and eight public tasks, reasoning solved 5% more cases than not reasoning. On a 300-case banking test it cost 51 times the energy for a 2.6-point accuracy gain. Worth having for hard cases, expensive as a default.
On HumanEval+, a stock 7B coding model with a five-tier retry ladder passed 144 of 164 hidden tests; a 132-sample fine-tune matched it within noise at 17.3% fewer tokens and 20.4% less time per task. A follow-up run metered it: 1,367 J per task with the ladder against 728 J for a single attempt. The write-up, with the legend. A follow-up put small vs large models on the same 164 tasks: the 7B passed 144, a 27B passed 152.
The JSON-schema against label-grammar comparison is the one worth running on your own stack. It needs no vendor and no data sharing, and an engineer can do it in an afternoon. If it holds for you, the question worth asking is which of your other workloads it applies to.
Follow the measurements
Each refresh adds workloads, fills in a disclosure field we couldn’t fill before, or corrects something. One email when that happens, nothing else.
No spam. Unsubscribe in one click.