Calibration, honestly

The definition#

From TypeSafe's own AI primer, and this is exactly right:

Followed immediately by the qualifier that does most of the work: "These rates describe groups of predictions, not a guarantee about any single answer."

Why it matters more than accuracy#

An uncalibrated model that is 95% accurate cannot be automated, because you do not know which 5% you are in. A calibrated model that is 80% accurate can be, because it tells you when it is unsure and you route those cases elsewhere. The useful quantity is not accuracy — it is accuracy on the subset you chose to act on, and calibration is what lets you choose that subset.

This is the genuine gap in ordinary structured outputs. Constrained decoding guarantees you get a valid enum member. It says nothing about whether the model knew what it was doing, and asking an LLM for a confidence score in the same JSON object produces a number that is, empirically, close to decorative.

Calibrated is not the same as useful#

The sharpest counterweight came from Anthony Maio, and it is a genuine limitation rather than a quibble:

If 12% of your tickets are refund requests and a model answers 0.12 to every single one, it is perfectly calibrated and completely worthless. Calibration is necessary for automation and nowhere near sufficient. The property you actually want is calibration plus sharpness — probabilities that are both honest and concentrated near 0 and 1.

In practice this means you should look at the distribution of confidences your system produces, not just the accuracy within each band. A model returning 0.55 on everything is telling you something is wrong — usually with your criteria rather than with the model.

Three ways calibration decays in production#

1. Distribution shift. Calibration is measured on a distribution. Changed policies, unfamiliar customers, new fraud patterns and adversarial inputs can quietly undermine probabilities that were reliable last month. Nothing errors; the numbers just stop meaning what they meant.
2. Composition. Individually calibrated judgments do not automatically compose into a calibrated workflow once you run them through thresholds, weights and branches. Correlated mistakes survive composition — and in a fan-out, every question saw the same state, so the errors are correlated by construction.
3. Version drift. A threshold tuned on jev-1.13.0 encodes that model's distribution shape. Point jev-latest at something new and your thresholds are silently re-aimed. Pin the version.

What has actually been measured#

TypeSafe has published no calibration metric of its own — no ECE, no Brier score, no reliability diagram. For a company whose entire differentiator is calibration, and for the single easiest property in machine learning to measure, that remains the most conspicuous omission of the launch. Diogo Almeida has defended the absence of benchmark tables as policy, arguing in an essay called Lies, Damned Lies, and Benchmarks for "the clean slate: no standard benchmark table in our model releases."

The community did it instead, within days of launch. The results are genuinely split — which is more informative than either a clean pass or a clean failure would have been.

Jev is now on OpenRouter, Vercel AI Gateway and Cloudflare Workers AI, so running this yourself no longer needs waitlist access. There is runnable reliability-diagram code here. Given the split in the published results, running it on your own distribution is not optional diligence — it is the only version of this claim that will ever apply to your system.

What TypeSafe has and has not said#

None of this means the claim is false. The early independent evidence is genuinely mixed rather than negative, and TypeSafe's willingness to publish a nine-item list of its own model's failure modes suggests a team that is not primarily in the business of overclaiming. What it means is that calibration is a property of a model on a distribution, that TypeSafe has told you nothing measurable about which distributions, and that the gap is yours to close with your own labelled data.