Jev is useful without training data, but its probabilities aren't calibrated for you

Jev is TypeSafe's first "System One Model". Instead of generating text, it takes unstructured input and returns typed decisions from a set of outputs defined in advance, each with a probability attached. TypeSafe says the answers come with calibrated probabilities. In a post published on September 23, 2026, Alex Molas argues that Jev is useful, but that the calibration claim cannot hold in general: its outputs should be treated as scores, not probabilities.

Useful without training data

You can point Jev at any classification problem without collecting training data and still get reasonable results. Many people on Twitter called it "just a fine-tuned BERT", but fine-tuning BERT requires data, so where data is missing Jev is a good alternative. It is also a universal classifier, while a fine-tuned BERT only handles its own task. And that is exactly why its probabilities cannot be calibrated for you.

Calibration is a property of your data

TypeSafe says Jev was trained with RLCD, reinforcement learning for calibrated decisions, and that the probabilities it produces are calibrated. Molas disagrees: a model can be calibrated on TypeSafe's data and still be miscalibrated on yours. A calibrated model satisfies one condition: for any predicted probability p, the true probability of the positive class given that prediction is p. If Jev scores an email at a 0.7 spam probability, such emails should be spam about 70% of the time.

Calibration is not only a property of the model, but of your data distribution as well. The same model can be calibrated on one dataset and not on another. Two companies can define spam the same way and still have very different data; Jev returns the same probability to both for the same input and prompt. Even if RLCD trains Jev to be calibrated on its training and evaluation distribution, its probabilities may not stay calibrated on your production traffic.

The problem runs deeper than drift

The author also found examples where Jev says a fair coin lands heads with probability 0.92. The true probability is stated in the prompt, and the model still does not report it, which is worse than ordinary distribution shift. If the meaning of the probabilities changes with the primitive you use, what do "calibrated probabilities" mean?

Recalibrate, or treat the output as a score

If you need calibrated probabilities, you will still have to recalibrate Jev's numbers on your own data. The good news is that this is cheap: a few hundred labeled examples from your own traffic are often enough for a Platt scaling on top of Jev's scores.

Treat Jev's outputs as good scores, which rank examples well, rather than good probabilities. If your system depends on the actual number, for thresholds, expected costs or model combinations, measure calibration on your own data first.