Reading the Numbers: Scores, Confidence and Thresholds

Jev and OpenAI's Decisions API both return numbers instead of prose, which is their whole appeal: code can act on a number. They are also easy to misuse. We see two mistakes over and over in code reviews. The first is reading a Score as if it were a level. The second is picking a confidence threshold because it sounds safe.

Both are easier to see than to explain, so this article is mostly two labs.

A score is an average, not an answer

A Score question asks the model to place something on ordered levels: not urgent, low, high, critical. Both Jev and OpenAI return the same three things: a probability for each level, a confidence, and a single score.

The score is defined the same way by both vendors: each level's 0-based index times its probability, summed. Probabilities of 0.1, 0.7 and 0.2 on levels 0, 1 and 2 give 0 × 0.1 + 1 × 0.7 + 2 × 0.2 = 1.1. That is why scores are fractional, and why OpenRouter's documented example returns 1.99 rather than 2.

Averages hide shape. Pick Torn: Low or Critical below, or drag the bars yourself:

Half the probability on "Low" and half on "Critical" averages to exactly 2.0, which rounds to "High", the one level the model put no probability on. Code that does if (score >= 1.5) page_oncall() treats "I am torn between nothing much and an emergency" as a confident "high". Sometimes that is the right call. It should never be an accident.

What we do instead:

• Branch on the distribution for actions that matter. For paging, refunds or anything irreversible, read probabilities directly: "page if P(critical) > 0.3" says what you mean.
• Use the score for ranking and sorting. A probability-weighted position is a fine sort key for a queue. It is a poor trigger.
• Treat low confidence on a Score as a signal. A spread or two-humped distribution usually means the input is ambiguous or the levels overlap. Both are worth a look before you blame the model.

The confidence in the lab is our own stand-in, one minus the normalized entropy of the distribution. Neither TypeSafe nor OpenAI publishes the formula behind its confidence field, so do not hard-code assumptions about how it behaves; measure it.

A threshold is a price, not a feeling

The second mistake is choosing a confidence threshold like 0.9 because it sounds careful. A threshold is a trade between two costs: the cost of acting on a wrong answer, and the cost of sending a right one to a person. The best threshold depends on those two numbers, and nothing else is a good reason.

The lab below simulates 400 answers from a perfectly calibrated model. Each square is one answer. Above the threshold, code acts; below it, a person reviews. Move the threshold, then change what a mistake and a review cost.

Play with the costs and the optimal threshold moves a long way. When a wrong action costs twenty times a review, the best threshold sits high and a lot of traffic goes to people. When mistakes are cheap and reviews are expensive, it drops, and automation does most of the work. The same model, the same answers, very different right answers.

This is also why we keep thresholds per action, not per model:

const THRESHOLD = {
  route_to_team: 0.6, // a wrong route costs a hand-off
  auto_reply: 0.85, // a wrong reply costs a confused customer
  issue_refund: 0.97, // a wrong refund costs money
} as const;

Calibration is something you measure

The simulation cheats in one way: it is perfectly calibrated by construction, so a confidence of 0.8 is right 80% of the time. Real models are only approximately calibrated, and the approximation depends on your data. TypeSafe trains Jev specifically for calibration and is careful to say it describes groups of predictions, not any single answer. OpenAI's guide says to pick thresholds from labelled examples from your own application, and to weigh false positives against false negatives.

The procedure is the same for both:

1. Label a few hundred real inputs. A spreadsheet is fine. Cover the edge cases you actually see.
2. Run them through the model and bucket by confidence. For each bucket, 0.5 to 0.6 up to 0.9 to 1.0, count how often the answer was right.
3. Put your costs on it. Multiply errors above each candidate threshold by the cost of a wrong action, reviews below it by the cost of a review, and take the cheapest.
4. Pin the model version, and redo this when it changes. Thresholds are tuned to one model's probabilities. A new release, or a switch from Jev to OpenAI, shifts them. Pin jev-1.14.0 rather than jev-latest, and treat an upgrade as a re-tune.

For the background on why confidence and probability are different numbers, read Confidence vs probability. For what happens to the messages above the threshold, see Jev decides, Fable 5.2 writes.

If you want a second pair of eyes on thresholds before they go live, that is part of our AI engineering work. Get in touch.

FAQ

Why is a Jev or OpenAI Decisions score not a whole number?

The score is a probability-weighted average of the 0-based level indices, so it falls between levels when the probability is spread. A score of 1.99 means nearly all the probability is on level 2.

Should I round the score to pick a level?

Not for actions that matter. A score near 2.0 can come from a distribution split between levels 1 and 3. Read the per-level probabilities when the decision triggers something costly or irreversible.

How do I choose a confidence threshold?

From costs and labelled data. Estimate what a wrong automatic action costs and what a human review costs, measure accuracy by confidence on a few hundred labelled examples, and pick the threshold with the lowest total cost. Use separate thresholds for actions with different risks.