The Technique Behind Jev

What Jev does

TypeSafe AI released Jev last week. You give it a document and a list of questions whose possible answers are fixed in advance. It returns one answer per question, each with a probability.

It never writes a sentence. There is no token-by-token generation, so a call takes one pass through the model instead of dozens. That is why they can charge for input and give output away free. There is no output to charge for.

No paper came with it. No parameter count, no training details, and no calibration numbers, which is odd, because calibrated probabilities are the whole pitch.

Three kinds of question

You do not describe what you want in a sentence and hope. You declare questions, and every question is one of exactly three shapes.

The yes/no type is the one that gives the design away. It returns a probability rather than a boolean, so the threshold lives in your code rather than in the model. You decide what counts as confident enough.

The score type is quietly clever too. It is not a number the model writes down. It is the average across an ordered scale, weighted by how likely each step is, which is why it moves smoothly instead of clustering on round numbers the way a model asked to "rate this 1 to 10" tends to.

What actually changed inside the model

The body of the model is an ordinary transformer: embeddings, then a stack of layers. Nothing there is new.

The change is the last layer. A language model ends with a layer that scores every word in the vocabulary, picks one, appends it, and runs the whole model again for the next word. That final layer is enormous: for Qwen3-0.6B it scores against a 1,024 × 151,936 matrix, about 156 million numbers. In this model that matrix does double duty as the input embedding table, so it stays. The work it does at the output does not.

Stop doing that step and put a layer that produces one score per option in its place. That layer is 1,025 numbers, and it replaces a matrix multiply against the entire vocabulary at every position. The generation loop goes with it, because there is no next word to guess.

This is also where the type guarantee comes from, and it isn’t learned. The scores are turned into probabilities that sum to 1 across the options you listed. There is no leftover probability to assign to anything else, so an answer outside your list isn’t unlikely. There is nowhere for it to come from.

What they claim

70 to 500 milliseconds a call, four cents per million input tokens, nothing for output. Against frontier models on four workflows: security triage, agent-trace review, invoices and customer service.

That is a price argument, not a capability one. It matches the cheaper frontier model at about a seventy-sixth of the cost per decision, and gives up six points to the bigger ones.

Two things to hold onto. The "right answers" in that table were taken to be the average of what two other frontier models predicted, so accuracy there measures agreement with an ensemble rather than with reality. And there is no paper: no parameter count, no word on the training data, and, for a product sold on the honesty of its probabilities, no calibration numbers at all.

Where it is weak

TypeSafe are refreshingly direct about this, and the list is what you would expect of a single pass with no room to work.

• Arithmetic and counting. No intermediate steps, so nowhere to do the arithmetic.
• Dates. Treated as ordinary text. Work out "older than thirty days" yourself and hand over the answer.
• Chains of reasoning. Accuracy falls away as the hops between document and answer pile up.
• Irrelevant detail. A long, noisy document costs accuracy, and you cannot instruct your way around it.
• Hostile input. Not hardened against it, which matters given that jailbreak screening is a pitched use case.

The toy I built

That is everything I could learn about Jev from the outside. The rest of this is what happened when I built a small version to check whether the parts that sounded clever actually do anything.

206,081 parameters. Four transformer layers. It trains on a laptop CPU in forty seconds. All of it is here: about 600 lines, no framework beyond PyTorch.

Nothing here is pretrained. The weights start random and only ever see the made-up data below. Where I quote Qwen3-0.6B figures, that is a size comparison to show what the same change costs on a real model. I never trained it, and none of the results come from it.

The training data is made up, and that is the point. I generate each document from a recipe I choose, so I can compute the correct answer’s exact probability with Bayes’ rule.

That gives me something real data never does: a ceiling. On this task the best possible score is 0.687. When my model scores 0.687 it isn’t mediocre, it’s perfect, and the other 31% is the task being genuinely undecidable.

How it was trained

Four steps, and the second one is the unusual part.

The whole run is twenty thousand documents, ten epochs, forty seconds on a laptop CPU. Testing is five thousand held-out documents, and because the recipe is known I can also compute the best score anyone could possibly get, which makes “is this good?” a question with a number rather than a feeling.

What I found

Two things. One that held up, and one that did not.

Why honest probabilities need honest targets

Two identical models. One trained on the right answer. One trained on the right answer’s probability.

Training on a plain label throws away the fact that the label was uncertain. When two annotators disagree and you record only the majority vote, you have deleted the most useful thing you had.

One honest caveat: plain labels aren’t broken in principle. With unlimited data both models end up in the same place. The problem shows up with limited data, where a model with room to spare drives its own confidence to 100% and stays there.

New answer lists: right answer, wrong confidence

I trained the model on a four-way question only, then asked it two-way and three-way questions it had never seen.

The label survives. The probability does not, and the probability is the reason you’d build this instead of an ordinary classifier.

What this doesn’t show

This is a 206,000-parameter model on invented words. It does not tell you how Jev is actually built, because TypeSafe has confirmed nothing. What I measured is how this shape of model behaves on a task I made up, and a bigger one on real text may well behave differently.

What it does establish is narrower and still useful. Training on probabilities rather than labels reaches the best score the task allows and stays honest about its confidence, across every difficulty I tried. Answer lists that change between calls give you the right answer, and a confidence that does not survive an answer list the model has never seen.

Finding that out cost twenty cents.

The obvious next step is the one I cannot run yet. All of this is a tiny model on invented words, and the interesting version is a real one on real text: a few hundred million parameters, actual support tickets and moderation data, and the same four questions asked again. That needs a GPU, and my cloud quota request for one is still sitting in a queue. When it clears I will write up what changes.

The model, the four ablations and every number in this post are at github.com/sahasrarjn/system-one. python -m tiny.run reproduces the calibration result; python -m tiny.ablate runs all four. Both are CPU-only.