Ask for features, not answers

The finding#

An independent benchmark run in launch week put Jev against 2,000 emails and asked it to judge whether each was a phishing attempt. Jev's own verdict scored 62.6%, against Claude Haiku 4.5's 81.3% — a clear loss, statistically significant.

The same author then ignored Jev's verdict and instead fed five narrow signal questions into a plain logistic regression. Same model, same emails, same API. Accuracy: 95.1%, AUROC 0.988.

Why this happens#

Jev is trained for System One judgments — the call a knowledgeable person makes in a second given the right context. "Is this phishing?" is not that. It is a conclusion that depends on a dozen observations, weighed against each other. TypeSafe's own docs warn about exactly this shape: "Analyze this message and determine the best course of action" is not [a good question]. That needs slow reasoning, and it is a signal to break the task into small questions.

Asked for a verdict, the model has to do the weighing somewhere you cannot see, cannot adjust, and cannot audit. And it is not good at it — multi-hop indirection is one of its nine documented failure modes.

Asked to a founder directly on X, the answer was the same. Someone asked whether they could send their whole codebase and ask "is it safe to deploy?" Diogo Almeida's reply:

Rewriting a question as signals#

The move is mechanical once you see it. Take the verdict you want, and ask: what would I have to notice to reach it? Each of those observations is a question. The verdict is arithmetic.

Two developers reported the same effect in the same week, in different domains. One found that "it cannot answer 'quality', but it can answer 'duplicated_code' very well". Another, working on prose:

In code#

from typesafe_sdk import Noul, Score, TypeSafeClient

client = TypeSafeClient(model="jev-1.13.0")

SIGNALS = {
    "domain_mismatch": Noul(
        instructions=(
            "Does the sender address in `headers.from` use a domain other than "
            "the organisation named in `body`?"
        ),
    ),
    "asks_for_credentials": Noul(
        instructions="Does `body` ask the reader to enter a password, code, or payment detail?",
    ),
    "manufactured_urgency": Noul(
        instructions=(
            "Does `body` claim the reader must act within a short deadline "
            "to avoid a negative consequence?"
        ),
    ),
    "link_text_mismatch": Noul(
        instructions="In `links`, does any anchor text name a different domain than its href?",
    ),
    "generic_greeting": Noul(
        instructions="Does `body` open without using the recipient's name?",
    ),
}

def signals(email) -> dict[str, float]:
    r = client.system_one(state=email, questions=SIGNALS)
    return {key: r.nouls[key].noul for key in SIGNALS}

Now the verdict is yours, and it is ordinary software. Start with weights you can defend in a meeting:

import math

WEIGHTS = {
    "domain_mismatch": 2.4,
    "asks_for_credentials": 1.9,
    "manufactured_urgency": 1.1,
    "link_text_mismatch": 2.1,
    "generic_greeting": 0.4,
}
BIAS = -3.2

def phishing_score(s: dict[str, float]) -> float:
    z = BIAS + sum(WEIGHTS[k] * s[k] for k in WEIGHTS)
    return 1 / (1 + math.exp(-z))          # logistic, nothing exotic

And once you have a few hundred labelled examples, stop guessing the weights and fit them. This is the step that took that benchmark from 62.6% to 95.1%, and it is four lines:

from sklearn.linear_model import LogisticRegression

X = [[s[k] for k in SIGNALS] for s in collected_signals]   # one row per email
model = LogisticRegression().fit(X, labels)

print(dict(zip(SIGNALS, model.coef_[0])))   # your weights, learned

The counterintuitive part#

Every instinct from classical ML says to keep the number of questions small, because more outputs means more chances to be wrong. Jev inverts that, and practitioners noticed immediately:

The reason is structural. Questions in a request are evaluated independently and in parallel, so a wrong answer on generic_greeting does not corrupt domain_mismatch. Errors do not compound across questions the way they compound across the steps of a chain of thought. They land as noise in separate features, and a weighted combination is exactly the right tool for averaging noise out.

Where this leaves the verdict field#

You will still use Choice and Score for genuine decisions — routing a ticket to one of four teams is not a hidden aggregation, it is a real classification. The rule is narrower than "never ask for a conclusion":

• Ask directly when the answer is an observation: which language is this, which team owns this, is there a date in this field.
• Ask for signals when the answer is a judgment built from several observations: is this risky, is this high quality, should we approve this.
• The test: if you would need to explain your reasoning to justify the answer, ask for the inputs to that reasoning instead, and do the reasoning yourself.

Next#

This pattern and composite scoring are the same idea seen from two angles — one starts from a verdict you want to decompose, the other from dimensions you want to weigh. Read them together. Then what the numbers actually say for the rest of the independent evidence.