Tutorial 08/13PatternsIntermediate10 min

The cascade: Jev in front of an LLM

The highest-value place for a System One model is not doing the work. It is deciding who should. Most requests that reach an LLM today did not need one, and the cheapest token is the one you never generate.

By the end you'll be able to

Build a three-way fork: code, model, human
Route between fast and expensive models by intent
Do the cost arithmetic on a real volume
Measure the whole workflow rather than one cheap call

On this page

The three-way fork
The router
Routing between models
Do the arithmetic before you believe the pitch
Where the cascade goes wrong
Next

The three-way fork#

The clearest statement of the pattern came from heise's launch coverage, describing customer service: "A simple question about order status can go to normal program code, a product question to a language model, and a complex or uncertain case to a human."

Three destinations, three costs, three latencies. The routing decision itself needs to be fast enough to be invisible and cheap enough to run on every request — which is precisely the shape of judgment Jev exists for.

Destination	When	Marginal cost
Deterministic code	The intent maps onto a lookup or a standard procedure	Effectively zero
A specialist LLM	The answer needs generation, explanation or open-ended reasoning	Cents
A human	Low confidence, high stakes, or an unusual edge case	Minutes of someone's day

The router#

cascade.py

from typesafe_sdk import Choice, Score, TypeSafeClient

client = TypeSafeClient(model="jev-1.13.0")

def handle(message):
    r = client.system_one(
        state=message,
        questions={
            "intent": Choice(
                instructions="Primary intent of this message",
                criteria={
                    "order_status": "Asking about an existing order",
                    "product_question": "Asking about a product",
                    "return_exchange": "Wants to return or exchange",
                    "complaint": "Unhappy, wants resolution",
                    "other": "None of the above",
                },
            ),
            "complexity": Score(
                instructions="How complex is this to resolve",
                criteria=[
                    "Simple lookup or standard procedure",
                    "Requires judgment or multiple steps",
                    "Unusual edge case, escalation needed",
                ],
            ),
        },
    )

    intent = r.answers["intent"]
    complexity = r.answers["complexity"]

    # The uncertainty floor comes first, before any branch.
    if intent.confidence < 0.5 or intent.choice == "other":
        return route_to_human(message)

    if intent.choice == "order_status":
        return lookup_order(message)            # no model involved at all

    if intent.choice == "product_question":
        return handle_with_llm(message, PRODUCT_SPECIALIST)

    if intent.choice == "return_exchange":
        return handle_with_llm(message, RETURNS_SPECIALIST)

    if intent.choice == "complaint":
        if complexity.score > 1 or complexity.confidence < 0.5:
            return route_to_human(message)
        return handle_with_llm(message, COMPLAINT_RESOLUTION)

Routing between models#

The same idea applies one level down: when the request does need a generative model, which one? LangChain shipped a langchain-typesafe integration on launch week with exactly this middleware. Note the experimental namespace — this is three-day-old software.

router.py

from langchain.agents import create_agent
from langchain_typesafe.experimental.middleware import (
    ModelChoice,
    ModelRouterMiddleware,
)

router = ModelRouterMiddleware(
    choices={
        "fast": ModelChoice(
            model="openai:luna",
            criteria="Direct lookups, extraction, and localized changes.",
        ),
        "powerful": ModelChoice(
            model="openai:sol",
            criteria="Architecture and high-stakes decisions.",
        ),
    },
    instructions="Choose the least costly model that can complete the task.",
)

agent = create_agent("openai:gpt-5.6-luna", middleware=[router])

Jev reads the latest user message, picks a model, and that model handles the run. The probabilities and confidence stay available in agent state, so your own code can still override the pick — which you will want for anything irreversible.

Do the arithmetic before you believe the pitch#

Take a million support messages a month, averaging 1,000 billable input tokens each.

Step	Figure
Jev input price	$0.042 per million tokens (TypeSafe's published rate)
Tokens routed	1,000,000 messages × 1,000 tokens = 1bn tokens
Routing layer cost	≈ $42
Output tokens	$0 — not charged

The Rundown AI ran this same sum on launch week and landed on the same $42.

Forty-two dollars to triage a million messages is genuinely striking, and it is also not your bill. Your bill is the LLM calls that survive the routing, plus the human review queue that low confidence feeds, plus the engineering time to maintain a policy layer that did not exist before.

Instrument the split from day one. If you know what fraction of traffic lands on each branch, you can price the design; if you only know your Jev bill, you know the least interesting number in the system.

cascade.py

from collections import Counter

ROUTES = Counter()

def record(branch, answer):
    ROUTES[branch] += 1
    # The versioned model ID that actually answered — log it, always.
    metrics.observe(
        "jev.route",
        branch=branch,
        model=answer_model_id,
        confidence=round(answer.confidence, 2),
    )

Where the cascade goes wrong#

No other option. Every message must land on one of your intents, so unfamiliar ones land on the nearest — confidently. Always include an explicit escape route.
One threshold for every branch. Reading an order status and issuing a refund should not clear the same bar. See confidence gating.
Routing on state you did not filter. A whole conversation history in the state when only the latest message matters is context rot, and it costs accuracy.
Trusting the router with untrusted text. Jev does not treat state as hostile. A message engineered to route itself to the cheap automated branch will succeed if nothing downstream checks.
Forgetting the router is a dependency. When TypeSafe returns 429 or 529, what happens? The safe default is to fall through to the expensive path, not to fail the request.

cascade.py

from typesafe_sdk import TypeSafeAPIError, TypeSafeAPIConnectionError

def route_or_fallback(message):
    try:
        return handle(message)
    except (TypeSafeAPIError, TypeSafeAPIConnectionError):
        # Degrade towards the safe, expensive path rather than dropping work.
        return handle_with_llm(message, GENERALIST)

Next#

The cascade puts Jev in front of your models. The next tutorial puts it inside an agent loop, where it decides which tool runs and whether that is a good idea. Continue to Building an agent harness.

Sources for this page

Last reviewed 2026-09-18. Jev is days old and moving — where a claim is TypeSafe's own rather than independently verified, this page says so in the sentence that carries it.