Welcome back. Two months ago, a startup named TypeSafe went viral after dropping Jev, an AI model that cannot write a single word. Since then, OpenAI, Microsoft, Cloudflare, AWS, and Google have all shipped their own versions. Most of them make money every time you use an LLM, and now they’re racing to sell you something much cheaper.

Today, you'll see what decision models are good for, then set up 3 workflows that put one to work. Let's get started:

Today’s Brief

  • What decision models are and how they work

  • Workflow #1: Catch breaking PRs before review

  • Workflow #2: Replace a costly LLM the way Amazon did

  • Workflow #3: Pick a decision model and build evals

DEEP DIVE

Click here to see how PostHog runs decision models in production.

What it is. A decision model answers a closed question about whatever you hand it, such as a diff, a log line, or an agent trace. It picks from the answers you allow and tells you how sure it is, so your code gets a value it can branch on.

How it works. Jev, Gemini Decision 1, and OpenAI's Decisions API all take the same three kinds of question, and each answer comes back with a probability:

  • Yes/no. Does this PR change a public API? → Yes, 0.94

  • Choice. Which team should review it? → Payments, 0.88

  • Score. How risky is this change, from 1 to 5? → 4, 0.71

Why the number matters. In a well-calibrated model, answers scored 0.8 are right about 80% of the time. So you set a cutoff. Above it, your code acts on the answer, and below it, the case goes to an LLM or a person. Not every model's scores are that honest, so test them on your own data before you trust them.

Where it fits. Use one for a narrow question with a clear answer, where an LLM is too slow or too expensive to ask every time. Skip it for anything that takes several steps of reasoning, and keep those jobs for an LLM.

Here are 3 workflows worth trying out today:

WORKFLOW #1: PR TRIAGE

How to catch breaking PRs before review

Engineers tend to review small PRs faster and with less scrutiny than large ones. If a diff is only ten lines and labeled "cleanup," it usually gets approved in thirty seconds. 

That speed creates a blind spot. Those few lines can change a default or a function's behavior and break another team's build.

To catch this before code merges, a Vercel engineer built a script that checks any PR you point it at. It asks OpenAI's Decisions API two simple questions: What kind of review does this change need? And will existing callers break?

Here's how you can implement it:

  • Step 1: Install Node 22 or later.

  • Step 2: Set "OPENAI_API_KEY" and "GITHUB_TOKEN" in your terminal. The GitHub token only needs "Pull requests: read" access.

  • Step 3: Copy the script from Vercel's guide and save it as "triage-pr.mjs".

  • Step 4: Run this command on a real PR from your repo. It handles PRs with up to 100 changed text files:

node triage-pr.mjs OWNER REPO PR_NUMBER
  • Step 5: Check the JSON it prints. Under "answers" you get "review_track" and "migration_needed", the probability that callers have to change.

  • Step 6: Show the suggested track to your reviewers and log where they disagree. Automate only after you've seen those cases.

Your opinion matters!

This is our eighth Sunday Special. It’s a quick 5 min email designed to show how to use the latest and most relevant AI tools to code faster. Let us know what you think of this email in the poll at the bottom of this email (or by replying to this email).

We’re writing this email just for you, so your feedback will shape what we write next!

WORKFLOW #2: MODEL SWAP

How Amazon saves millions with its own decision model

Two Amazon engineers, Carr Onstott and Mehul Jaiswal, were paying Claude Sonnet about two cents to label each ticket in one of the company's largest internal queues.

They replaced it with Strands Decider, a small decision model built at AWS. The cost per ticket dropped by 98%, and answers came back almost five times faster.

The team made the swap carefully. Both models ran side by side on live traffic first, and switching back took about 45 seconds.

Here's how you can implement it:

  • Step 1: Find an LLM call in your stack that only picks from a fixed list, like a label or a queue.

  • Step 2: Install Strands Decider with "pip install strands-decider". The 2B model in its README runs on a GPU, an Apple-silicon Mac, or a CPU.

  • Step 3: Start its server with "strands-decider serve" and the model name. Your rules only work through the server, not through the one-line "ask" command.

  • Step 4: Rewrite each option as a one or two sentence rule, and add a "none" option. Amazon did this for its 20 labels.

  • Step 5: Add a switch with three settings: old model only, both models, and new model only. Run "both" on live traffic, and keep using the old model's answers.

  • Step 6: Review every case where the two disagree and fix your rules, then flip the switch. Amazon's write-up says this taught the team more than offline testing did.

WORKFLOW #3: EVALS

How to pick a decision model and build evals around it

There are now more than a dozen decision models to choose from. The only way to know which one works for you is to test them on your own data.

AI evals expert Hamel Husain ran that test on six of them, using 98 real conversations from an AI agent. Only three were reliable enough to catch the agent's mistakes.

His guide shows how to run the same test yourself:

  • Step 1: Collect real conversations from your agent and label each one pass or fail. Hamel used 98.

  • Step 2: Write one pass/fail question for a single type of failure. Hamel's own judge prompt is in the appendix of his post.

  • Step 3: Set your minimums before you compare models. Hamel required the judge to catch 80% of real failures and to be right on 80% of its flags.

  • Step 4: Run two or three decision models on the same conversations. Save each model's scores next to your labels.

  • Step 5: Paste the prompt under "Tuning the decision threshold" in his post into your coding agent to find the best cutoff for each model. Drop any model that still misses your minimums.

  • Step 6: If you plan to treat a score as confidence, run his calibration prompt too. It sits in the same post.

MORE IDEAS TO TRY
  • Run only the end-to-end tests a PR can actually break (link).

  • Stop prompt injections in GitHub issues from reaching your coding agent (link).

  • Write UI tests that don't break when the wording changes (link).

  • Let a cheap model grade your evals and call Opus only when it's unsure (link).

  • Match a PR's UI screenshot to the parts of your system it changed (link).

  • Compare three decision models on your own video footage (link).

  • Train your own decision model on 4GB of VRAM (link).

STARTING POINT

How to start today. Run the PR triage script on one real PR from your repo. Within ten minutes, you'll see how a decision model reads a diff. If you have the afternoon, label a batch of conversations from your own agent and turn your biggest failure type into one pass/fail question. By tonight, you'll know whether a decision model can be trusted with that call.

Whenever you're ready to dive deeper

If you enjoyed this newsletter, you should definitely check out our IT and cybersecurity newsletters. Subscribe using the links below:

We put together 20+ hands-on guides covering coding agents, agentic engineering, and leadership frameworks to help you level up your career. You can get them here.

Advertise with us: Join companies like Google, IBM, and Datadog. Showcase your product to our 350K+ engineers and 150K+ followers on socials. Get in touch.

What did you think of today's newsletter?

Your feedback helps us create better emails for you!

Login or Subscribe to participate

You can also reply directly to this email if you have suggestions, feedback, or questions.

Until next time — The Code team