Skip to content
Jason Ehmke
esc
  • Loading index…
↑↓ navigate↵ open
Project
Evaluating AI systems on the judgement calls engineers make: when to act, when to stop, and when to ask
By
Jason Ehmke
Date
5 Nov 2026
Sheets
2 entries, 131 archived
Checked
0 of 0 claims resolved
Fig. 0. The sentence as one attention head sees it: a row per word, shaded by how much it draws on earlier words. Weights set by hand.

I used to manage engineers. Now I study the machines that work beside them.

Point at a word to see which earlier words it pays attention to.

For a few years I wrote short essays about leading engineering teams. Now I build evals: tests of whether an AI system makes the calls a careful engineer would, and whether it knows when it doesn’t know. I write the work up here as I go, with methods, data and my confidence attached.

Research questions

What I’m trying to find out, and the papers I’m reading for each. Point at a question to trace it.
Reading list

  1. Q1 Go, stop, or ask (designing)

    Can a model make the call a careful engineer would, including the call not to act?

    Reading

  2. Q2 Knowing it doesn’t know (reading)

    Does the AI know when it doesn’t know?

    Reading

  3. Q3 Dependability (designing)

    Does it give the same answer the fifth time you ask?

    Reading

  4. Q4 Building the test (designing)

    What does it take to build an eval you can trust?

    Reading

Latest writing

All writing

Latest experiment,

What Breaks When You Split a Job Across AI Agents

Five AI agents worked a bank fraud case in sequence, each passing its work to the next. Giving each agent a list of fields to fill in mattered most: it raised completed cases from 17 to 49 of 60. With JSON records, a chain came close to one agent doing the whole job; with plain text, it fell well short. Chains lost details, added conditions nobody could meet, and passed along wrong conclusions no later agent could check.

31 minute read. Confidence: likely.

  • Accidental Harm From an LLM Agent: Acting Before Reading Its Tools

    LLM agents patching servers caused damage not by breaking policy but by acting before they understood their tools. A small model rebooted servers by accident in nearly half of runs; both models restarted nodes still serving traffic in 88% of rolling-cluster runs. Simple interface changes stopped the reboots.


    Experiment, 32 min

On the reading desk

Reading list