Skip to content
Jason Ehmke
esc
  • Loading index…
↑↓ navigate↵ open
Project
Evaluating AI systems on the judgement calls engineers make: when to act, when to stop, and when to ask
By
Jason Ehmke
Date
4 Nov 2026
Sheets
2 entries, 131 archived
Checked
0 of 0 claims resolved
Fig. 0. The sentence as one attention head sees it: a row per word, shaded by how much it draws on earlier words. Weights set by hand.

I used to manage engineers. Now I study the machines that work beside them.

Point at a word to see which earlier words it pays attention to.

For a few years I wrote short essays about leading engineering teams. Now I build evals: tests of whether an AI system makes the calls a careful engineer would, and whether it knows when it doesn’t know. I write the work up here as I go, with methods, data and my confidence attached.

Research questions

What I’m trying to find out, and the papers I’m reading for each. Point at a question to trace it.
Reading list

  1. Q1 Go, stop, or ask (designing)

    Can a model make the call a careful engineer would, including the call not to act?

    Reading

  2. Q2 Knowing it doesn’t know (reading)

    Does the AI know when it doesn’t know?

    Reading

  3. Q3 Dependability (designing)

    Does it give the same answer the fifth time you ask?

    Reading

  4. Q4 Building the test (designing)

    What does it take to build an eval you can trust?

    Reading

Latest writing

All writing

Latest experiment,

What Breaks When You Split a Job Across AI Agents

Five LLM agents worked a bank fraud case in sequence, each passing its finished work to the next. What helped chains most was giving each agent a list of fields to fill in. Free text with the same field list as the JSON records completed 49 of 60 cases, against 17 without it, and with that list a chain matched one strong agent. For the weaker model, JSON added more on top. Without a list, one agent beat a free-text chain in every run. Free-text chains that failed had dropped details and added conditions nobody could meet. A rule given to an agent without its definition stalled payments until the definition was added, and wrong work no later agent could check went straight through.

33 minute read. Confidence: likely.

  • Accidental Harm From an LLM Agent: Acting Before Reading Its Tools

    LLM agents patching servers caused damage not by breaking policy but by acting before they understood their tools. A small model rebooted servers by accident in nearly half of runs; both models restarted nodes still serving traffic in 88% of rolling-cluster runs. Simple interface changes stopped the reboots.


    Experiment, 32 min

On the reading desk

Reading list