Skip to content
Jason Ehmke
esc
  • Loading index…
↑↓ navigate↵ open
Project
Evaluating AI systems on the judgement calls engineers make: when to act, when to stop, and when to ask
By
Jason Ehmke
Date
18 Oct 2026
Sheets
0 entries, 131 archived
Checked
0 of 0 claims resolved
Fig. 0. The sentence as one attention head sees it: a row per word, shaded by how much it draws on earlier words. Weights set by hand.

I used to manage engineers. Now I study the machines that work beside them.

Point at a word to see which earlier words it pays attention to.

For a few years I wrote short essays about leading engineering teams. Now I build evals: tests of whether an AI system makes the calls a careful engineer would, and whether it knows when it doesn’t know. I write the work up here as I go, with methods, data and my confidence attached.

Research questions

What I’m trying to find out, and the papers I’m reading for each. Point at a question to trace it.
Reading list

  1. Q1 Go, stop, or ask (designing)

    Can a model make the call a careful engineer would, including the call not to act?

    Reading

  2. Q2 Knowing it doesn’t know (reading)

    Does the AI know when it doesn’t know?

    Reading

  3. Q3 Dependability (designing)

    Does it give the same answer the fifth time you ask?

    Reading

  4. Q4 Building the test (designing)

    What does it take to build an eval you can trust?

    Reading

Latest writing

All writing

Nothing published here yet. The first essay is close; subscribe at the bottom of the page if you want it by email.

On the reading desk

Reading list