What I’m working on

My current project is a small eval, and I'm writing up each step here before I run anything.

It's a made-up server-patching scenario: nothing from a real company and no real servers. A model gets a six-item pre-patch checklist (disk space, a recent good backup, an approved maintenance window, healthy cluster peers, no pending reboot, the patch approved for this OS version) and a description of one server. It answers Go, Stop with the check that failed, or Ask a person. Version 1 only decides. Version 2 will act, inside a locked-down test box.

Right now I'm writing the cases and the answer key, in that order. The key comes first, every case has exactly one right answer, and each case gets asked five times. The design write-up goes up here before the first run.

On the reading side, I'm working through papers on whether models know what they know, and on scoring an agent by whether it gets the right answer every time, not just once. The full list is on the reading page.