Reading
The reading desk
Primary sources first. What I’m working through now, what’s next, and the papers I keep going back to. Books from the leadership years are at the bottom; they still hold up.
On the desk
3 papers
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmark↗Evaluation
Rein et al., COLM 2024
The “science test” kind of eval: expert-written questions, one checked answer each. The opposite end from a patching decision.
- 2022Language Models (Mostly) Know What They Know↗Evaluation
Kadavath et al.
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains↗Agents
Yao et al.
Its pass^k metric asks whether an agent succeeds on all k tries, which is the dependability question.
Up next
6 papers
- 2024Are We Done with MMLU?↗Evaluation
Gema et al.
- 2022Teaching Models to Express Their Uncertainty in Words↗Evaluation
Lin, Hilton & Evans, TMLR
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models↗Evaluation
Wang et al., ICLR 2023
- 2024Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet↗Interpretability
Templeton et al., Transformer Circuits
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?↗Evaluation
Jimenez et al., ICLR 2024
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena↗Evaluation
Zheng et al., NeurIPS 2023
Read
9 papers
- 2022ReAct: Synergizing Reasoning and Acting in Language Models↗Agents
Yao et al., ICLR 2023
- 2022Toy Models of Superposition↗Interpretability
Elhage et al., Transformer Circuits
- 2022Alignment
- 2022Training Language Models to Follow Instructions with Human Feedback↗Alignment
Ouyang et al., NeurIPS 2022
- 2022Training Compute-Optimal Large Language Models↗Scaling
Hoffmann et al., NeurIPS 2022
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models↗Agents
Wei et al., NeurIPS 2022
- 2021Evaluation
- 2020Scaling Laws for Neural Language Models↗Scaling
Kaplan et al.
- 2017Attention Is All You Need↗Foundations
Vaswani et al., NeurIPS 2017
The earlier shelf
Books from the leadership years. Links are Amazon affiliate links.
