tldr is that the post asks how much of our ability to plan comes from our brains being designed to plan, and how much is purely learned (by social imitation of other people’s planning, or explicit instruction from others on how to plan). It answers that a surprising amount is purely learned. (This summary does not do the post justice and you should really go and read it.)
Below is a copy of a comment I left:
Thank you for writing this!
I have been developing similar kinds of suspicions for a little while (that even though we know how to program algorithms like MCTS, that doesn’t mean human brains have a built-in analogue, and that many things like this are learned, not implemented on an architecture level), but I was missing a lot of the pieces you have here, and reading this really crystallized it for me.
A priori, we would expect the first, naturally arising, agent to attain this agency in the stupidest, most hacky way possible.
Re this, as you note later, LLM planning capabilities also arise from learning some kind of prior over chains of thought, and then applying a bit of “cherry on top” RL. So it also seems that the first general[1]artificially arising agent also attained agency in the stupidest, most hacky way possible, exactly as humans did. We are, in this respect at least, following in the footsteps of evolution on our quest to build AI.
I’m not sure the extent to which this affects the AI ruin hypothesis, and how powerful we can expect intelligence to become. I share your opinion that it must change the story somehow, but at the same time, strong planning abilities can be dangerous, regardless of whether they’re learned or architectural. It’s unclear whether the ultimate, optimally-designed successor agent would have built-in planning, or would have to learn to plan, but I find myself expecting it to be extraordinarily powerful either way.
You mostly describe the social aspect of learning to plan here, but I do think that it’s a mix of both social imitation and RL. Eg someone learning to play chess would socially absorb the idea of looking ahead several moves, and various heuristics for evaluating board positions, but they’d also just improve from playing several games and seeing what planning techniques are actually effective.[2]
Nothing left to elevate the hypothesis of a simple core structure of general planning or agency to our attention.
Re this, I think it is slightly overstated. All the old structure of VNM rationality still exists, it’s just intractable to compute in the real world, just like we already knew it was. I think what changes here is our estimate of how well an algorithm of a given cost can approximate it. In particular, it starts looking harder to approximate very well, and we might start hoping instead that we can get good real-world results from things that are actually quite far from VNM rational.
Questions:
World modelling does seem pretty architectural to me right now. There is a move that is often used in planning where we “deform” our world model to correspond to some hypothetical scenario, and get it to spit out predictions for what values various variables are likely to take. I tend to think that this kind of “deformation” into fake scenarios is also a built-in ability, do you agree? Or an alternate hypothesis could be that we have a built-in direct sensory world model, and also a learned far-mode world model, and only the latter is deformable?
This all makes me think that the kinds of modifications to LLMs needed to trigger the singularity or whatever, are not actually that big? Like, if planning is learned anyways, we don’t need to drastically modify the transformer architecture to wedge some kind of planning into it, RL on chain of though is already all we need. Given that “big dumb blob of intelligence” is going to be the paradigm that wins, are our options for getting friendliness basically just “use good training data” and “make the big blob more sample-efficient”? Does increasing sample-efficiency even help?
This is a great article. I hope you are planning to crosspost to lesswrong?
Maybe RL in the weak sense. Humans don’t really seem to be able to directly reward ourselves purely mentally in the same way as we’d get a direct reward for eating a cookie while hungry. So arguably the kind of tuning that happens here could be better classified as epistemic, though it does affect the distribution of planning actions like a true reward-update would.
Rhymes with Is Clickbait Destroying Our General Intelligence? I suspect that writing a good answer to a silly question on r/whowouldwin, arguing with a friend about what Light Yagami’s next move should be, and the art of best response are all pointing in a similar direction. Likewise, I suspect some amount of personality and internal ‘self-hood’ is built socially.
I thought this recent substack post was very insightful: https://eliasschmied.substack.com/p/social-agency
tldr is that the post asks how much of our ability to plan comes from our brains being designed to plan, and how much is purely learned (by social imitation of other people’s planning, or explicit instruction from others on how to plan). It answers that a surprising amount is purely learned. (This summary does not do the post justice and you should really go and read it.)
Below is a copy of a comment I left:
Fairly general, at least. I specified this because in many specialized domains agents do use a built-in search process. Eg. alpha go using MCTS.
Maybe RL in the weak sense. Humans don’t really seem to be able to directly reward ourselves purely mentally in the same way as we’d get a direct reward for eating a cookie while hungry. So arguably the kind of tuning that happens here could be better classified as epistemic, though it does affect the distribution of planning actions like a true reward-update would.
Rhymes with Is Clickbait Destroying Our General Intelligence? I suspect that writing a good answer to a silly question on r/whowouldwin, arguing with a friend about what Light Yagami’s next move should be, and the art of best response are all pointing in a similar direction. Likewise, I suspect some amount of personality and internal ‘self-hood’ is built socially.
The post is now here on LW too: https://www.lesswrong.com/posts/xopGsfQxiLcjXEkbE/social-agency