Agency requires reasoning about the consequences of one’s actions. “I need to do such-and-such, to get to my goal.” This requires counterfactual, causal reasoning.
Have you ever tried to explain the difference between correlation and causation to someone who didn’t understand it? I’m not convinced that this is even something humans innately have, rather than some higher-level correction by systems that do that.
A computer chess engine trained exclusively on one format for representing the game would generally not be able to transfer its knowledge to a different format.
You can hook a chess-playing network up to a vision network and have it play chess using images of boards—it’s not difficult. Perhaps a better example is that language models can be easily coupled to image models to get prompted image generation. You can also translate between language pairs that didn’t have direct translations in the training data.
thus we do not know how to build machines that can pursue goals coherently and persistently
This post seems rather specific to LLMs for how much it’s trying to generalize; I think there’s been more progress on that than Sarah seems to realize.
Have you ever tried to explain the difference between correlation and causation to someone who didn’t understand it?
Understanding a high-level description of an abstract concept is different from having more low-level cognitive machinery that can apply the concept intuitively; you can have one without having the other (this goes in both directions). One classic example is that if you catch a flying ball with your hand, your brain needs to do something like solving a set of differential equations in order to predict the path of the ball… but this doesn’t imply that the people who were good at catching balls would be any good with solving explicit sets of differential equations. (Nor that people who were good at math would be any good at catching balls, for that matter.)
‘Have you ever tried to explain the difference between correlation and causation to someone who didn’t understand it? I’m not convinced that this is even something humans innately have, rather than some higher-level correction by systems that do that.’
You are outside and feel wind on your face. In front of you, you can see trees swaying in the wind. Did the swaying of the trees cause the wind? Or did the wind cause the trees to sway?
The cat bats at a moving toy. Usually he misses it. If he hits it, it usually makes a noise, but not always. The presence of the noise is more closely correlated with the cat successfully hitting the toy than the cat batting at the ball. But did the noise cause the cat to hit the ball or did the cat batting at the ball cause the hit?
The difference between correlation and causation is something we humans have a great sense for, so these questions seem really stupid. But they’re actually very challenging to answer using only observations (without being able to intervene).
You can hook a chess-playing network up to a vision network and have it play chess using images of boards—it’s not difficult.
I think you have to be careful here. In this setup, you have two different AI’s: One vision network that classified images, and the chess AI that plays chess, and presumably connecting code that translates the output of the vision into a format suitable for the chess player.
I think what Sarah is referring to is that if you tried to directly hook up the images to the chess engine, it wouldn’t be able to figure it out, because reading images is not something it’s trained to do.
I honestly think of specialised models not as brains in their own right, but as cortexes. Pieces of a brain. But you can obviously hook them up together to do all sorts of things (for example, a multimodal LLM could take an image of a board and turn it into a series of coordinates and piece names). The one thing is that these models all would exist one level below the emergent simulacra that have actual agency. They’re the book or the operator or the desk in the Chinese Room. But it’s the Room as a whole that is intelligent and agentic.
Or in other words: our individual neurons don’t optimise for world-referenced goals either. Their goal is just “fire if stimulated so-and-so”.
Yes and networks of sensory neurons are apparently minimizing prediction error similar to LLM with next word prediction but with neurons also minimizing prediction across hierarchies. They are obviously not agents but combine into one.
Have you ever tried to explain the difference between correlation and causation to someone who didn’t understand it? I’m not convinced that this is even something humans innately have, rather than some higher-level correction by systems that do that.
You can hook a chess-playing network up to a vision network and have it play chess using images of boards—it’s not difficult. Perhaps a better example is that language models can be easily coupled to image models to get prompted image generation. You can also translate between language pairs that didn’t have direct translations in the training data.
This post seems rather specific to LLMs for how much it’s trying to generalize; I think there’s been more progress on that than Sarah seems to realize.
Understanding a high-level description of an abstract concept is different from having more low-level cognitive machinery that can apply the concept intuitively; you can have one without having the other (this goes in both directions). One classic example is that if you catch a flying ball with your hand, your brain needs to do something like solving a set of differential equations in order to predict the path of the ball… but this doesn’t imply that the people who were good at catching balls would be any good with solving explicit sets of differential equations. (Nor that people who were good at math would be any good at catching balls, for that matter.)
‘Have you ever tried to explain the difference between correlation and causation to someone who didn’t understand it? I’m not convinced that this is even something humans innately have, rather than some higher-level correction by systems that do that.’
You are outside and feel wind on your face. In front of you, you can see trees swaying in the wind. Did the swaying of the trees cause the wind? Or did the wind cause the trees to sway?
The cat bats at a moving toy. Usually he misses it. If he hits it, it usually makes a noise, but not always. The presence of the noise is more closely correlated with the cat successfully hitting the toy than the cat batting at the ball. But did the noise cause the cat to hit the ball or did the cat batting at the ball cause the hit?
The difference between correlation and causation is something we humans have a great sense for, so these questions seem really stupid. But they’re actually very challenging to answer using only observations (without being able to intervene).
I was talking about (time-shifted correlation) vs causation. That’s what people get confused about.
Mark Teixeira wears 2 different socks when playing baseball. That’s because he did that once and things went better. Why do you think he does that?
I think you have to be careful here. In this setup, you have two different AI’s: One vision network that classified images, and the chess AI that plays chess, and presumably connecting code that translates the output of the vision into a format suitable for the chess player.
I think what Sarah is referring to is that if you tried to directly hook up the images to the chess engine, it wouldn’t be able to figure it out, because reading images is not something it’s trained to do.
I honestly think of specialised models not as brains in their own right, but as cortexes. Pieces of a brain. But you can obviously hook them up together to do all sorts of things (for example, a multimodal LLM could take an image of a board and turn it into a series of coordinates and piece names). The one thing is that these models all would exist one level below the emergent simulacra that have actual agency. They’re the book or the operator or the desk in the Chinese Room. But it’s the Room as a whole that is intelligent and agentic.
Or in other words: our individual neurons don’t optimise for world-referenced goals either. Their goal is just “fire if stimulated so-and-so”.
Yes and networks of sensory neurons are apparently minimizing prediction error similar to LLM with next word prediction but with neurons also minimizing prediction across hierarchies. They are obviously not agents but combine into one.