So regarding things that involve active prioritizing of compute resources, I think that would fairly clearly fall no longer under epistemic rationality. Because “spending compute resources on this rather than that” is an action, which are only part of instrumental rationality. So in that sense it wouldn’t be part of intelligence. Which makes some sense given that intuitively smart people often concentrate their mental efforts on things that are not necessarily very useful to them.
This relates also to what you write about level 1 and 2 compared to level 3. In the first two cases you mention actions, but not in the third. Which makes sense if level 3 is about epistemic rationality. Assuming level 1 and 2 are about instrumental rationality then, this would be an interesting difference to my previous conceptualization: On my picture, epistemic rationality was a necessary but not sufficient condition for instrumental rationality, but on your picture, instead level 1 and 2 (~instrumental rationality) are a necessary but not sufficient condition for level 3 (~epistemic rationality). I’m not sure what we can conclude from these inversed pictures.
I think of generality of intelligence as relatively conceptually trivial. At the end of the day, a system is given a sequence of data via observation, and is now tasked with finding a function or set of functions that both corresponds to plausible transition rules of the given sequence, and has a reasonably high chance of correctly predicting the next element of the sequence
Okay, but terminology-wise I wouldn’t describe this as generality. Because the narrow/general axis seems to have more to to with instrumental rationality / competence than with epistemic rationality / intelligence. The latter can be described as a form of prediction, or building causal models / a world model. But generality seems to be more about what a system can do overall in terms of actions. GPT-4 may have a quite advanced world model, but at its heart it only imitates Internet text, and doesn’t do so in real time, so it can hardly be used for robotics. So I would describe it as a less general system than most animals, though more general than a Go AI.
Regarding an overall model of cognition, a core part that describes epistemic rationality seems to be captured well by a theory called predictive coding or predictive processing. Scott Alexander has an interesting article about it. It’s originally a theory from neuroscience, but Yann LeCun also sees it as a core part of his model of cognition. The model is described here on pages 6 to 9. Predictive coding is responsible for the part of cognition that he calls the world model.
Basically, predictive coding is the theory that an agent constantly does self-supervised learning (SSL) on sensory data (real-time / online) by continuously predicting its experiences and continuously updating the world-model depending on whether those predictions were correct. This creates a world model, which is the basis for the other abilities of the agent, like creating and executing action plans. LeCun calls the background knowledge created by this type of predictive coding the “dark matter” of intelligence, because it includes fundamental common sense knowledge, like intuitive physics.
The current problem is that currently self-supervised learning only really works for text (in LLMs), but not yet properly for things like video. Basically the difference is that with text we have a relatively small number of discrete tokens with quite low redundancy, while for sensory inputs we have basically continuous data with a very large amount of redundancy. It makes no computational sense to predict probabilities of individual frames of video data like it makes sense for an LLM to “predict” probabilities for the next text token. Currently LeCun tries to make SSL work for these types of sensory data by using his “Joint Embedding Predictive Architecture” (JEPA), described in the paper above.
To the extent that creating a world model is handled by predictive coding, and if we call the ability to create accurate world models “epistemic rationality” or “intelligence”, we seem to have a pretty good grasp of what we are talking about. (Even though we don’t yet have a working implementation of predictive coding, like JEPA.)
But if we talk about a general theory of cognition/competence/instrumental rationality, the picture is much less clear. All we have is things like LeCun’s very coarse model of cognition (pages 6ff in the paper above), or completely abstract models like AIXI. So there is a big gap in understanding what the cognition of a competent agent even looks like.
So regarding things that involve active prioritizing of compute resources, I think that would fairly clearly fall no longer under epistemic rationality. Because “spending compute resources on this rather than that” is an action, which are only part of instrumental rationality. So in that sense it wouldn’t be part of intelligence. Which makes some sense given that intuitively smart people often concentrate their mental efforts on things that are not necessarily very useful to them.
This relates also to what you write about level 1 and 2 compared to level 3. In the first two cases you mention actions, but not in the third. Which makes sense if level 3 is about epistemic rationality. Assuming level 1 and 2 are about instrumental rationality then, this would be an interesting difference to my previous conceptualization: On my picture, epistemic rationality was a necessary but not sufficient condition for instrumental rationality, but on your picture, instead level 1 and 2 (~instrumental rationality) are a necessary but not sufficient condition for level 3 (~epistemic rationality). I’m not sure what we can conclude from these inversed pictures.
Okay, but terminology-wise I wouldn’t describe this as generality. Because the narrow/general axis seems to have more to to with instrumental rationality / competence than with epistemic rationality / intelligence. The latter can be described as a form of prediction, or building causal models / a world model. But generality seems to be more about what a system can do overall in terms of actions. GPT-4 may have a quite advanced world model, but at its heart it only imitates Internet text, and doesn’t do so in real time, so it can hardly be used for robotics. So I would describe it as a less general system than most animals, though more general than a Go AI.
Regarding an overall model of cognition, a core part that describes epistemic rationality seems to be captured well by a theory called predictive coding or predictive processing. Scott Alexander has an interesting article about it. It’s originally a theory from neuroscience, but Yann LeCun also sees it as a core part of his model of cognition. The model is described here on pages 6 to 9. Predictive coding is responsible for the part of cognition that he calls the world model.
Basically, predictive coding is the theory that an agent constantly does self-supervised learning (SSL) on sensory data (real-time / online) by continuously predicting its experiences and continuously updating the world-model depending on whether those predictions were correct. This creates a world model, which is the basis for the other abilities of the agent, like creating and executing action plans. LeCun calls the background knowledge created by this type of predictive coding the “dark matter” of intelligence, because it includes fundamental common sense knowledge, like intuitive physics.
The current problem is that currently self-supervised learning only really works for text (in LLMs), but not yet properly for things like video. Basically the difference is that with text we have a relatively small number of discrete tokens with quite low redundancy, while for sensory inputs we have basically continuous data with a very large amount of redundancy. It makes no computational sense to predict probabilities of individual frames of video data like it makes sense for an LLM to “predict” probabilities for the next text token. Currently LeCun tries to make SSL work for these types of sensory data by using his “Joint Embedding Predictive Architecture” (JEPA), described in the paper above.
To the extent that creating a world model is handled by predictive coding, and if we call the ability to create accurate world models “epistemic rationality” or “intelligence”, we seem to have a pretty good grasp of what we are talking about. (Even though we don’t yet have a working implementation of predictive coding, like JEPA.)
But if we talk about a general theory of cognition/competence/instrumental rationality, the picture is much less clear. All we have is things like LeCun’s very coarse model of cognition (pages 6ff in the paper above), or completely abstract models like AIXI. So there is a big gap in understanding what the cognition of a competent agent even looks like.