There seems to be some missing linkage between what a computer knows and what it can do. I feel like there is some notion of action that is missing: how the heck does an AI of any given sophistication add new actions to its repertoire? This can’t happen in any software we currently have—even the ability to categorize or define actions doesn’t imply the ability to create new ones.
Logical Actions are Optimization Channels: intuition from Information Theory—a given action is like a channel, and the message is optimization of the environment vis-a-vis goals. Logical actions are not the same as actual actions: for example, it is obvious to humans that we can look at what a computer actually does to get information about its intentions, so having drones steal candy from babies can turn us against it regardless of what it displays on the monitor. But a logical action of ‘Signal Good Intentions’ encompasses both what the monitor displays and how humans perceive the drone activity. Further, we can look at how dividing bandwidth up into multiple channels impacts the efficiency of transmitting a message as an intuition for how more logical actions increase the capability of the AI.
This seems to be orthogonal to the question of agency—even an AI with many logical actions that it optimizes won’t generate new ones unless one of those logical actions is ‘search action space for new actions’. This makes it clear that Tool AIs with a large action set will be strictly more powerful than Agent AIs that only start with ‘search action space’ up until a certain point.
3. Naively, actions feel like they require causal reasoning, and causal reasoning of any kind seems to require the ability to reason about two parts of the environment. One of these parts can be you (or the AI).
But I am not sure this is the case. Strong correlation seems to be good enough for a human brain—we do all kinds of actions without any understanding of what we are doing or why. This can go as far as provoking conscious confusion during the action. Based on this lower standard, what correlation would be needed?
Boundaries of some kind, because we need some way to localize what we are doing and looking at. Strong, chiefly as a matter of efficiency. We want a way to describe a correlation such that we can chain it with other correlations, and then eventually bundle them together as an action. Then doing new actions is a matter of chaining the correlations back to something we can currently do.
I feel like a Rube-Goldberg device would be a good intuition pump here. How can we describe a Rube-Goldberg device in terms of correlations? What is a good way to break it into chunks, and then also a good way to connect those chunks? Since they are usually built of simple machines, everything is mathematically tractable—we have a good grip on those.
2. Is thinking about actions just rephrasing the agent-environment question? It feels like the answer is no, because it isn’t as though being able to specify the relationship between the agent and the environment changes the need to compute the specific details of any particular action.
But it might be impossible to specify an action exactly without being able to specify the agent-environment relationship exactly. Could it be (or is it) stated implicitly?
Actions are not just Embedded Agency in a different guise. From the Full-Text Version it looks to me like what actions are and how to discover them is abstracted away, which makes sense in the context of that project.
It appears most relevant to problems associated with multi-level models.
The older conversations about Tool AI seemed to focus on the difference between an Oracle that answers questions and one that does things. I feel like this distinction is bigger and in a different way than it was made out to be, because doing things is really hard. If feels like the paradox of sensing being complicated should go two ways.
Checking the Wikipedia page for Moravec’s Paradox, “sensorimotor” is how they describe it, so both sensing and motor skills (inputs and outputs) are covered. My intuition fairly screams that this should generalize to any other environment-affecting action. So:
The more inputs an AI starts with, the easier it is to recognize other inputs/outputs.
The more outputs an AI starts with, the easier it is to add other inputs/outputs.
This still doesn’t identify what causes the machine to try to affect the world at all.
For the Book Review: Reframing Superintelligence (SSC) linkpost:
There seems to be some missing linkage between what a computer knows and what it can do. I feel like there is some notion of action that is missing: how the heck does an AI of any given sophistication add new actions to its repertoire? This can’t happen in any software we currently have—even the ability to categorize or define actions doesn’t imply the ability to create new ones.
Logical Actions are Optimization Channels: intuition from Information Theory—a given action is like a channel, and the message is optimization of the environment vis-a-vis goals. Logical actions are not the same as actual actions: for example, it is obvious to humans that we can look at what a computer actually does to get information about its intentions, so having drones steal candy from babies can turn us against it regardless of what it displays on the monitor. But a logical action of ‘Signal Good Intentions’ encompasses both what the monitor displays and how humans perceive the drone activity. Further, we can look at how dividing bandwidth up into multiple channels impacts the efficiency of transmitting a message as an intuition for how more logical actions increase the capability of the AI.
This seems to be orthogonal to the question of agency—even an AI with many logical actions that it optimizes won’t generate new ones unless one of those logical actions is ‘search action space for new actions’. This makes it clear that Tool AIs with a large action set will be strictly more powerful than Agent AIs that only start with ‘search action space’ up until a certain point.
3. Naively, actions feel like they require causal reasoning, and causal reasoning of any kind seems to require the ability to reason about two parts of the environment. One of these parts can be you (or the AI).
But I am not sure this is the case. Strong correlation seems to be good enough for a human brain—we do all kinds of actions without any understanding of what we are doing or why. This can go as far as provoking conscious confusion during the action. Based on this lower standard, what correlation would be needed?
Boundaries of some kind, because we need some way to localize what we are doing and looking at. Strong, chiefly as a matter of efficiency. We want a way to describe a correlation such that we can chain it with other correlations, and then eventually bundle them together as an action. Then doing new actions is a matter of chaining the correlations back to something we can currently do.
I feel like a Rube-Goldberg device would be a good intuition pump here. How can we describe a Rube-Goldberg device in terms of correlations? What is a good way to break it into chunks, and then also a good way to connect those chunks? Since they are usually built of simple machines, everything is mathematically tractable—we have a good grip on those.
2. Is thinking about actions just rephrasing the agent-environment question? It feels like the answer is no, because it isn’t as though being able to specify the relationship between the agent and the environment changes the need to compute the specific details of any particular action.
But it might be impossible to specify an action exactly without being able to specify the agent-environment relationship exactly. Could it be (or is it) stated implicitly?
Actions are not just Embedded Agency in a different guise. From the Full-Text Version it looks to me like what actions are and how to discover them is abstracted away, which makes sense in the context of that project.
It appears most relevant to problems associated with multi-level models.
1. I am deeply confused by this.
The older conversations about Tool AI seemed to focus on the difference between an Oracle that answers questions and one that does things. I feel like this distinction is bigger and in a different way than it was made out to be, because doing things is really hard. If feels like the paradox of sensing being complicated should go two ways.
Checking the Wikipedia page for Moravec’s Paradox, “sensorimotor” is how they describe it, so both sensing and motor skills (inputs and outputs) are covered. My intuition fairly screams that this should generalize to any other environment-affecting action. So:
The more inputs an AI starts with, the easier it is to recognize other inputs/outputs.
The more outputs an AI starts with, the easier it is to add other inputs/outputs.
This still doesn’t identify what causes the machine to try to affect the world at all.