I don’t see how Take 4 is anything other than simplicity (in the human/computational language). As you say, it’s a priori unclear whether a an agent is an instance of a human or the other way around. You say the important bit is that you are subtracting properties from a human to get an agent. But how shall we define subtraction here? In one formal language, the definition of human will indeed be a superset of that of agent… but in another one it will not. So you need to choose a language. And the natural way forward every time this comes up (many times), is to just “weigh by Turing computations in the real world” (instead of choosing a different and odd-looking Universal Turing Machine), that is, a simplicity prior.
I expect that it’s strongly correlated with simplicity in practice, yes. However, the two definitions could diverge.
Consider trees, again. You can imagine a sequence of concepts/words such that (1) each subsequent concept is simple to define in terms of the preceding concepts, (2) one of the concepts in the sequence is “trees”. (Indeed, that’s basically how human minds work.)
Now consider some highly complicated mathematical concept, which is likewise simple to define in terms of the preceding concepts. It seems plausible that there are “purely mathematical” concepts like this such that their overall complexity (the complexity of the chain of concepts leading to them) is on par with the complexity of the “trees” concept.
So an agent that’s allowed to reason about concepts which are simple to define in its language, and which can build arbitrarily tall “towers” of abstractions, can still stumble upon ways to reason about real-life concepts. By comparison, if Take 4 is correct, and we have a reference global ontology on hand[1], we could correctly forbid it from thinking about concepts instantiated within this universe without crippling its ability to think about complex theoretical concepts.
(The way this is correlated with simplicity is if there’s some way to argue that the only concepts that “spontaneously reoccur” at different levels of organization of our universe, are those concepts that are very simple. Perhaps because moving up/down abstraction levels straight-up randomizes the concepts you’re working with. That would mean the probability of encountering two copies of a concept is inversely proportional with its bitwise “length”.)
I was about to comment with something similar to what Martín said. I think what you want is an AI that solves problems that can be fully specified with a low Kolmogorov complexity in some programming language. A crisp example of this sort of AI is AlphaProof, which gets a statement in Lean as input and has to output a series of steps that causes the prover to output “true” or “false.” You could also try to specify problems in a more general programming language like Python.
A “highly complicated mathematical concept” is certainly possible. It’s easy to construct hypothetical objects in math (or Python) that have arbitrarily large Kolmogorov complexity: for example, a list of random numbers with length 3^^^3.
Another example of a highly complicated mathematical object which might be “on par with the complexity of the ‘trees’ concept” is a neural network. In fact, if you have a neural network that looks at an image and tells you whether or not it’s a tree with very high accuracy, you might say that this neural network formally encodes (part of) the concept of a tree. It’s probably hard to do much better than a neural network if you hope to encode the “concept of a tree,” and this requires a Python program a lot longer than the program that would specify a vector or a market.
Eliezer’s post on Occam’s Razor explains this better than I could—it’s definitely worth a read.
I don’t see how Take 4 is anything other than simplicity (in the human/computational language). As you say, it’s a priori unclear whether a an agent is an instance of a human or the other way around. You say the important bit is that you are subtracting properties from a human to get an agent. But how shall we define subtraction here? In one formal language, the definition of human will indeed be a superset of that of agent… but in another one it will not. So you need to choose a language. And the natural way forward every time this comes up (many times), is to just “weigh by Turing computations in the real world” (instead of choosing a different and odd-looking Universal Turing Machine), that is, a simplicity prior.
I expect that it’s strongly correlated with simplicity in practice, yes. However, the two definitions could diverge.
Consider trees, again. You can imagine a sequence of concepts/words such that (1) each subsequent concept is simple to define in terms of the preceding concepts, (2) one of the concepts in the sequence is “trees”. (Indeed, that’s basically how human minds work.)
Now consider some highly complicated mathematical concept, which is likewise simple to define in terms of the preceding concepts. It seems plausible that there are “purely mathematical” concepts like this such that their overall complexity (the complexity of the chain of concepts leading to them) is on par with the complexity of the “trees” concept.
So an agent that’s allowed to reason about concepts which are simple to define in its language, and which can build arbitrarily tall “towers” of abstractions, can still stumble upon ways to reason about real-life concepts. By comparison, if Take 4 is correct, and we have a reference global ontology on hand[1], we could correctly forbid it from thinking about concepts instantiated within this universe without crippling its ability to think about complex theoretical concepts.
(The way this is correlated with simplicity is if there’s some way to argue that the only concepts that “spontaneously reoccur” at different levels of organization of our universe, are those concepts that are very simple. Perhaps because moving up/down abstraction levels straight-up randomizes the concepts you’re working with. That would mean the probability of encountering two copies of a concept is inversely proportional with its bitwise “length”.)
Step 2: Draw the rest of the owl.
I was about to comment with something similar to what Martín said. I think what you want is an AI that solves problems that can be fully specified with a low Kolmogorov complexity in some programming language. A crisp example of this sort of AI is AlphaProof, which gets a statement in Lean as input and has to output a series of steps that causes the prover to output “true” or “false.” You could also try to specify problems in a more general programming language like Python.
A “highly complicated mathematical concept” is certainly possible. It’s easy to construct hypothetical objects in math (or Python) that have arbitrarily large Kolmogorov complexity: for example, a list of random numbers with length 3^^^3.
Another example of a highly complicated mathematical object which might be “on par with the complexity of the ‘trees’ concept” is a neural network. In fact, if you have a neural network that looks at an image and tells you whether or not it’s a tree with very high accuracy, you might say that this neural network formally encodes (part of) the concept of a tree. It’s probably hard to do much better than a neural network if you hope to encode the “concept of a tree,” and this requires a Python program a lot longer than the program that would specify a vector or a market.
Eliezer’s post on Occam’s Razor explains this better than I could—it’s definitely worth a read.