However, having exactly two layers of structure seems rather artificial.
Sometimes, you only need a finite number of kinds of things. For example, in category theory you have objects and morphisms (dots and arrows which form a category) then functors (higher-level arrows which map between categories) then natural transformations (even higher-level arrows which map between functors) but then that’s it, you’re all good. In fact, you can consider functors as morphisms in the category of categories, and natural transformations as morphisms in categories of functors.
Unrelatedly, but similarly, traders in Garrabrant markets are perfectly capable of doing reasoning on logical statements about Garrabrant inductors.
2
Interesting point about goals as limited belief deltas. This may relate to coalitional agency, in a way. If each agent delegates a chunk of goal to a sub-agent, then the chunk of goal which can be delegated is limited in size by the capabilities of the sub-agent: too much goal and the sun-agent breaks down.
Traders in Garrabrant markets are perfectly capable of doing reasoning on logical statements about Garrabrant inductors
Yepp, agreed. Garrabrant induction feels big/rich enough that it wouldn’t surprise me if we “found” a bunch of complex structure in how traders interact mediated by their reasoning about propositions about each other.
However, it does seem like the details of how traders interact might be important. For example, a trader can have a policy of doing the opposite of what the market says it’ll do if the market becomes too confident, which limits the resolution at which market propositions can “see” the traders.
Whereas Scott Garrabrant has suggested to me the possibility that you could just let traders “see” each other’s trades directly before they happen, then formulate their own trades as a function of those other trades (though figuring out what trades actually happen would then require finding the fixed point of all those functions). This seems like it’d be meaningfully different from the existing setup.
If each agent delegates a chunk of goal to a sub-agent, then the chunk of goal which can be delegated is limited in size by the capabilities of the sub-agent: too much goal and the sun-agent breaks down.
Ah, interesting point. This also basically seems right. I’m reminded of a comment I saw on twitter that one way you can harm someone is by giving them goals that far exceed the scope of their agency/control.
Garrabrant induction feels big/rich enough that it wouldn’t surprise me if we “found” a bunch of complex structure in how traders interact mediated by their reasoning about propositions about each other.
There’s some of this in the original paper. I think two traders can “interact” if a third trader models both of them and makes trades accordingly (e.g. market making).
My general feeling is that Garrabrant Induction, despite being narrower/more specific than Solomonoff induction, is still not specific enough in spotlighting the parts of intelligence/agency that we actually care about and interact with in practice.
Some thoughts
1
Sometimes, you only need a finite number of kinds of things. For example, in category theory you have objects and morphisms (dots and arrows which form a category) then functors (higher-level arrows which map between categories) then natural transformations (even higher-level arrows which map between functors) but then that’s it, you’re all good. In fact, you can consider functors as morphisms in the category of categories, and natural transformations as morphisms in categories of functors.
Unrelatedly, but similarly, traders in Garrabrant markets are perfectly capable of doing reasoning on logical statements about Garrabrant inductors.
2
Interesting point about goals as limited belief deltas. This may relate to coalitional agency, in a way. If each agent delegates a chunk of goal to a sub-agent, then the chunk of goal which can be delegated is limited in size by the capabilities of the sub-agent: too much goal and the sun-agent breaks down.
Great comment, ty.
Yepp, agreed. Garrabrant induction feels big/rich enough that it wouldn’t surprise me if we “found” a bunch of complex structure in how traders interact mediated by their reasoning about propositions about each other.
However, it does seem like the details of how traders interact might be important. For example, a trader can have a policy of doing the opposite of what the market says it’ll do if the market becomes too confident, which limits the resolution at which market propositions can “see” the traders.
Whereas Scott Garrabrant has suggested to me the possibility that you could just let traders “see” each other’s trades directly before they happen, then formulate their own trades as a function of those other trades (though figuring out what trades actually happen would then require finding the fixed point of all those functions). This seems like it’d be meaningfully different from the existing setup.
Ah, interesting point. This also basically seems right. I’m reminded of a comment I saw on twitter that one way you can harm someone is by giving them goals that far exceed the scope of their agency/control.
There’s some of this in the original paper. I think two traders can “interact” if a third trader models both of them and makes trades accordingly (e.g. market making).
My general feeling is that Garrabrant Induction, despite being narrower/more specific than Solomonoff induction, is still not specific enough in spotlighting the parts of intelligence/agency that we actually care about and interact with in practice.