Thank you for writing this, I feel like it makes the core idea you’re expressing at much clearer.
My intuition is that abstract Wiener spaces won’t get you the sort of measure you’re looking for alone, based off my experience with measures over big spaces in physics. But, that said, I feel like there should be some such measure over large physical spaces, as presumably power has a definition in terms of physical concepts, or else how the heck can we recover our intuition of power in our world? It should all add up to normality, after all. It seems to me that looking over those physics papers which descibed single particles as agentic because our distributions over them tend towards max entropy, which we can view as the particle seeking the greatest “option value” it can, would be a good place to build up the latter intuition.
I think I am undecided as to whether you can use the rich structre of reward functions to limit the allowed transormations in a useful way. Partly because I suspect that this rich structure reflects a physical structure (something like the natural abstractions thesis + selection pressure from reality for the sorts of rewards we typically see) or perhaps a simplicity prior of some sort. But maybe it will work out. I don’t know.
My lack of optimism as to the possibility of your agenda is basically why I was willing to accept the strange probability distribution TurnTrout went with, I guess. But on reflection, perhaps I should have used that as an existence proof of distribution over reward which allows something like our intuitive picture of power seeking. And tried to see if I could interpret it to be something less weird, use it to find something less weird, or just go look for less weird things because maybe they’d work.
Sorry for the long post, but I just realized I didn’t update based off Turntrout’s results. It seems more likely to me now that your agenda might work. Though I’d be more optimistic if you were using Turntrout’s distribution as inspiration for what to look for in some way.
Thank you for writing this, I feel like it makes the core idea you’re expressing at much clearer.
My intuition is that abstract Wiener spaces won’t get you the sort of measure you’re looking for alone, based off my experience with measures over big spaces in physics. But, that said, I feel like there should be some such measure over large physical spaces, as presumably power has a definition in terms of physical concepts, or else how the heck can we recover our intuition of power in our world? It should all add up to normality, after all. It seems to me that looking over those physics papers which descibed single particles as agentic because our distributions over them tend towards max entropy, which we can view as the particle seeking the greatest “option value” it can, would be a good place to build up the latter intuition.
I think I am undecided as to whether you can use the rich structre of reward functions to limit the allowed transormations in a useful way. Partly because I suspect that this rich structure reflects a physical structure (something like the natural abstractions thesis + selection pressure from reality for the sorts of rewards we typically see) or perhaps a simplicity prior of some sort. But maybe it will work out. I don’t know.
My lack of optimism as to the possibility of your agenda is basically why I was willing to accept the strange probability distribution TurnTrout went with, I guess. But on reflection, perhaps I should have used that as an existence proof of distribution over reward which allows something like our intuitive picture of power seeking. And tried to see if I could interpret it to be something less weird, use it to find something less weird, or just go look for less weird things because maybe they’d work.
Sorry for the long post, but I just realized I didn’t update based off Turntrout’s results. It seems more likely to me now that your agenda might work. Though I’d be more optimistic if you were using Turntrout’s distribution as inspiration for what to look for in some way.