This gives a nice intuitive explanation for the Jeffery-Bolker rotation which basically is a way of interpreting a belief as a utility, and vice versa.
Some thoughts:
What do probabilities mean without reference to any sort of agent? Presumably it has something to do with the ability to “win” De Finetti games in expectation. For avoiding subtle anthropomorphization, it might be good to think of this sort of probability as being instantiated in a bacterium’s chemical sensor, or something like that. And in this setting, it’s clear it wouldn’t mean anything without the context of the bacterium. Going further, it seems to me like the only mechanism which makes this mean anything is the fact that it helps make the bacterium “exist more” i.e. reproduce and thrive. So I think having a probability mean a probability inherently requires some sort of self-propagation—it means something if it’s part of why it exists. This idea can be taken to an even deeper level, where according to Zureck you can get the Born probabilities by looking at what quantum states allow information to persist through time (from within the system).
Does this imply anything about the difficulty of value learning? An AGI will be able to make accurate models of the world, so it will have the raw algorithms needed to do value learning… the hard part seems to be, as usual, pointing to the “correct” values. Not sure this helps with that so much.
A bounded agent creating a model will have to make decisions about how much detail to model various aspects of the world in. Can we use this idea to “factor” out that sort of trade-off as part of the utility function?
I don’t see the connection to the Jeffrey-Bolker rotation? There, to get the shouldness coordinate, you need to start with the epistemic probability measure, and multiply it by utility; here, utility is interpreted as a probability distribution without reference to a probability distribution used for beliefs.
This gives a nice intuitive explanation for the Jeffery-Bolker rotation which basically is a way of interpreting a belief as a utility, and vice versa.
Some thoughts:
What do probabilities mean without reference to any sort of agent? Presumably it has something to do with the ability to “win” De Finetti games in expectation. For avoiding subtle anthropomorphization, it might be good to think of this sort of probability as being instantiated in a bacterium’s chemical sensor, or something like that. And in this setting, it’s clear it wouldn’t mean anything without the context of the bacterium. Going further, it seems to me like the only mechanism which makes this mean anything is the fact that it helps make the bacterium “exist more” i.e. reproduce and thrive. So I think having a probability mean a probability inherently requires some sort of self-propagation—it means something if it’s part of why it exists. This idea can be taken to an even deeper level, where according to Zureck you can get the Born probabilities by looking at what quantum states allow information to persist through time (from within the system).
Does this imply anything about the difficulty of value learning? An AGI will be able to make accurate models of the world, so it will have the raw algorithms needed to do value learning… the hard part seems to be, as usual, pointing to the “correct” values. Not sure this helps with that so much.
A bounded agent creating a model will have to make decisions about how much detail to model various aspects of the world in. Can we use this idea to “factor” out that sort of trade-off as part of the utility function?
I don’t see the connection to the Jeffrey-Bolker rotation? There, to get the shouldness coordinate, you need to start with the epistemic probability measure, and multiply it by utility; here, utility is interpreted as a probability distribution without reference to a probability distribution used for beliefs.