Cool post! I have random thoughts you may or may not find interesting:
In the specific case of the bigfoot example, focusing on mind-world correlation measure seems worthwhile. You have the belief that your mind has become linked to the state of whether bigfoot is in the next room by the process of your mind remembering times you walked into rooms that didn’t have bigfoot, and your understanding of society and the prevailing scientific beliefs on bigfoot. With common assumptions it seems there is a strong link between the state of your mind and the actual absence of bigfoot in the next room. You can apply the same process of examining the mind-world correlation of your friend who thinks bigfoot in the next room is 50⁄50. Maybe you happen to know your friend has only very limited experience with probability, thinking of 50⁄50 as a shorthand for “it either happens or it doesn’t”. Or they have a history of magical thinking, preferring fantasies to facing a cold and uncaring reality. These may imply that your friends language isn’t mapping to reality in the same way as yours, or that your friends model has a weaker connection to reality than yours. Much easier way to dismiss the specific bigfoot credence than consulting the Solomonoff prior.
For the malign Solomonoff simulation capture situation, I have an intuition that you would want to strategies over possible worlds to take actions that work best when applied uniformly, including modelling the relationship between the possible versions of yourself taking actions and how they possibly relate to future simulated versions of yourself. Probably actions taken by base level, earlier timeline, versions are more significant, since they can have butterfly effects on future worlds. The doesn’t really protect against attacks from spaces that are not causally linked, but I think the no free lunch theorem applies there. So from a pragmatic standpoint it makes sense to act as if you are not a simulated version I would think.
In the discussion of betting money, experience, and terminal value, I think what is being reached for is an abstraction of “wantingness”, and terminal value makes the most sense, but runs into the issue that I don’t think that individual humans have consistent, coherent terminal values. But it isn’t a show stopper, because the goal is just that the agent assigning a probability is trying to make a bet such that they maximize their winnings, so we can just suppose that. Say “suppose you are given 100 probabilitrons and want to maximize your probabilitrons by betting them”. We can’t expect actual agents to actually care about the results of the thought experiment, but it does point at what we are trying to point at with assigning probabilities. However, it does seem worthwhile to explicitly note that people are not always incentivized to give accurate probability estimates, and that the purpose of probability estimates is for decision systems to make use of.
In saying “I want to choose between action A and B, and taking into account all considerations, I want to know which action leads to a better world according to my values.” you have provided “A and B” as possible actions, but it is important how A and B were located in the space of possible actions. This seems like a question corresponding to the problem of locating hypotheses worthy of consideration, and the problem of finding strategies to actually make these considerations that are not intractable/incomputable.
Cool post! I have random thoughts you may or may not find interesting:
In the specific case of the bigfoot example, focusing on mind-world correlation measure seems worthwhile. You have the belief that your mind has become linked to the state of whether bigfoot is in the next room by the process of your mind remembering times you walked into rooms that didn’t have bigfoot, and your understanding of society and the prevailing scientific beliefs on bigfoot. With common assumptions it seems there is a strong link between the state of your mind and the actual absence of bigfoot in the next room. You can apply the same process of examining the mind-world correlation of your friend who thinks bigfoot in the next room is 50⁄50. Maybe you happen to know your friend has only very limited experience with probability, thinking of 50⁄50 as a shorthand for “it either happens or it doesn’t”. Or they have a history of magical thinking, preferring fantasies to facing a cold and uncaring reality. These may imply that your friends language isn’t mapping to reality in the same way as yours, or that your friends model has a weaker connection to reality than yours. Much easier way to dismiss the specific bigfoot credence than consulting the Solomonoff prior.
For the malign Solomonoff simulation capture situation, I have an intuition that you would want to strategies over possible worlds to take actions that work best when applied uniformly, including modelling the relationship between the possible versions of yourself taking actions and how they possibly relate to future simulated versions of yourself. Probably actions taken by base level, earlier timeline, versions are more significant, since they can have butterfly effects on future worlds. The doesn’t really protect against attacks from spaces that are not causally linked, but I think the no free lunch theorem applies there. So from a pragmatic standpoint it makes sense to act as if you are not a simulated version I would think.
In the discussion of betting money, experience, and terminal value, I think what is being reached for is an abstraction of “wantingness”, and terminal value makes the most sense, but runs into the issue that I don’t think that individual humans have consistent, coherent terminal values. But it isn’t a show stopper, because the goal is just that the agent assigning a probability is trying to make a bet such that they maximize their winnings, so we can just suppose that. Say “suppose you are given 100 probabilitrons and want to maximize your probabilitrons by betting them”. We can’t expect actual agents to actually care about the results of the thought experiment, but it does point at what we are trying to point at with assigning probabilities. However, it does seem worthwhile to explicitly note that people are not always incentivized to give accurate probability estimates, and that the purpose of probability estimates is for decision systems to make use of.
In saying “I want to choose between action A and B, and taking into account all considerations, I want to know which action leads to a better world according to my values.” you have provided “A and B” as possible actions, but it is important how A and B were located in the space of possible actions. This seems like a question corresponding to the problem of locating hypotheses worthy of consideration, and the problem of finding strategies to actually make these considerations that are not intractable/incomputable.