In the old counterfactual mugging problem, agents who precommit are trading utilities across possible worlds, each world having a utility-gain called a prior that expresses how much the agent wants its utilities to lie in those worlds instead of silly ones. From that perspective, it’s true that nothing in reality will be different as a result of the agent’s decision, just because of determinism, but the agent is still deciding what reality (across all possible worlds) will look like, just like in Newcomb’s problem.
So when I read in Nesov’s post that “Direct prediction of your actions can’t include the part where you observe that the digit is even, because the digit is odd”, what I’m really seeing is someone saying, “I give zero weight to possible worlds in which math doesn’t work sensibly, and tiny weights to worlds in which math does work, but my confusion or the conspiring of a malicious / improbable / senseless / invaluable universe cause me to think it does not.”
One of the reasons why I think possible worlds of the first kind (different causal / programmatic histories but the same underlying ontology-stuff) are valuable / real, is that we sort of know how to calculate their properties using causal networks or timeless networks or whatever kind of networks you get when you combine the not-quite specified mathematical machinery in TDT with UDT. Our ability to calculate their properties reifies them, opens them up to interacting with this world even more directly via simulation.
The next step seems to be to ask, “for agents that do care about those impossible possible worlds, how would they act?” If omega is choosing in a way that can be computed in our world, using our math (and somehow that other universe and our calculations don’t explode when it gets to the contradiction (or it does! I suppose you can care about worlds where math explodes, even if I can’t visualize them)), then we can simulate his reasoning in all respects save the identify of the logical fact in question, and use that to calculate which behaviour maximizes the utility across possible worlds via their dependence on our decision.
So in the example problem, if a valuer of contradictory worlds has roughly equal priors for both the world we’re examining and the other world in which she find herself where the digit was even (the impossible one, which isn’t impossible for her, because it wasn’t assigned zero prior weight), then sure, she can go ahead and give up control. That’s of course assuming that she has an expectation the staple maximizer will reciprocate in the impossible world, which you didn’t spell out in your post, but that dependence on decisions is standard for counterfactual mugging problems. Please correct me if that’s not the intended setup.
As as aside, this comment feels silly and wrong; an example of diseased thoughts unconnected with reality. It reminds me a bit of Greg Egan’s short story Dark Integers. I would really love to see a more sensible interpretation that this.
While I haven’t given it much though outside the context of fiction, one could adopt the point of view/vocabulary of this being “the level 5 tegmark mutiverse”.
Now, if that is true in any sense, it’s probably a much less literal one, and not based on the same reasoning as the other four, but it might still be an useful heuristic for humans.
Another interesting note: By default my brain seems to assume utility is linear with paperclips when considering say different Everett branches, but the logarithm of it when considering logical uncertainty. That’s kinda odd and unjustified, but the intuition might have some point about humans utility function.
Really the only things that seems in any way novel here are the idea that the space of possible worlds might include worlds that work by different mathematical rules and that possibility is contingent on the agent’s priors. I don’t know how to characterize how math works in a different world, other than by saying explicitly what the outcome of a given computation will be. You can think of that as forcing the structural equation that would normally compute “1+1” to output “5″, where the graph setup would somehow keep that logical fact from colliding with proofs that “3-1=2” (for worlds that don’t explode) (which is what I thought Eliezer meant by creating a factored DAG of mathematics here). That’s for a very limited case of illogical-calculation where our reasoning process produced results close enough to their analogues in the target world that we’re even able to make some valid deductions. Maybe other worlds don’t have a big book of platonic truths (ambiguity or instability) and cross-world utility calculations just don’t work. In that case, I can’t think of any sensible course of action.
I don’t think this is totally worthless speculation, even if you don’t agree that “a world with different math” makes sense, because an AI with faulty hardware / reasoning will still need to reason about mathematics that work differently from its mistaken inferences, and that probably requires a partial correspondence between how the agent reasons and how the world works, just like how the partial correspondence between worlds with different mathematical rules allows some limited deductions with cross-world or other-world validity.
In the old counterfactual mugging problem, agents who precommit are trading utilities across possible worlds, each world having a utility-gain called a prior that expresses how much the agent wants its utilities to lie in those worlds instead of silly ones. From that perspective, it’s true that nothing in reality will be different as a result of the agent’s decision, just because of determinism, but the agent is still deciding what reality (across all possible worlds) will look like, just like in Newcomb’s problem.
So when I read in Nesov’s post that “Direct prediction of your actions can’t include the part where you observe that the digit is even, because the digit is odd”, what I’m really seeing is someone saying, “I give zero weight to possible worlds in which math doesn’t work sensibly, and tiny weights to worlds in which math does work, but my confusion or the conspiring of a malicious / improbable / senseless / invaluable universe cause me to think it does not.”
One of the reasons why I think possible worlds of the first kind (different causal / programmatic histories but the same underlying ontology-stuff) are valuable / real, is that we sort of know how to calculate their properties using causal networks or timeless networks or whatever kind of networks you get when you combine the not-quite specified mathematical machinery in TDT with UDT. Our ability to calculate their properties reifies them, opens them up to interacting with this world even more directly via simulation.
The next step seems to be to ask, “for agents that do care about those impossible possible worlds, how would they act?” If omega is choosing in a way that can be computed in our world, using our math (and somehow that other universe and our calculations don’t explode when it gets to the contradiction (or it does! I suppose you can care about worlds where math explodes, even if I can’t visualize them)), then we can simulate his reasoning in all respects save the identify of the logical fact in question, and use that to calculate which behaviour maximizes the utility across possible worlds via their dependence on our decision.
So in the example problem, if a valuer of contradictory worlds has roughly equal priors for both the world we’re examining and the other world in which she find herself where the digit was even (the impossible one, which isn’t impossible for her, because it wasn’t assigned zero prior weight), then sure, she can go ahead and give up control. That’s of course assuming that she has an expectation the staple maximizer will reciprocate in the impossible world, which you didn’t spell out in your post, but that dependence on decisions is standard for counterfactual mugging problems. Please correct me if that’s not the intended setup.
As as aside, this comment feels silly and wrong; an example of diseased thoughts unconnected with reality. It reminds me a bit of Greg Egan’s short story Dark Integers. I would really love to see a more sensible interpretation that this.
While I haven’t given it much though outside the context of fiction, one could adopt the point of view/vocabulary of this being “the level 5 tegmark mutiverse”.
Now, if that is true in any sense, it’s probably a much less literal one, and not based on the same reasoning as the other four, but it might still be an useful heuristic for humans.
Another interesting note: By default my brain seems to assume utility is linear with paperclips when considering say different Everett branches, but the logarithm of it when considering logical uncertainty. That’s kinda odd and unjustified, but the intuition might have some point about humans utility function.
Sorry, I couldn’t follow.
That’s okay, there’s no formalized theory behind it. But for the sake of conversation:
It seems you once agreed that multiple agents in the same epistemic state in different possible worlds can define strategies over their future observations in a way that looks like trading utilities: http://lesswrong.com/lw/102/indexical_uncertainty_and_the_axiom_of/sht
When I treat priors as a kind of utility, that’s interpretation #4 from this Wei Dai post: http://lesswrong.com/lw/1iy/what_are_probabilities_anyway/
Really the only things that seems in any way novel here are the idea that the space of possible worlds might include worlds that work by different mathematical rules and that possibility is contingent on the agent’s priors. I don’t know how to characterize how math works in a different world, other than by saying explicitly what the outcome of a given computation will be. You can think of that as forcing the structural equation that would normally compute “1+1” to output “5″, where the graph setup would somehow keep that logical fact from colliding with proofs that “3-1=2” (for worlds that don’t explode) (which is what I thought Eliezer meant by creating a factored DAG of mathematics here). That’s for a very limited case of illogical-calculation where our reasoning process produced results close enough to their analogues in the target world that we’re even able to make some valid deductions. Maybe other worlds don’t have a big book of platonic truths (ambiguity or instability) and cross-world utility calculations just don’t work. In that case, I can’t think of any sensible course of action.
I don’t think this is totally worthless speculation, even if you don’t agree that “a world with different math” makes sense, because an AI with faulty hardware / reasoning will still need to reason about mathematics that work differently from its mistaken inferences, and that probably requires a partial correspondence between how the agent reasons and how the world works, just like how the partial correspondence between worlds with different mathematical rules allows some limited deductions with cross-world or other-world validity.