Well, as long as you believe Omega enough to think no box contains sudden death or otherwise negative utility, you’d open them to see what was inside. But yes, you might not believe Omega at all.
General question: suppose we encounter an alien. We have no idea what its motivations, values, goals, or abilities are. On the other hand, if may have observed any amount of human comm traffic from wireless EM signals since the invention of radio, and from actual spy-probes before the human invention of high tech that would detect them.
It signals us in Morse code from its remote starship, offering mutually benefitial trade.
What prior should we have about the alien’s intention? Should we use a native uniform prior that would tell us it’s as likely to mean us good as harm, and so never reply because we don’t know how it will try to influence our actions via communications? Should it tell us different agents who don’t explicitly value one another will conflict to the extent their values differ, and so since value-space is vast and a randomly selected alien is unlikely to share many values with us, we should prepare for war? Should it tell us we can make some assumptions (which?) about naturally evolved agents or their Friendly-to-themselves creations? How safe are we if we try to “just read” English text written by an unknown, possibly-superintelligence which may have observed all our broadcast traffic since the age of radio? What does our non-detection of this alien civ until they chose to initiate contact tell us? Etc.
A 50% chance of meaning us good vs harm isn’t a prior I find terribly compelling.
There’s a lot to say here, but my short answer is that this is both an incredibly dangerous and incredibly valuable situation, in which both the potential opportunity costs and the potential actual costs are literally astronomical, and in which there are very few things I can legitimately be confident of.
The best I can do in such a situation is to accept that my best guess is overwhelmingly likely to be wrong, but that it’s slightly less likely to be wrong than my second-best guess, so I should operate on the basis of my best guess despite expecting it to be wrong. Where “best guess” here is the thing I consider most likely to be true, not the thing with the highest expected value.
I should also note that my priors about aliens in general—that is, what I consider likely about a randomly selected alien intelligence—are less relevant to this scenario than what I consider likely about this particular intelligence, given that it has observed us for long enough to learn our language, revealed itself to us, communicated with us in Morse code, offered mutually beneficial trade, etc.
The most tempting belief for me is that the alien’s intentions are essentially similar to ours. I can even construct a plausible sounding argument for that as my best guess… we’re the only other species I know capable of communicating the desire for mutually beneficial trade in an artificial signalling system, so our behavior constitutes strong evidence for their behavior. OTOH, it’s pretty clear to me that the reason I’m tempted to believe that is because I can do something with that belief; it gives me a lot of traction for thinking about what to do next. (In a nutshell, I would conclude from that assumption that it means to exploit us for its long-term benefit, and whether that’s good or bad for us depends entirely on what our most valuable-to-it resources are and how it can most easily obtain them and whether we benefit from that process.) Since that has almost nothing to do with the likelihood of it being true, I should distrust my desire to believe that.
Ultimately, I think what I do is reply that I value mutually beneficial trade with them, but that I don’t actually trust them and must therefore treat them as a potential threat until I have gathered more information about them, while at the same time refraining from doing anything that would significantly reduce our chances of engaging in mutually beneficial trade in the future, and what do they think about all that?
For instance, if you think Omega is lying and completely ignore everything he says, you obviously two-box.
Why not zero-box in this case? I mean, what reason would I have to expect any money at all?
Well, as long as you believe Omega enough to think no box contains sudden death or otherwise negative utility, you’d open them to see what was inside. But yes, you might not believe Omega at all.
General question: suppose we encounter an alien. We have no idea what its motivations, values, goals, or abilities are. On the other hand, if may have observed any amount of human comm traffic from wireless EM signals since the invention of radio, and from actual spy-probes before the human invention of high tech that would detect them.
It signals us in Morse code from its remote starship, offering mutually benefitial trade.
What prior should we have about the alien’s intention? Should we use a native uniform prior that would tell us it’s as likely to mean us good as harm, and so never reply because we don’t know how it will try to influence our actions via communications? Should it tell us different agents who don’t explicitly value one another will conflict to the extent their values differ, and so since value-space is vast and a randomly selected alien is unlikely to share many values with us, we should prepare for war? Should it tell us we can make some assumptions (which?) about naturally evolved agents or their Friendly-to-themselves creations? How safe are we if we try to “just read” English text written by an unknown, possibly-superintelligence which may have observed all our broadcast traffic since the age of radio? What does our non-detection of this alien civ until they chose to initiate contact tell us? Etc.
A 50% chance of meaning us good vs harm isn’t a prior I find terribly compelling.
There’s a lot to say here, but my short answer is that this is both an incredibly dangerous and incredibly valuable situation, in which both the potential opportunity costs and the potential actual costs are literally astronomical, and in which there are very few things I can legitimately be confident of.
The best I can do in such a situation is to accept that my best guess is overwhelmingly likely to be wrong, but that it’s slightly less likely to be wrong than my second-best guess, so I should operate on the basis of my best guess despite expecting it to be wrong. Where “best guess” here is the thing I consider most likely to be true, not the thing with the highest expected value.
I should also note that my priors about aliens in general—that is, what I consider likely about a randomly selected alien intelligence—are less relevant to this scenario than what I consider likely about this particular intelligence, given that it has observed us for long enough to learn our language, revealed itself to us, communicated with us in Morse code, offered mutually beneficial trade, etc.
The most tempting belief for me is that the alien’s intentions are essentially similar to ours. I can even construct a plausible sounding argument for that as my best guess… we’re the only other species I know capable of communicating the desire for mutually beneficial trade in an artificial signalling system, so our behavior constitutes strong evidence for their behavior. OTOH, it’s pretty clear to me that the reason I’m tempted to believe that is because I can do something with that belief; it gives me a lot of traction for thinking about what to do next. (In a nutshell, I would conclude from that assumption that it means to exploit us for its long-term benefit, and whether that’s good or bad for us depends entirely on what our most valuable-to-it resources are and how it can most easily obtain them and whether we benefit from that process.) Since that has almost nothing to do with the likelihood of it being true, I should distrust my desire to believe that.
Ultimately, I think what I do is reply that I value mutually beneficial trade with them, but that I don’t actually trust them and must therefore treat them as a potential threat until I have gathered more information about them, while at the same time refraining from doing anything that would significantly reduce our chances of engaging in mutually beneficial trade in the future, and what do they think about all that?