https://dorcishlibrarian.net/ https://www.admonymous.co/isaacking
KingSupernova(Isaac King)
Karma: 62
Question Gravity
I think you’re confusing ignorance with other people’s beliefs about that agent’s ignorance. In your example of the police or the STD test, there is no benefit gained by that person being ignorant of the information. There is however a benefit of other people thinking the person was ignorant. If someone is able to find out whether they have an STD without anyone else knowing they’ve had that test, that’s only a benefit for them. (Not including the internal cognitive burden of having to explicitly lie.)
An open-ended probability calibration test is something I’ve been planning to build. I’d be curious to hear your thoughts on how the specifics should be implemented. How should they grade their own test in a way that avoids bias and still gives useful results?
Whether Omega ended up being right or wrong is irrelevant to the problem, since the players only find out if it was right or wrong after all decisions have been made. It has no bearing on what decision is correct at the time; only our prior probability of whether Omega will be right or wrong matters.
I think you have to consider what winning means more carefully.
A rational agent doesn’t buy a lottery ticket because it’s a bad bet. If that ticket ends up winning, does that contradict the principle that “rational agents win”?
That doesn’t seem at all analogous. At the time they had the opportunity to purchase the ticket, they had no way to know it was going to win.
An Irene who acts like your model of Irene will win slightly more when omega makes an incorrect prediction (she wins the lottery), but will be given the million dollars far less commonly because Omega is almost always correct. On average she loses. And rational agents win on average.
By average I don’t mean average within a particular world (repeated iteration), but on average across all possible worlds.
I agree with all of this. I’m not sure why you’re bringing it up?
I think you’re missing my point. After the $1,000,000 has been taken, Irene doesn’t suddenly lose her free will. She’s perfectly capable of taking the $1000; she’s just decided not to.
You seem to think I’m making some claim like “one-boxing is irrational” or “Newcomb’s problem is impossible”, which is not at all what I’m doing. I’m trying to demonstrate that the idea of “rational agents just do what maximizes their utility and don’t worry about having to have a consistent underlying decision theory” appears to result in a contradiction as soon as Irene’s decision has been made.
Ah, that makes sense.
Some clarifications on my intentions writing this story.
Omega being dead and Irene having taken the money from one box before having the conversation with Rachel are both not relevant to the core problem. I included them as a literary flourish to push people’s intuitions towards thinking that Irene should open the second box, similar to what Eliezer was doing here.
Omega was wrong in this scenario, which departs from the traditional Newcomb’s problem. I could have written an ending where Rachel made the same arguments and Irene still decided against doing it, but that seemed less fun. It’s not relevant whether Omega was right or wrong, because after Irene has made her decision, she always has the “choice” to take the extra money and prove Omega wrong. My point here is that leaving the $1000 behind falls prey to the same “rational agents should win” problem that’s usually used to justify one-boxing. After taking the $1,000,000 you can either have some elaborate justification for why it would be irrational to open the second box, or you could just… do it.
Here’s another version of the story that might demonstrate this more succinctly:
Irene wakes up in her apartment one morning and finds Omega standing before her with $1,000,000 on her bedside table and a box on the floor next to it. Omega says “I predicted your behavior in Newcomb’s problem and guessed that you’d take only one box, so I’ve expedited the process and given you the money straight away, no decision needed. There’s $1000 in that box on the floor, you can throw it away in the dumpster out back. I have 346 other thought experiments to get to today, so I really need to get going.”
I just did that to be consistent with the traditional formulation of Newcomb’s problem, it’s not relevant to the story. I needed some labels for the boxes, and “box A” and “box B” are not very descriptive and make it easy for the reader to forget which is which.
I don’t find the simulation argument very compelling. I can conceive of many ways for Omega to arrive at a prediction with high probability of being correct that don’t involve a full, particle-by-particle simulation of the actors.
In the case where you find yourself holding the $1,000,000 and the $1000 are still available, sure, you can pick them up. That only happens if either Omega failed to predict what you will do, or if you somehow set things up such that you couldn’t, or had to pay a big price, to break your precommitment.
I don’t think that’s true. The traditional Newcomb’s problem could use the exact setup that I used here, the only difference would be that either the opaque box is empty, or Irene never opens the transparent box. The idea that the $1000 is always “available” to the player is central to Newcomb’s problem.
I don’t find the simulation argument very compelling. I can conceive of many ways for Omega to arrive at a prediction with high probability of being correct that don’t involve a full, particle-by-particle simulation of the actors.
“Rational Agents Win”
making piece
should be
making peace
so it includes both asymptomatic cases
I think that “includes” should be “excludes”?
This is an interesting question, but I think your hypothesis is wrong.
Any pattern of physics that eventually exerts control over a region much larger than its initial configuration does so by means of perception, cognition, and action that are recognizably AI-like.
In order to not include things like an exploding supernova as “controlling a region much larger than its initial configuration” we would want to require that such patterns be capable of arranging matter and energy into an arbitrary but low-complexity shape, such as a giant smiley face in Life.
If it is indeed possible to build some sort of simple and robust “area clearer” pattern as the other comments discuss (related post here), then nothing approaching the complexity of an AI is necessary. Simply initialize the small region with the area clearer pattern facing outwards and a smiley face constructer behind it.
It seems to me that the only reason you’d need an AI for this is if that sort of “area clearer” pattern is not possible. In that case, the only way to clear the large square would be to individually sense and interact with each piece of ash and tailor a response to its structure, and you’d need something somewhat “intelligent” to do that. Even in this case, I’m unconvinced that the amount of “intelligence” necessary to do this would put it on par with what we think of as a general intelligence.
Wouldn’t an observed mismatch between assigned probability and observed probability count as Bayesian evidence towards miscalibration?