If I imagine being as confused as economists are about CDT, I do really repeatedly end up making very dumb and wrong predictions about what e.g. AIs would do when you have many copies of them, and they try to coordinate with each other.
Like, it rarely happens that I have a conversation about either technical AI safety, or about AI strategy, where decision theory considerations don’t at least come up once in some form or another. Not having an answer here feels probably like people must have felt before we had probability theory, which also I have no idea what I would do without.
I think this is a good argument for understanding basic decision theory points, but I don’t think it leads to you needing to develop any fancier decision theory—arguing about what decision theory AIs will use just requires thinking about descriptive facts about decision theories, rather than coming up with decision theories that work well in limits that aren’t important for the most important kinds of AI futurism (including the AI futurism questions I think you’re talking about here).
“Basic decision theory points” feels like a pretty weird description of something that even quite smart people still frequently disagree on, has no formal description, and indeed often turns out to be the crux of an argument.
I currently don’t think it’s worth my time figuring things out here much more, but that’s mostly because I do have some reasonable confidence that thinking about decision theory harder probably won’t produce any quick breakthroughs. But if I was in the world that MIRI faced 15 years ago, my guess is I would have thought it was worth investing in quite a bit, in case it does turn out to be relatively straightforward (which it so far has not turned out to be).
rather than coming up with decision theories that work well in limits that aren’t important for the most important kinds of AI futurism
Pushing more straightforwardly back on this: I do not think our current understanding of decision-theory is better in the mundane case than the limit case. Of course the reason to look at limiting cases is because you always do that in math because the limiting cases often turn out easier, not harder than the mundane case.
If I imagine being as confused as economists are about CDT, I do really repeatedly end up making very dumb and wrong predictions about what e.g. AIs would do when you have many copies of them, and they try to coordinate with each other.
Like, it rarely happens that I have a conversation about either technical AI safety, or about AI strategy, where decision theory considerations don’t at least come up once in some form or another. Not having an answer here feels probably like people must have felt before we had probability theory, which also I have no idea what I would do without.
I think this is a good argument for understanding basic decision theory points, but I don’t think it leads to you needing to develop any fancier decision theory—arguing about what decision theory AIs will use just requires thinking about descriptive facts about decision theories, rather than coming up with decision theories that work well in limits that aren’t important for the most important kinds of AI futurism (including the AI futurism questions I think you’re talking about here).
“Basic decision theory points” feels like a pretty weird description of something that even quite smart people still frequently disagree on, has no formal description, and indeed often turns out to be the crux of an argument.
I currently don’t think it’s worth my time figuring things out here much more, but that’s mostly because I do have some reasonable confidence that thinking about decision theory harder probably won’t produce any quick breakthroughs. But if I was in the world that MIRI faced 15 years ago, my guess is I would have thought it was worth investing in quite a bit, in case it does turn out to be relatively straightforward (which it so far has not turned out to be).
Pushing more straightforwardly back on this: I do not think our current understanding of decision-theory is better in the mundane case than the limit case. Of course the reason to look at limiting cases is because you always do that in math because the limiting cases often turn out easier, not harder than the mundane case.