I’m not really sure what you’re asking here. The way I see it, we currently do not have any game theory or single-player decision theory that is both well-defined and do not have obvious problems. Take conventional game theory for example. Before you can use it, you have to tell it where “agents” are located in the world. How do we distinguish between an agent and a non-agent (some piece of machinery that happens to exist in the world)?
If you’re asking whether you should work on these problems, well, somebody has to, why not you? If you’re asking whether you should work on single-player decision theory and hope that multi-player solutions fall out, or work on ideas or math that may be more directly applicable to multi-player game, that itself seems like a hard problem involving lots of logical uncertainty. Given that we don’t have good theories about what to do when faced with such logical uncertainty, it’s not clear that doing lots of explicit analysis is better than just following your gut instincts.
Right, we don’t know how to choose the best direction of inquiry under logical uncertainty. But unexamined gut instinct can lead you down a blind alley, as I tried to show in the post.
I spent way too much time trying to fix the current proof-theoretic algorithms and the effort has mostly failed. Maybe it failed because I wishfully thought that a good algorithm must exist without any good reason, like Eliezer with his failed disproof of Cantor’s theorem.
Now I’m asking, perhaps in a roundabout way, what directions of inquiry could have better odds of making things clearer. My current answer is to reallocate time to searching for impossibility proofs (like W/U/A but hopefully better) and studying multiplayer games (like dividing a cake by majority vote, or Stuart’s problem about uncertainty over utility functions). I’d like to hear others’ opinions, though. Especially yours. What direction does your gut instinct consider the most fruitful?
I’m not really sure what you’re asking here. The way I see it, we currently do not have any game theory or single-player decision theory that is both well-defined and do not have obvious problems. Take conventional game theory for example. Before you can use it, you have to tell it where “agents” are located in the world. How do we distinguish between an agent and a non-agent (some piece of machinery that happens to exist in the world)?
If you’re asking whether you should work on these problems, well, somebody has to, why not you? If you’re asking whether you should work on single-player decision theory and hope that multi-player solutions fall out, or work on ideas or math that may be more directly applicable to multi-player game, that itself seems like a hard problem involving lots of logical uncertainty. Given that we don’t have good theories about what to do when faced with such logical uncertainty, it’s not clear that doing lots of explicit analysis is better than just following your gut instincts.
Right, we don’t know how to choose the best direction of inquiry under logical uncertainty. But unexamined gut instinct can lead you down a blind alley, as I tried to show in the post.
I spent way too much time trying to fix the current proof-theoretic algorithms and the effort has mostly failed. Maybe it failed because I wishfully thought that a good algorithm must exist without any good reason, like Eliezer with his failed disproof of Cantor’s theorem.
Now I’m asking, perhaps in a roundabout way, what directions of inquiry could have better odds of making things clearer. My current answer is to reallocate time to searching for impossibility proofs (like W/U/A but hopefully better) and studying multiplayer games (like dividing a cake by majority vote, or Stuart’s problem about uncertainty over utility functions). I’d like to hear others’ opinions, though. Especially yours. What direction does your gut instinct consider the most fruitful?
ETA: Wei has replied by private communication :-)