Where does decision theory come from? It seems to come from philosophers trying to do philosophy.
An alternate view is that certain philosophical and mathematical concepts are “spotlighted”, in the sense that they seem likely to recur in a wide variety of minds above a certain intelligence / capabilities level.
A concept which is mathematically simple or elegant to describe and also instrumentally useful across a wide variety of possible universes is likely to be instrumentally convergent. The simpler and more widely useful the concept is, the more likely it is to occur by convergence in a wider variety of minds, at lower overall capabilities thresholds.
This doesn’t necessarily mean that a sufficiently spotlighted concept is “objective” or a philosophical truth, and there might be exotic corners of mindspace in which there are extremely powerful minds which can’t or don’t converge on these concepts, e.g. the one you give in Metaphilosophical Mysteries.
But note that recognizing that there are recurring useful patterns in mindspace and being able to reason correctly about them (i.e. being able to do philosophy) is itself likely to be an instrumentally useful (though perhaps not very mathematically simple) concept, and is thus itself a spotlighted concept.
(BTW, another place where Eliezer appears to at least allude to his own views on this general topic in his fiction is here.)
As for your meta question of why more people aren’t interested in this, one possibility is that there’s just not that many people who make it all the way up the ladder of intellectual journey the way you did:
Lots of people in AI and even AI safety probably haven’t heard that much, or aren’t that interested in, even the basics of mainstream game theory (Prisoners’ Dilemma, Nash Equilibria, etc.), though on LW these concepts have very high penetration.
Lots of people who have heard of the basics find object-level questions and thought experiments about specific decision theories more than interesting enough to keep arguing about forever.
Among the remaining people who tentatively accept something like UDT (or at least get far enough to have interesting takes on it), most of them probably don’t wonder about the internal mental processes by which its inventors came up with it, nor try to re-create that process in themselves.
Personally, I think something like the spotlighting concept is probably correct, and thus that it is unlikely that practical AGIs will struggle with philosophy.
I do enjoy reading about this general topic though, including your posts and some loosely related things @TsviBT has written, and I’d like to see more of them, from you and others. I’m not sure that simply drawing attention to the issue will have much effect though—the number of people both qualified and interested in writing about this is probably very small. To make real progress, you might need to bootstrap a larger community of peers.
Finding ways to move people who are stuck at lower rungs of the ladder through the funnel is one way to do that. This might take the form of more distillations and explanations of metaphilosophy, decision theory, and other object-level concepts that target a wider audience. (Or, if you’re inclined, write 1M+ words of collaborative fiction about these topics.)
An alternate view is that certain philosophical and mathematical concepts are “spotlighted”, in the sense that they seem likely to recur in a wide variety of minds above a certain intelligence / capabilities level.
A concept which is mathematically simple or elegant to describe and also instrumentally useful across a wide variety of possible universes is likely to be instrumentally convergent. The simpler and more widely useful the concept is, the more likely it is to occur by convergence in a wider variety of minds, at lower overall capabilities thresholds.
This doesn’t necessarily mean that a sufficiently spotlighted concept is “objective” or a philosophical truth, and there might be exotic corners of mindspace in which there are extremely powerful minds which can’t or don’t converge on these concepts, e.g. the one you give in Metaphilosophical Mysteries.
But note that recognizing that there are recurring useful patterns in mindspace and being able to reason correctly about them (i.e. being able to do philosophy) is itself likely to be an instrumentally useful (though perhaps not very mathematically simple) concept, and is thus itself a spotlighted concept.
(BTW, another place where Eliezer appears to at least allude to his own views on this general topic in his fiction is here.)
As for your meta question of why more people aren’t interested in this, one possibility is that there’s just not that many people who make it all the way up the ladder of intellectual journey the way you did:
Lots of people in AI and even AI safety probably haven’t heard that much, or aren’t that interested in, even the basics of mainstream game theory (Prisoners’ Dilemma, Nash Equilibria, etc.), though on LW these concepts have very high penetration.
Lots of people who have heard of the basics find object-level questions and thought experiments about specific decision theories more than interesting enough to keep arguing about forever.
Among the remaining people who tentatively accept something like UDT (or at least get far enough to have interesting takes on it), most of them probably don’t wonder about the internal mental processes by which its inventors came up with it, nor try to re-create that process in themselves.
Personally, I think something like the spotlighting concept is probably correct, and thus that it is unlikely that practical AGIs will struggle with philosophy.
I do enjoy reading about this general topic though, including your posts and some loosely related things @TsviBT has written, and I’d like to see more of them, from you and others. I’m not sure that simply drawing attention to the issue will have much effect though—the number of people both qualified and interested in writing about this is probably very small. To make real progress, you might need to bootstrap a larger community of peers.
Finding ways to move people who are stuck at lower rungs of the ladder through the funnel is one way to do that. This might take the form of more distillations and explanations of metaphilosophy, decision theory, and other object-level concepts that target a wider audience. (Or, if you’re inclined, write 1M+ words of collaborative fiction about these topics.)