I agree with this statement denotatively, and my own interests/work have generally been “driven by open-ended curiosity and a drive to uncover deep truths”, but isn’t this kind of motivation also what got humanity into its current mess? In other words, wasn’t the main driver of AI progress this kind of curiosity (until perhaps the recent few years when it has been driven more by commercial/monetary/power incentives)?
Interestingly, I was just having a conversation with Critch about this. My contention was that, in the first few decades of the field, AI researchers were actually trying to understand cognition. The rise of deep learning (and especially the kind of deep learning driven by massive scaling) can be seen as the field putting that quest on hold in order to optimize for more legible metrics.
I don’t think you should find this a fully satisfactory answer, because it’s easy to “retrodict” ways that my theory was correct. But that’s true of all explanations of what makes the world good at a very abstract level, including your own answer of metaphilosophical competence. (Also, we can perhaps cash my claim out in predictions, like: was a significant barrier to more researchers working on deep learning the criticism that it didn’t actually provide good explanations of or insight into cognition? Without having looked it up, I suspect so.)
consistently good strategy requires a high amount of consequentialist reasoning
I don’t think that’s true. However I do think it requires deep curiosity about what good strategy is and how it works. It’s not a coincidence that my own research on a theory of coalitional agency was in significant part inspired by strategic failures of EA and AI safety (with this post being one of the earliest building blocks I laid down). I also suspect that the full theory of coalitional agency will in fact explain how to do metaphilosophy correct, because doing good metaphilosophy is ultimately a cognitive process and can therefore be characterized by a sufficiently good theory of cognition.
Again, I don’t expect you to fully believe me. But what I most want to read from you right now is an in-depth account of which the things in the world have gone or are going most right, and the ways in which you think metaphilosophical competence or consequentialist reasoning contributed to them. Without that, it’s hard to trust metaphilosophy or even know what it is (though I think you’ve given a sketch of this in a previous reply to me at some point).
I should also try to write up the same thing, but about how virtues contributed to good things. And maybe also science, insofar as I’m trying to defend doing more science (of cognition and intelligence) in order to help fix risks caused by previous scientific progress.
But what I most want to read from you right now is an in-depth account of which the things in the world have gone or are going most right, and the ways in which you think metaphilosophical competence or consequentialist reasoning contributed to them.
(First a terminological note: I wouldn’t use the phrase “metaphilosophical competence”, and instead tend to talk about either “metaphilosophy”, meaning studying the nature of philosophy and philosophical reasoning, how should philosophical problems be solved, etc., or “philosophical competence”, meaning how good someone is at solving philosophical problems or doing philosophical reasoning. And sometimes I talk about them together, like in “metaphilosophy / AI philosophical competence” because I think solving metaphilosophy is the best way to improve AI philosophical competence. Here I’ll interpret you to just mean “philosophical competence”.)
To answer your question, it’s pretty hard to think of really good examples, I think because humans are very bad at both philosophical competence and consequentialist reasoning, but here are some:
the game theory around nuclear deterrence, helping to prevent large-scale war so far
economics and its influence on government policy, e.g., providing support for property rights, markets, and regulations around things like monopolies and externalities (but it’s failing pretty badly on AGI/ASI)
analytical philosophy making philosophical progress in so far as asking important questions and delineating various plausible answers (but doing badly as far as individually having inappropriate levels of confidence, as well as failing to focus on the really important problems, e.g., related to AI safety)
certain philosophers / movements (rationalists, EA) emphasizing philosophical (especially moral) uncertainty to some extent, and realizing the importance of AI safety
MIRI updating on evidence/arguments and pivoting strategy in response (albeit too slowly)
I guess this isn’t an “in-depth account” but I’m also not sure why you’re asking for “in-depth”, i.e., why doesn’t a list like this suffice?
I should also try to write up the same thing, but about how virtues contributed to good things.
I think non-consequentialist reasoning or ethics probably worked better in the past, when the world changed more slowly and we had more chances to learn from our mistakes (and refine our virtues/deontology over time), so I wouldn’t necessarily find this kind of writing very persuasive, unless it somehow addressed my central concern that virtues do not seem to be a kind of thing that is capable of doing enough “compute/reasoning” to find consistently good strategies in a fast changing environment on the first try.
Interestingly, I was just having a conversation with Critch about this. My contention was that, in the first few decades of the field, AI researchers were actually trying to understand cognition. The rise of deep learning (and especially the kind of deep learning driven by massive scaling) can be seen as the field putting that quest on hold in order to optimize for more legible metrics.
I don’t think you should find this a fully satisfactory answer, because it’s easy to “retrodict” ways that my theory was correct. But that’s true of all explanations of what makes the world good at a very abstract level, including your own answer of metaphilosophical competence. (Also, we can perhaps cash my claim out in predictions, like: was a significant barrier to more researchers working on deep learning the criticism that it didn’t actually provide good explanations of or insight into cognition? Without having looked it up, I suspect so.)
I don’t think that’s true. However I do think it requires deep curiosity about what good strategy is and how it works. It’s not a coincidence that my own research on a theory of coalitional agency was in significant part inspired by strategic failures of EA and AI safety (with this post being one of the earliest building blocks I laid down). I also suspect that the full theory of coalitional agency will in fact explain how to do metaphilosophy correct, because doing good metaphilosophy is ultimately a cognitive process and can therefore be characterized by a sufficiently good theory of cognition.
Again, I don’t expect you to fully believe me. But what I most want to read from you right now is an in-depth account of which the things in the world have gone or are going most right, and the ways in which you think metaphilosophical competence or consequentialist reasoning contributed to them. Without that, it’s hard to trust metaphilosophy or even know what it is (though I think you’ve given a sketch of this in a previous reply to me at some point).
I should also try to write up the same thing, but about how virtues contributed to good things. And maybe also science, insofar as I’m trying to defend doing more science (of cognition and intelligence) in order to help fix risks caused by previous scientific progress.
(First a terminological note: I wouldn’t use the phrase “metaphilosophical competence”, and instead tend to talk about either “metaphilosophy”, meaning studying the nature of philosophy and philosophical reasoning, how should philosophical problems be solved, etc., or “philosophical competence”, meaning how good someone is at solving philosophical problems or doing philosophical reasoning. And sometimes I talk about them together, like in “metaphilosophy / AI philosophical competence” because I think solving metaphilosophy is the best way to improve AI philosophical competence. Here I’ll interpret you to just mean “philosophical competence”.)
To answer your question, it’s pretty hard to think of really good examples, I think because humans are very bad at both philosophical competence and consequentialist reasoning, but here are some:
the game theory around nuclear deterrence, helping to prevent large-scale war so far
economics and its influence on government policy, e.g., providing support for property rights, markets, and regulations around things like monopolies and externalities (but it’s failing pretty badly on AGI/ASI)
analytical philosophy making philosophical progress in so far as asking important questions and delineating various plausible answers (but doing badly as far as individually having inappropriate levels of confidence, as well as failing to focus on the really important problems, e.g., related to AI safety)
certain philosophers / movements (rationalists, EA) emphasizing philosophical (especially moral) uncertainty to some extent, and realizing the importance of AI safety
MIRI updating on evidence/arguments and pivoting strategy in response (albeit too slowly)
I guess this isn’t an “in-depth account” but I’m also not sure why you’re asking for “in-depth”, i.e., why doesn’t a list like this suffice?
I think non-consequentialist reasoning or ethics probably worked better in the past, when the world changed more slowly and we had more chances to learn from our mistakes (and refine our virtues/deontology over time), so I wouldn’t necessarily find this kind of writing very persuasive, unless it somehow addressed my central concern that virtues do not seem to be a kind of thing that is capable of doing enough “compute/reasoning” to find consistently good strategies in a fast changing environment on the first try.