“More research needed” but here are some ideas to start with:
Try to design alignment/safety schemes that are agnostic or don’t depend on controversial philosophical ideas. For certain areas that seem highly relevant and where there could potentially be hidden dependencies (such as metaethics), explicitly understand and explain why, under each plausible position that people currently hold, the alignment/safety scheme will result in a good or ok outcome. (E.g., why it leads to a good outcome regardless of whether moral realism or anti-realism is true, or any one of the other positions.)
Try to solve metaphilosophy, where potentially someone could make a breakthrough that everyone can agree is correct (after extensive review), which can then be used to speed up progress in all other philosophical fields. (This could also happen in another philosophical field, but seems a lot less likely due to prior efforts/history. I don’t think it’s very likely in metaphilosophy either, but perhaps worth a try, for those who may have very strong comparative advantage in this.)
If 1 and 2 look hard or impossible, make this clear to non-experts (your boss, company leaders/board, government officials, the public), don’t let them accept a “roll your own metaethics” solution, or a solution with implicit/hidden philosophical assumptions.
#2 feels like it’s injecting some frame that’s a bit weird to inject here (don’t roll your own metaethics… but rolling your own metaphilosophy is okay?)
But also, I’m suddenly confused about who this post is trying to warn. Is it more like labs, or more like EA-ish people doing a wider variety of meta-work?
#2 feels like it’s injecting some frame that’s a bit weird to inject here (don’t roll your own metaethics… but rolling your own metaphilosophy is okay?)
Maybe you missed my footnote?
To preempt a possible misunderstanding, I don’t mean “don’t try to think up new metaethical ideas”, but instead “don’t be so confident in your ideas that you’d be willing to deploy them in a highly consequential way, or build highly consequential systems that depend on them in a crucial way”. Similarly “don’t roll your own crypto” doesn’t mean never try to invent new cryptography, but rather don’t deploy it unless there has been extensive review, and consensus that it is likely to be secure.
and/or this part of my answer (emphasis added):
Try to solve metaphilosophy, where potentially someone could make a breakthrough that everyone can agree is correct (after extensive review)
But also, I’m suddenly confused about who this post is trying to warn. Is it more like labs, or more like EA-ish people doing a wider variety of meta-work?
I think I mostly had alignment researchers (in and out of labs) as the target audience in mind, but it does seem relevant to others so perhaps I should expand the target audience?
To preempt a possible misunderstanding, I don’t mean “don’t try to think up new metaethical ideas”, but instead “don’t be so confident in your ideas that you’d be willing to deploy them in a highly consequential way, or build highly consequential systems that depend on them in a crucial way”.
I think I had missed this, but, it doesn’t resolve the confusion in my #2 note. (like, still seems like something is weird about saying “solve metaphilosophy such that every can agree is correct” is more worth considering than “solve metaethics such that everyone can agree is correct”. I can totally buy that they’re qualitatively different and maybe have some guesses for why you think that. But I don’t think the post spells out why and it doesn’t seem that obvious to me)
I hinted at it with “prior efforts/history”, but to spell it out more, metaethics seems to have a lot more effort gone into it in the past, so there’s less likely to be some kind of low hanging fruit in idea space, that once picked, everyone will agree is the right solution.
What are you supposed to do other than roll your own metaethics?
“More research needed” but here are some ideas to start with:
Try to design alignment/safety schemes that are agnostic or don’t depend on controversial philosophical ideas. For certain areas that seem highly relevant and where there could potentially be hidden dependencies (such as metaethics), explicitly understand and explain why, under each plausible position that people currently hold, the alignment/safety scheme will result in a good or ok outcome. (E.g., why it leads to a good outcome regardless of whether moral realism or anti-realism is true, or any one of the other positions.)
Try to solve metaphilosophy, where potentially someone could make a breakthrough that everyone can agree is correct (after extensive review), which can then be used to speed up progress in all other philosophical fields. (This could also happen in another philosophical field, but seems a lot less likely due to prior efforts/history. I don’t think it’s very likely in metaphilosophy either, but perhaps worth a try, for those who may have very strong comparative advantage in this.)
If 1 and 2 look hard or impossible, make this clear to non-experts (your boss, company leaders/board, government officials, the public), don’t let them accept a “roll your own metaethics” solution, or a solution with implicit/hidden philosophical assumptions.
Support AI pause/stop.
Hmm, I like #1.
#2 feels like it’s injecting some frame that’s a bit weird to inject here (don’t roll your own metaethics… but rolling your own metaphilosophy is okay?)
But also, I’m suddenly confused about who this post is trying to warn. Is it more like labs, or more like EA-ish people doing a wider variety of meta-work?
Maybe you missed my footnote?
and/or this part of my answer (emphasis added):
I think I mostly had alignment researchers (in and out of labs) as the target audience in mind, but it does seem relevant to others so perhaps I should expand the target audience?
I think I had missed this, but, it doesn’t resolve the confusion in my #2 note. (like, still seems like something is weird about saying “solve metaphilosophy such that every can agree is correct” is more worth considering than “solve metaethics such that everyone can agree is correct”. I can totally buy that they’re qualitatively different and maybe have some guesses for why you think that. But I don’t think the post spells out why and it doesn’t seem that obvious to me)
I hinted at it with “prior efforts/history”, but to spell it out more, metaethics seems to have a lot more effort gone into it in the past, so there’s less likely to be some kind of low hanging fruit in idea space, that once picked, everyone will agree is the right solution.