Adele suggested this above. You can see my and Eliezer’s response there. The basic worry is that Cartesians have no way to FOOM, because they’re unlikely to form intelligent hypotheses about self-modifications. So a real Cartesian won’t be an AGI, or will only barely be an AGI. Our work should go into something more useful than that, since it’s possible that in the time it takes us to build a moderately useful Cartesian AI that doesn’t immediately destroy itself, we could have invented FAI or proto-FAI.
Non-FAI isn’t what we’re acutely scared of; UFAI (i.e., superintelligence without human values) is. Failing to build a superintelligence is not the same thing as preventing others from building a dangerous superintelligence. So self-handicapping isn’t generically useful, especially when most AI researchers won’t handicap themselves in the same way.
Adele suggested this above. You can see my and Eliezer’s response there. The basic worry is that Cartesians have no way to FOOM, because they’re unlikely to form intelligent hypotheses about self-modifications. So a real Cartesian won’t be an AGI, or will only barely be an AGI. Our work should go into something more useful than that, since it’s possible that in the time it takes us to build a moderately useful Cartesian AI that doesn’t immediately destroy itself, we could have invented FAI or proto-FAI.
Non-FAI isn’t what we’re acutely scared of; UFAI (i.e., superintelligence without human values) is. Failing to build a superintelligence is not the same thing as preventing others from building a dangerous superintelligence. So self-handicapping isn’t generically useful, especially when most AI researchers won’t handicap themselves in the same way.