How important is conceptual reasoning capability for safety work?
FWIW I think conceptual reasoning is essential for good safety work. It’s also possible that humans aren’t good enough at conceptual reasoning to achieve a flourishing future (e.g. see Wei Dai’s shortform).
Unfortunately, conceptual reasoning also makes misaligned AI far more dangerous. So I am very worried about attempts to improve AIs’ conceptual reasoning, and I lean toward it being net harmful right now.
Hmmm, maybe it was unclear. I was just trying to communicate that, I think there are areas where we could use AIs to help us, including things that could help us build safe AIs. But I think the tasks you’d hope be helped by conceptual reasoning, are ones it would be very dangerous to have AIs be good at, and very dangerous to have them work on, so we shouldn’t do that. But it would be very helpful if we could.
Cruxes:
How important is conceptual reasoning capability for safety work? (You suggest it’s not super important; the authors think it is.)
How scary is conceptual reasoning capability?
FWIW I think conceptual reasoning is essential for good safety work. It’s also possible that humans aren’t good enough at conceptual reasoning to achieve a flourishing future (e.g. see Wei Dai’s shortform).
Unfortunately, conceptual reasoning also makes misaligned AI far more dangerous. So I am very worried about attempts to improve AIs’ conceptual reasoning, and I lean toward it being net harmful right now.
I would answer “extremely” and “extremely” to both of those questions, so I don’t think the first one is a crux.
OK. fyi my inference was based on your last two paragraphs.
Hmmm, maybe it was unclear. I was just trying to communicate that, I think there are areas where we could use AIs to help us, including things that could help us build safe AIs. But I think the tasks you’d hope be helped by conceptual reasoning, are ones it would be very dangerous to have AIs be good at, and very dangerous to have them work on, so we shouldn’t do that. But it would be very helpful if we could.