I’d just say that we’ve never known anything in advance for certain. The idea that we’d be able to prove analytically that an ASI was safe was always hopeless. And that’s been accepted by most of the alignment community for some time. I don’t see that this perspective changes the predictive power of the theories we’re applying.
I think the interesting discussion is not about how certain exactly our predictions of doom are or can be.
Let me put the central point another way: However pessimistic you are about the success of alignment you should become more pessimistic once you realize that alignment requires the prediction of an AIs actions. Any notion that we could circumvent this by engineering values into the system is illusory.
I’d just say that we’ve never known anything in advance for certain. The idea that we’d be able to prove analytically that an ASI was safe was always hopeless. And that’s been accepted by most of the alignment community for some time. I don’t see that this perspective changes the predictive power of the theories we’re applying.
I think the interesting discussion is not about how certain exactly our predictions of doom are or can be.
Let me put the central point another way: However pessimistic you are about the success of alignment you should become more pessimistic once you realize that alignment requires the prediction of an AIs actions. Any notion that we could circumvent this by engineering values into the system is illusory.