Untestability: you cannot safely experiment on near-ASI (I mean, you can, but you’re not guaranteed not to cross the threshold into the danger zone, and the authors believe that anything you can learn from before won’t be too useful).
I think “won’t be too useful” is kinda misleading. Point is more like “it’s at least as difficult as launching a rocket into space without good theory about how gravity works and what the space is”. Early tests and experiments are useful! They can help you with the theory! You just want to be completely sure that you are not in your test rocket yourself.
At times the authors appeal to prominent figures as evidence that the danger is widely acknowledged. At other times, the book paints the entire ML and AI safety ecosystem as naive, reckless, or intellectually unserious.
I see no contradiction between these two statements:
Prominent figures and also median experts believe that the risks are at the level we can surely call totally unacceptable (even if some experts themselves consider it acceptable)
Current field of AI research can’t make much progress on AI alignment problem.
People totally can know about the risk without also knowing what to do about it.
I think “won’t be too useful” is kinda misleading. Point is more like “it’s at least as difficult as launching a rocket into space without good theory about how gravity works and what the space is”. Early tests and experiments are useful! They can help you with the theory! You just want to be completely sure that you are not in your test rocket yourself.
I see no contradiction between these two statements:
Prominent figures and also median experts believe that the risks are at the level we can surely call totally unacceptable (even if some experts themselves consider it acceptable)
Current field of AI research can’t make much progress on AI alignment problem.
People totally can know about the risk without also knowing what to do about it.