Please notice that I never said that an AGI won’t be unsafe.
If you admit that it is possible that at some point we can be using AGIs to verify certain theorems, then we pretty much agree. Other people wouldn’t agree with that because they will tell you that humanity ends as soon as we have an AGI, and this is the idea I am trying to fight against
I think no one, incuding EY, doesn’t think “humanity ends as soon as we have an AGI”. Actual opinion is “Agentic AGI that optimize something and ends humanity in the process will probably by default be created before we will solve alignment or will be able to commit pivotal act that prevents the creation of such AGI”. As I understand, EY thinks that we probably can create non-agentic or weak AGI that will not kill us all, but it will not prevent strong agentic AGI that will.
Please notice that I never said that an AGI won’t be unsafe.
If you admit that it is possible that at some point we can be using AGIs to verify certain theorems, then we pretty much agree. Other people wouldn’t agree with that because they will tell you that humanity ends as soon as we have an AGI, and this is the idea I am trying to fight against
The AGI will kill us in other ways than its theorem proofs being either-hard-to-check-or-useless, but it will kill us nevertheless.
I think no one, incuding EY, doesn’t think “humanity ends as soon as we have an AGI”. Actual opinion is “Agentic AGI that optimize something and ends humanity in the process will probably by default be created before we will solve alignment or will be able to commit pivotal act that prevents the creation of such AGI”. As I understand, EY thinks that we probably can create non-agentic or weak AGI that will not kill us all, but it will not prevent strong agentic AGI that will.