Unaugmented humanity can’t be a signatory to a non-fake deal with a superintelligence, because we can’t enforce it or verify its validity. Any such “deal” would end with the superintelligence backstabbing us once we’re no longer useful. See more here.
A possible counter-proposal is to request, as part of the deal, that the superintelligence provides us with tools we can use to verify that it will comply with the deal/tools to bind it to the deal. That also won’t work: any tools it provides us will be poisoned in some manner, guaranteed not to actually work.
Yes, even if we request those tools to be e. g. mathematically verifiable or something. They would just be optimized to exploit bugs in our proof-verifiers, or bugs in human minds that would cause us to predictably and systemically misunderstand what the tools actually do, etc. See more here.
Unaugmented humanity can’t be a signatory to a non-fake deal with a superintelligence, because we can’t enforce it or verify its validity. Any such “deal” would end with the superintelligence backstabbing us once we’re no longer useful. See more here.
A possible counter-proposal is to request, as part of the deal, that the superintelligence provides us with tools we can use to verify that it will comply with the deal/tools to bind it to the deal. That also won’t work: any tools it provides us will be poisoned in some manner, guaranteed not to actually work.
Yes, even if we request those tools to be e. g. mathematically verifiable or something. They would just be optimized to exploit bugs in our proof-verifiers, or bugs in human minds that would cause us to predictably and systemically misunderstand what the tools actually do, etc. See more here.