While I agree with the existence of severe problems with controllable superintelligence, I completely disagree that the arguments in this post prove anything about impossibility. The arguments are primarily about grossly non-central semantics, and not the meaningful substance of these terms.
Furthermore, addressing two extreme examples of a multidimensional space (poorly) and handwaving the rest by saying “trades off between control and safety, but without guaranteeing either” is just lazy writing. No support is given for the assertion that nothing in that space can possibly work, let alone the wider assertion that nothing at all can work.
There are certainly some good points in this post, but I’d much prefer to see the definitive claims weakened to match the support given for them.
While I agree with the existence of severe problems with controllable superintelligence, I completely disagree that the arguments in this post prove anything about impossibility. The arguments are primarily about grossly non-central semantics, and not the meaningful substance of these terms.
Furthermore, addressing two extreme examples of a multidimensional space (poorly) and handwaving the rest by saying “trades off between control and safety, but without guaranteeing either” is just lazy writing. No support is given for the assertion that nothing in that space can possibly work, let alone the wider assertion that nothing at all can work.
There are certainly some good points in this post, but I’d much prefer to see the definitive claims weakened to match the support given for them.