Einstein and von Neumann were also nowhere near superintelligent; they are far better representatives of regular humans than of superintelligences. I think the problem goes deeper. As you apply more and more optimization pressure, statistical guarantees begin to fall apart. You don’t get sub-agent alignment for free, whether it’s made of carbon or silicon. Case in point, human values have become adrift over time relative to the original goal of inclusive genetic fitness.
Einstein and von Neumann were also nowhere near superintelligent; they are far better representatives of regular humans than of superintelligences. I think the problem goes deeper. As you apply more and more optimization pressure, statistical guarantees begin to fall apart. You don’t get sub-agent alignment for free, whether it’s made of carbon or silicon. Case in point, human values have become adrift over time relative to the original goal of inclusive genetic fitness.
OK, yeah, fair enough. Still though, the danger seems less than it is in the machine intelligence case.