It seems worth calling out that Scott isn’t saying that orthagonality is impossible, just claiming that it’s harder than non-orthagonality:
Yes, there could be a superintelligence that cared for nothing but maximizing paperclips—in the same way that there exist humans with 180 IQs, who’ve mastered philosophy and literature and science as well as any of us, but who now mostly care about maximizing their orgasms or their heroin intake. But, like, that’s a nontrivial achievement! When intelligence and goals are that orthogonal, there was normally some effort spent prying them apart.
So I think the claim regarding your graphs is that it’s easier to build above the line than below, not that that it’s impossible.
It seems worth calling out that Scott isn’t saying that orthagonality is impossible, just claiming that it’s harder than non-orthagonality:
So I think the claim regarding your graphs is that it’s easier to build above the line than below, not that that it’s impossible.
Okay, a “hard zone” rather than a no-go zone. Which begs the question “How hard?” and consequently how much comfort should one take in the belief?
Thank you for reading and commenting.