I was going to say that this seems like another one of Eliezer’s founder effects, but he actually wrote about not trusting humans with too much power, in Creating Friendly AI 1.0 (2001):
Among humans, the only practical way to maximize actual freedom (the percentage of actions executed without interference) is to ensure that no human entity has the ability to interfere with you—a consequence of humans having an innate, evolved tendency to abuse power. Thus, a lot of our ethical guidelines (especially the ones we’ve come up with in the twentieth century) state that it’s wrong to acquire too much power.
If this is one of those things that simply doesn’t apply in the spaces beyond the Singularity—if, having no evolved tendency to abuse power, no injunction against the accumulation of power is necessary—one of the possible resolutions of the Singularity would be the Sysop Scenario. The initial seed-AI-turned-Friendly-superintelligence, the Transition Guide, would create (or self-modify into) a superintelligence that would act as the underlying operating system for all the matter in human space—a Sysop.
I wonder how to interpret e.g. the ending of HPMOR in light of this.
HJPEV is bound by a magical oath that prevents this human failing in the same way it is prevented in an agent that meets tiling desiderata. This is explicit in the text. E-Book draft, 2015, chapter 113.
Admittedly this both assumes that the “time of peril” hypothesis is correct and can be handled while maintaing human freedom, and the solution only (in maximum robustness) binds until the end of this time.
My understanding of HPMOR is limited as I’ve only read a few chapters, but looking up the text you cite, it doesn’t seem to prevent most forms of abuse of power.
I was going to say that this seems like another one of Eliezer’s founder effects, but he actually wrote about not trusting humans with too much power, in Creating Friendly AI 1.0 (2001):
I wonder how to interpret e.g. the ending of HPMOR in light of this.
Spoiler
HJPEV is bound by a magical oath that prevents this human failing in the same way it is prevented in an agent that meets tiling desiderata. This is explicit in the text. E-Book draft, 2015, chapter 113.
Admittedly this both assumes that the “time of peril” hypothesis is correct and can be handled while maintaing human freedom, and the solution only (in maximum robustness) binds until the end of this time.
My understanding of HPMOR is limited as I’ve only read a few chapters, but looking up the text you cite, it doesn’t seem to prevent most forms of abuse of power.