Oh yeah, an AGI with consequentialist preferences would definitely want to grab control of that button. (Other things equal.) I’ll edit to mention that explicitly. Thanks.
I think I mentioned in the section that I didn’t (and still don’t) have any actual good AI-alignment-helping plan for the §9.7 thing. So arguably I could have omitted that section entirely. But I was figuring that someone else might think of something, I guess. :)
Oh yeah, an AGI with consequentialist preferences would definitely want to grab control of that button. (Other things equal.) I’ll edit to mention that explicitly. Thanks.
I think I mentioned in the section that I didn’t (and still don’t) have any actual good AI-alignment-helping plan for the §9.7 thing. So arguably I could have omitted that section entirely. But I was figuring that someone else might think of something, I guess. :)