What do you mean by a “more limited pivotal task”? Trying to align an AGI towards “be nice to humans in this complex manner we have trouble defining ourselves” seems more limited than “merely” aligning for a semi-static dystopia which I would imagine occupies more phase space in the range of possible future worlds.
What do you mean by a “more limited pivotal task”? Trying to align an AGI towards “be nice to humans in this complex manner we have trouble defining ourselves” seems more limited than “merely” aligning for a semi-static dystopia which I would imagine occupies more phase space in the range of possible future worlds.