the long-term survival of life will still probably require solving the alignment problem eventually.
Does it? What if we just don’t, and find other robust ways to prevent the creation of AGI, and other ways to have a very hopeworthy future?
There’s a possibility that even in (or aiming at) an AGI-less very hopeworthy future, we’d need to figure out a bunch of stuff that would be contained in The Textbook From The Future, e.g., regarding how to ensure that strong intelligence amplification (or whatever other strong intentional changes to how the human mind works) doesn’t change humans’ values in undesirable ways, or how to create a governance system that actually robustly respects and nourishes the agency of its constituents, while also preventing that agency from overflowing catastrophically.
I qualitatively agree, but maybe quantitatively disagree? Not sure what “a bunch” means in relative terms. I’d say the lion’s share isn’t necessary to solve in human(e)-only futures. You just get so many advantages with human(e) minds vs. alien minds. But yeah, I think figuring out good governance is important for building a hopeworthy future, especially for building one that is legibly so.
There’s a possibility that even in (or aiming at) an AGI-less very hopeworthy future, we’d need to figure out a bunch of stuff that would be contained in The Textbook From The Future, e.g., regarding how to ensure that strong intelligence amplification (or whatever other strong intentional changes to how the human mind works) doesn’t change humans’ values in undesirable ways, or how to create a governance system that actually robustly respects and nourishes the agency of its constituents, while also preventing that agency from overflowing catastrophically.
I qualitatively agree, but maybe quantitatively disagree? Not sure what “a bunch” means in relative terms. I’d say the lion’s share isn’t necessary to solve in human(e)-only futures. You just get so many advantages with human(e) minds vs. alien minds. But yeah, I think figuring out good governance is important for building a hopeworthy future, especially for building one that is legibly so.