With a norm-following AI system, the success story is primarily around accelerating our rate of progress. Humans remain in charge of the overall trajectory of the future, and we use AI systems as tools that enable us to make better decisions and create better technologies, which looks like “superhuman intelligence” from our vantage point today.
If we still want an AI system that colonizes space and optimizes it according to our values without our supervision, we can figure out what our values are over a period of reflection, solve the alignment problem for goal-directed AI systems, and then create such an AI system.
If I had to point towards a particular concrete path to a good future, it would be the one that I outlined in Following human norms. We build AI systems that have a good understanding of “common sense” or “how to behave normally in human society”; they accelerate technological development and improve decision-making; if we really want to have a goal-directed AI that is not under our control but that optimizes for our values then we solve the full alignment problem in the future. Inferring preferences or norms from the world state could be a crucial part of helping our AI systems understand “common sense”.
It’s not the same as your Interim Quality-of-Life Improver, but it’s got similar aspects.
It’s also related to the concept of a “Great Deliberation” where we stabilize the world and then figure out what we want to do. (I don’t have a reference for that though.)
If it wasn’t these (but it was me), it was probably something earlier; I think I was thinking along the lines of Interim Quality-of-Life Improver in early-to-mid 2018.
Thanks for the references. I think I should also credit you with being the first to use “success story” the way I’m using it here, in connection with AI safety, which gave me the idea to write this post.
It’s not the same as your Interim Quality-of-Life Improver, but it’s got similar aspects.
The main difference seems to be that you don’t explicitly mention strong global coordination to stop unaligned AI from arising. Is that something you also had in mind? (I seem to recall someone talking about that in connection with this kind of scenario.)
It’s also related to the concept of a “Great Deliberation” where we stabilize the world and then figure out what we want to do. (I don’t have a reference for that though.)
There’s also Will MacAskill and Toby Ord’s “the Long Reflection” (which may be the same thing that you’re thinking of), which as far as I know isn’t written up in detail anywhere yet. However I’m told that both of their upcoming books will have some discussions of it.
The main difference seems to be that you don’t explicitly mention strong global coordination to stop unaligned AI from arising. Is that something you also had in mind?
It’s more of a free variable—I could imagine the world turning out such that we don’t need very strong coordination (because the Quality of Life Improver AI could plausibly not sacrifice competitiveness), and I could also imagine the world turning out such that it’s really easy to build very powerful unaligned AI and we need strong global coordination to prevent it from happening.
I think the difference may just be in how we present it—you focus more on the global coordination part, whereas I focus more on the following norms + improving technology + quality of life part.
There’s also Will MacAskill and Toby Ord’s “the Long Reflection”
It might be from Following human norms?
Which was referenced again in Learning preferences by looking at the world:
It’s not the same as your Interim Quality-of-Life Improver, but it’s got similar aspects.
It’s also related to the concept of a “Great Deliberation” where we stabilize the world and then figure out what we want to do. (I don’t have a reference for that though.)
If it wasn’t these (but it was me), it was probably something earlier; I think I was thinking along the lines of Interim Quality-of-Life Improver in early-to-mid 2018.
Thanks for the references. I think I should also credit you with being the first to use “success story” the way I’m using it here, in connection with AI safety, which gave me the idea to write this post.
The main difference seems to be that you don’t explicitly mention strong global coordination to stop unaligned AI from arising. Is that something you also had in mind? (I seem to recall someone talking about that in connection with this kind of scenario.)
There’s also Will MacAskill and Toby Ord’s “the Long Reflection” (which may be the same thing that you’re thinking of), which as far as I know isn’t written up in detail anywhere yet. However I’m told that both of their upcoming books will have some discussions of it.
It’s more of a free variable—I could imagine the world turning out such that we don’t need very strong coordination (because the Quality of Life Improver AI could plausibly not sacrifice competitiveness), and I could also imagine the world turning out such that it’s really easy to build very powerful unaligned AI and we need strong global coordination to prevent it from happening.
I think the difference may just be in how we present it—you focus more on the global coordination part, whereas I focus more on the following norms + improving technology + quality of life part.
Yeah I think that’s the same concept.