Strong AGI: Artificial intelligence strong enough to build nanotech, while being at least as general as humans (probably more general). This definition doesn’t imply anything about the goals or values of such an AI, but being at least as general as humans does imply that it is an agent that can select actions, and also implies that it is at least as data-efficient as humans.
Humanity survives: At least one person who was alive before the AI was built is still alive 50 years later. Includes both humanity remaining biological and uploading, doesn’t include everyone dying.
Alignment problem: The problem of picking out an AI design that won’t kill everyone from the space of possible designs for Strong AGI.
Aligned by default: Maybe most of the space of possible designs for Strong AGI does in fact consist of AIs that won’t kill everyone. If so, then “pick a design at random” is a sufficient strategy for solving the alignment problem.
An attempt at solving the alignment problem: Some group of people who believe that the alignment problem is hard (i.e. picking a design at random has a very low chance of working) try to solve it. The group doesn’t have to be rationalists, or to call it the “alignment problem” though.
A successful attempt at solving the alignment problem: One of the groups in the above definition do in fact solve the alignment problem, i.e. they find a design for Strong AGI that won’t kill everyone. Important note: If Strong AGI is aligned by default, then no attempts are considered successful, since the problem wasn’t really a problem in the first place.
Thanks. These seem like good definitions. They actually set the bar high for your prediction, which is respectable. I appreciate you taking this seriously.
If you’ll permit just a little bit more pedantic nitpicking, do you mind if I request a precise definition of nanotech? I assume you mean self-replicating nanobots (grey goo) because, technically, we already have nanotech. However, putting the bar at grey goo (potential, of course—the system doesn’t have to actually make it for real) might be setting it above what you intended.
Self replicating nanotech is what I’m referring to, yes. Doesn’t have to be a bacteria-like grey goo sea of nanobots, though. I’d generally expect nanotech to look more like a bunch of nanofactories, computers, energy collectors, and nanomachines to do various other jobs, and some of the nanofactories have the job of producing other nanofactories so that the whole system replicates itself. There wouldn’t be the constraint that there is with bacteria where each cell is in competition with all the others.
What are your definitions for “Strong AGI”, “the alignment problem succeed[s]” and “humanity survives”?
Strong AGI: Artificial intelligence strong enough to build nanotech, while being at least as general as humans (probably more general). This definition doesn’t imply anything about the goals or values of such an AI, but being at least as general as humans does imply that it is an agent that can select actions, and also implies that it is at least as data-efficient as humans.
Humanity survives: At least one person who was alive before the AI was built is still alive 50 years later. Includes both humanity remaining biological and uploading, doesn’t include everyone dying.
Alignment problem: The problem of picking out an AI design that won’t kill everyone from the space of possible designs for Strong AGI.
Aligned by default: Maybe most of the space of possible designs for Strong AGI does in fact consist of AIs that won’t kill everyone. If so, then “pick a design at random” is a sufficient strategy for solving the alignment problem.
An attempt at solving the alignment problem: Some group of people who believe that the alignment problem is hard (i.e. picking a design at random has a very low chance of working) try to solve it. The group doesn’t have to be rationalists, or to call it the “alignment problem” though.
A successful attempt at solving the alignment problem: One of the groups in the above definition do in fact solve the alignment problem, i.e. they find a design for Strong AGI that won’t kill everyone. Important note: If Strong AGI is aligned by default, then no attempts are considered successful, since the problem wasn’t really a problem in the first place.
Thanks. These seem like good definitions. They actually set the bar high for your prediction, which is respectable. I appreciate you taking this seriously.
If you’ll permit just a little bit more pedantic nitpicking, do you mind if I request a precise definition of nanotech? I assume you mean self-replicating nanobots (grey goo) because, technically, we already have nanotech. However, putting the bar at grey goo (potential, of course—the system doesn’t have to actually make it for real) might be setting it above what you intended.
Self replicating nanotech is what I’m referring to, yes. Doesn’t have to be a bacteria-like grey goo sea of nanobots, though. I’d generally expect nanotech to look more like a bunch of nanofactories, computers, energy collectors, and nanomachines to do various other jobs, and some of the nanofactories have the job of producing other nanofactories so that the whole system replicates itself. There wouldn’t be the constraint that there is with bacteria where each cell is in competition with all the others.