Nod, I deliberately titled a section There is no safe “muddling through” without perfect safeguards (in an earlier draft I did just say “there is no safe muddling through”, and then was like “okay, that’s false, because, seems totally plausible muddle through into figuring out longerterm safeguards,
(and, in fact, I don’t have a plan to get longterm safeguards that don’t look like some kind of muddling through, in some sense)
I was just chatting with @1a3orn, and he brought up a similar point to the industrial revolution concern, and I totally agree.
Some background assumptions I have here:
you can’t reason your way all the way to “safely navigate the industrial revolution”, yeah. Some notable failures:
inventing communism
trying to invent the cotton gin to make slavery less bad, accidentally produce way more slavery via inducing demand
environmentalism ending up banning nuclear stuff that caused a lot of environmental damage
(there are positive examples too I think, but the existence of these negative examples should put the fear of god in you)
it’s still possible to do nonzero reasoning ahead. You can put constraints on what sort of things possibly make sense to be doing.
early industrial revolution: if you don’t see the first steam train and think “oh shit, everything is gonna change”, man you are going to be pointed in the wrong direction completely
analogously: if you don’t look at the oncoming AI (as well as general economic trends), and think “man, All Possible Views About Humanity’s Future Are Wild”, you’re not pointed in the right direction at all
Part of the point of this post was to lay out “here’s the rough class of thing that seems like it’s gonna happen by default. Seems like either we need to learn new facts, or, we need a process with an extreme amount of power and wisdom, or, we should expect some cluster of bad things to probably happen.
During my chat with 1a3orn, I did notice:
Okay, if I’m trying to solve the ‘death by evolution’ problem (assuming we got nice smooth takeoff still), an alternate plan from “build the machine god” is:
Send human-uploads with some von-neuman probes to every star in the universe, immediately, before we leave The Dreamtime. And then probably there will at least be a lot of subjective experience-timeslices and chances for some of them to figure out how to make good things happen, with (maybe) like a 10 year head start before hollow grabby AI comes after them.
I don’t actually believe in nice slow takeoff or 10 year lead times before Hollow Grabby AI comes after them, but, if I did, that’d at least be a coherent plan.
The problems with that is that are:
a) it’s still leaving a lot of risk of costly war between the human diaspora and the Hollow Grabby AI
b) many of the humans across the universe are probably going to do horrible S-risky mindcrime.
So, I’m not very satisfied with that plan, but I mention it to help broaden the creative range of solutions from “build a CEV god” to include at least one other type of option.
Nod, I deliberately titled a section There is no safe “muddling through” without perfect safeguards (in an earlier draft I did just say “there is no safe muddling through”, and then was like “okay, that’s false, because, seems totally plausible muddle through into figuring out longerterm safeguards,
(and, in fact, I don’t have a plan to get longterm safeguards that don’t look like some kind of muddling through, in some sense)
I was just chatting with @1a3orn, and he brought up a similar point to the industrial revolution concern, and I totally agree.
Some background assumptions I have here:
you can’t reason your way all the way to “safely navigate the industrial revolution”, yeah. Some notable failures:
inventing communism
trying to invent the cotton gin to make slavery less bad, accidentally produce way more slavery via inducing demand
environmentalism ending up banning nuclear stuff that caused a lot of environmental damage
(there are positive examples too I think, but the existence of these negative examples should put the fear of god in you)
it’s still possible to do nonzero reasoning ahead. You can put constraints on what sort of things possibly make sense to be doing.
early industrial revolution: if you don’t see the first steam train and think “oh shit, everything is gonna change”, man you are going to be pointed in the wrong direction completely
analogously: if you don’t look at the oncoming AI (as well as general economic trends), and think “man, All Possible Views About Humanity’s Future Are Wild”, you’re not pointed in the right direction at all
Part of the point of this post was to lay out “here’s the rough class of thing that seems like it’s gonna happen by default. Seems like either we need to learn new facts, or, we need a process with an extreme amount of power and wisdom, or, we should expect some cluster of bad things to probably happen.
During my chat with 1a3orn, I did notice:
Okay, if I’m trying to solve the ‘death by evolution’ problem (assuming we got nice smooth takeoff still), an alternate plan from “build the machine god” is:
Send human-uploads with some von-neuman probes to every star in the universe, immediately, before we leave The Dreamtime. And then probably there will at least be a lot of subjective experience-timeslices and chances for some of them to figure out how to make good things happen, with (maybe) like a 10 year head start before hollow grabby AI comes after them.
I don’t actually believe in nice slow takeoff or 10 year lead times before Hollow Grabby AI comes after them, but, if I did, that’d at least be a coherent plan.
The problems with that is that are:
a) it’s still leaving a lot of risk of costly war between the human diaspora and the Hollow Grabby AI
b) many of the humans across the universe are probably going to do horrible S-risky mindcrime.
So, I’m not very satisfied with that plan, but I mention it to help broaden the creative range of solutions from “build a CEV god” to include at least one other type of option.