By “the thing,” I mean something like developing a first-principles understanding of why you believe AI is dangerous, such that you could reconstruct the argument from scratch without appealing to authority.
It is easy.
We do not have a theory of victory, or a win condition.
The usual (best) answer is “we solve alignment and build a glorious transhumanist future”, without having a formal definition of what “solving alignment” means when it start involving real humans as the thing we want our AI systems to be aligned to (vague gestures towards CEV) or a clear aesthetic vision of what “glorious transhumanist future” means (vague gestures towards end of suffering).
If we have no theory of victory, we’re going to lose : the actual outcome is still a precise thing, and even in the best case where our vague intuition is somewhat somehow met, some process (“random shit go !”) will have to go from our fuzzy, confused desiderata to a precise outcome (that we can’t even foresee or judge because we don’t even know what we want or what it looks like).
Is there holes in this consideration ? Yes. Maybe we could build an ASI-teacher that guides us through that (but also : there is obvious problems to that there). Maybe we could do stuff sufficiently slowly that we could decide (“we” ? how ?) and steer (but see how it’s hard to even pause — steering is another beast entirely) as things advance and go clearer. The main takeaway is still “here be dragons”.
Related but still off-topic : the entire field is advancing with its priority backwards. We’re building ASI before solving alignment ; working on solving alignment before asking what we want collectively ; asking what we want collectively before asking what we want personally. Everyone is trying to run before even learning to walk. Of course we’re all going to fall.
Yeah there’s a few uncomfortable truths hidden in “asking what we want collectively” that mean that that question can’t be answered. Such as different groups wanting mutually exclusive things and who exactly “we” is.
Easy enough, pick a set of moral rules you like the best, and then work towards that AI winning. Who gets to set the tone while such a thing is possible? Amodei, Musk, Altman, Pichai, Xi?
It is easy.
We do not have a theory of victory, or a win condition.
The usual (best) answer is “we solve alignment and build a glorious transhumanist future”, without having a formal definition of what “solving alignment” means when it start involving real humans as the thing we want our AI systems to be aligned to (vague gestures towards CEV) or a clear aesthetic vision of what “glorious transhumanist future” means (vague gestures towards end of suffering).
If we have no theory of victory, we’re going to lose : the actual outcome is still a precise thing, and even in the best case where our vague intuition is somewhat somehow met, some process (“random shit go !”) will have to go from our fuzzy, confused desiderata to a precise outcome (that we can’t even foresee or judge because we don’t even know what we want or what it looks like).
Is there holes in this consideration ? Yes. Maybe we could build an ASI-teacher that guides us through that (but also : there is obvious problems to that there). Maybe we could do stuff sufficiently slowly that we could decide (“we” ? how ?) and steer (but see how it’s hard to even pause — steering is another beast entirely) as things advance and go clearer. The main takeaway is still “here be dragons”.
Related but still off-topic : the entire field is advancing with its priority backwards. We’re building ASI before solving alignment ; working on solving alignment before asking what we want collectively ; asking what we want collectively before asking what we want personally. Everyone is trying to run before even learning to walk. Of course we’re all going to fall.
Yeah there’s a few uncomfortable truths hidden in “asking what we want collectively” that mean that that question can’t be answered. Such as different groups wanting mutually exclusive things and who exactly “we” is.
Easy enough, pick a set of moral rules you like the best, and then work towards that AI winning. Who gets to set the tone while such a thing is possible? Amodei, Musk, Altman, Pichai, Xi?
My current vote is Amodei.