Unfortunately, there exist unkind people. We can select for people with eg long prosocial careers, high introspection, positive interviews with close friends / relatives, cognitomotor symptoms of empathy etc
I think a central property you don’t mention explicitly is [having kept promises in the past, especially in cases where these required doing difficult costly things and where the person thought they would not be rewarded in the future for having kept the promise]; also, honesty. I’d guess that an important class of potential successes with this kind of scheme, in fact maybe most of the successes (but idk), involve the fooming mind [keeping a promise]/[maintaining a commitment]. I think that maintaining some kind of kindness without a specific commitment to helping existing humans in some way can easily “misgeneralize” to eg some sort of utilitarianism, and nearly every kind of utilitarianism endorses the atoms and negentropy of all existing people being used for something else, or just more generally misgeneralize to caring about new people you can create and various other possible beings and activities over existing people.
also copying a note i wrote for myself on this topic:
”
some ideas for safe self-improvement
Most people currently thinking about AI alignment seem to hope that there is some sort of “formula” for safely/[value/character-preservingly]/whatever becoming more capable (and for alignment more broadly). I doubt there is some such formula to be found. Instead, I think that as one becomes more capable, one should keep thinking carefully about how to become more capable, and that there isn’t some “formula” for how to do this thinking. I think there is very much to be understood about how to become more capable “safely”. This note presents some basic ideas for that.
Why care about how to self-improve safely?
Here are some more concrete reasons to be interested in ideas for safe self-development:
You might hope to somehow make an AI at human intelligence that kinda “properly cares about humanity” for its intelligence level, to let it self-improve until it can take over the world, and to have it “preserve” this property of properly caring about humanity throughout this self-improvement well enough that it then ends the present period of (imo) high existential risk from AI. We could imagine the ideas here being developed into an initial guidebook for such an AI. This would especially matter for the period during which the self-improving AI is still kinda dumb so it couldn’t yet write a better guidebook itself. Once it becomes kind of smart, one might hope that it would do something like continually improving this guidebook as it proceeds. I think this “plan” is crazy and shouldn’t be attempted, but many people seem to think that it would be just fine to let 2025 Claude foom or whatever, so maybe you dear reader are interested in this.
mind upload case. both the step from human to mind upload and also later steps
humanity fooming together
Some ideas and questions for self-improving safely
maybe most importantly, you have to actually think about whether some idea for self-improving is fine. you shouldn’t just be doing stuff carelessly. you have to think about how to do this thinking. you will already be doing this by default when you’re doing “object-level” thinking, but i wanted to make this explicit. like, you have to be developing theory/understanding that helps you with these questions. you have to be writing ever better versions of something like this list for yourself
for example, as a mind upload, before making a bunch of clones of yourself and sharing power with them, you should try to think through analogous things you are familiar with to understand the sorts of issues you might run into (eg what are the issues with democracies? how is a society disanalogous to an individual in general)
making a new mind from scratch is an extremely extremely scary/stupid way to “self-improve”. (haha @ humanity.) you should probably basically only be doing stuff that looks much more like becoming smarter yourself — new versions should mostly have the same structure(s) as previous versions. like, you should do stuff much more like adding or switching out small parts one by one
you try self-modifications. you leave a previous version of yourself behind to analyze post-modification you, that is supposed to be able to roll back changes. i mean this might just look like terminating the version with a certain change
this works much better if you remain “honest/transparent about what you’re like to the evaluator” along the branches considered. so you should try to do that
my guess is that one should specify some initial such structure but this needs to be arbitrarily editable by some version of the guy, because otherwise it will eventually be a stupid broken formality
there could be monitors of various levels doing various things. eg you can have a more ancient version communicating its judgments to a more intermediate version who still has the ability to reroll its future
you can implement a voting system
maybe one could try to make you believe in reward/punishment after all is said and done. like an islamic suicide bomber?
basically, the mind will kinda have to believe a falsehood
but i mean this seems possible in humans. individual humans do great science and philosophy and still believe this often. humanity “believed” this for a long time
i mean maybe there’s some way to make it kinda-not-a-falsehood?
how do you maintain a belief in god over very much thinking / capability gain (and the thing being basically false)
could look at the literature on this
how do you stay committed to your partner? i mean: how do you stay in love, how do you stay friends, how do you keep wanting to do lots of stuff together?
could look at the literature on this or just say some obvious stuff here. and then see to what extent that generalizes to the various cases of interest
sth making it a good example is that it involves a lot of growth
the following is an important difference though: in this case you’re growing alone, not with what you’re supposed to stay committed to
another important difference: you will need to keep caring about entities you couldn’t really work with, couldn’t really be intellectual partners with
I’d guess that an important class of potential successes with this kind of scheme, in fact maybe most of the successes (but idk), involve the fooming mind [keeping a promise]/[maintaining a commitment]. I think that maintaining some kind of kindness without a specific commitment to helping existing humans in some way can easily “misgeneralize” to eg some sort of utilitarianism
Good point, yeah. I’m still confident in the overall machinery of “better understanding of one’s cognition and tooling to self-modify commensurately” → stability; but I really don’t have a principled way to select for this. I’m pretty confident Eliezer has demonstrated himself committed in this sense (see “genre savviness”), but I don’t know anyone else who would be a good starting point.
and nearly every kind of utilitarianism endorses the atoms and negentropy of all existing people being used for something else, or just more generally misgeneralize to caring about new people you can create and various other possible beings and activities over existing people.
Locally valid but connotationally wrong when read through; like, yes, we definitely lose a huge chunk of humanity-CEV in this scenario (which is what actually matters unless atemporal trade with our Everett branches can remedy the holes), but I’d expect a “kindess-foomed” entity to probably not kill people to repurpose their atoms for other entities. A hedonium-foom would, sure, but killing isn’t particularly kind to most people.
Most people currently thinking about AI alignment seem to hope that there is some sort of “formula” for safely/[value/character-preservingly]/whatever becoming more capable (and for alignment more broadly). I doubt there is some such formula to be found. Instead, I think that as one becomes more capable, one should keep thinking carefully about how to become more capable, and that there isn’t some “formula” for how to do this thinking.
A priori, I’m about 95% confident that there’s some coherent and robust math for Vingean reflection which we have yet to invent. But our chances of cracking it before ASI / human superintelligence HSI are quite thin, like maybe 10%, on my modal model.
many people seem to think that it would be just fine to let 2025 Claude foom
Claude 3.5 or any other LLM have vastly worse cognitive attractor dynamics under self-modification than humans do given a commercially-induced RSI capability. I have a draft story about this sort of thing; but basically, the internals range from maybe-aligned-but-horribly-incapable to unaligned-and-incapable to unaligned-and-capable-of-RSI; nowhere along that Pareto frontier do we see something as stable (wrt raw-utility-as-would-be-galactically-amortized post-foom) as a mildly above-average human.
I can however imagine models two generations from now, were they aligned like Opus 3.5, being sufficiently stable in the comparably more narrow action domain of doing a pivotal act to actually bump themselves another 2 generations’ worth of capacity and just execute a pivotal act. But I really don’t think we’ll get Claude Legolas 8.6 aligned like that (P < 0.03).
how do you maintain a belief in god over very much thinking / capability gain (and the thing being basically false)
Might be useful, but this conflates instrumental epistemics with a normative/value thing (which you acknowledge indirectly). This gap widens under intelligence augmentation, but on my model, values become more stable.
I think a central property you don’t mention explicitly is [having kept promises in the past, especially in cases where these required doing difficult costly things and where the person thought they would not be rewarded in the future for having kept the promise]; also, honesty. I’d guess that an important class of potential successes with this kind of scheme, in fact maybe most of the successes (but idk), involve the fooming mind [keeping a promise]/[maintaining a commitment]. I think that maintaining some kind of kindness without a specific commitment to helping existing humans in some way can easily “misgeneralize” to eg some sort of utilitarianism, and nearly every kind of utilitarianism endorses the atoms and negentropy of all existing people being used for something else, or just more generally misgeneralize to caring about new people you can create and various other possible beings and activities over existing people.
also copying a note i wrote for myself on this topic: ”
some ideas for safe self-improvement
Most people currently thinking about AI alignment seem to hope that there is some sort of “formula” for safely/[value/character-preservingly]/whatever becoming more capable (and for alignment more broadly). I doubt there is some such formula to be found. Instead, I think that as one becomes more capable, one should keep thinking carefully about how to become more capable, and that there isn’t some “formula” for how to do this thinking. I think there is very much to be understood about how to become more capable “safely”. This note presents some basic ideas for that.
Why care about how to self-improve safely?
Here are some more concrete reasons to be interested in ideas for safe self-development:
You might hope to somehow make an AI at
mind upload case. both the step from human to mind upload and also later steps
humanity fooming together
Some ideas and questions for self-improving safely
maybe most importantly, you have to actually think about whether some idea for self-improving is fine. you shouldn’t just be doing stuff carelessly. you have to think about how to do this thinking. you will already be doing this by default when you’re doing “object-level” thinking, but i wanted to make this explicit. like, you have to be developing theory/understanding that helps you with these questions. you have to be writing ever better versions of something like this list for yourself
for example, as a mind upload, before making a bunch of clones of yourself and sharing power with them, you should try to think through analogous things you are familiar with to understand the sorts of issues you might run into (eg what are the issues with democracies? how is a society disanalogous to an individual in general)
making a new mind from scratch is an extremely extremely scary/stupid way to “self-improve”. (haha @ humanity.) you should probably basically only be doing stuff that looks much more like becoming smarter yourself — new versions should mostly have the same structure(s) as previous versions. like, you should do stuff much more like adding or switching out small parts one by one
you try self-modifications. you leave a previous version of yourself behind to analyze post-modification you, that is supposed to be able to roll back changes. i mean this might just look like terminating the version with a certain change
this works much better if you remain “honest/transparent about what you’re like to the evaluator” along the branches considered. so you should try to do that
my guess is that one should specify some initial such structure but this needs to be arbitrarily editable by some version of the guy, because otherwise it will eventually be a stupid broken formality
there could be monitors of various levels doing various things. eg you can have a more ancient version communicating its judgments to a more intermediate version who still has the ability to reroll its future
you can implement a voting system
maybe one could try to make you believe in reward/punishment after all is said and done. like an islamic suicide bomber?
basically, the mind will kinda have to believe a falsehood
but i mean this seems possible in humans. individual humans do great science and philosophy and still believe this often. humanity “believed” this for a long time
i mean maybe there’s some way to make it kinda-not-a-falsehood?
how do you maintain a belief in god over very much thinking / capability gain (and the thing being basically false)
could look at the literature on this
how do you stay committed to your partner? i mean: how do you stay in love, how do you stay friends, how do you keep wanting to do lots of stuff together?
could look at the literature on this or just say some obvious stuff here. and then see to what extent that generalizes to the various cases of interest
sth making it a good example is that it involves a lot of growth
the following is an important difference though: in this case you’re growing alone, not with what you’re supposed to stay committed to
another important difference: you will need to keep caring about entities you couldn’t really work with, couldn’t really be intellectual partners with
how does a state stay democratic?
godspeed friend ”
Good point, yeah. I’m still confident in the overall machinery of “better understanding of one’s cognition and tooling to self-modify commensurately” → stability; but I really don’t have a principled way to select for this. I’m pretty confident Eliezer has demonstrated himself committed in this sense (see “genre savviness”), but I don’t know anyone else who would be a good starting point.
Locally valid but connotationally wrong when read through; like, yes, we definitely lose a huge chunk of humanity-CEV in this scenario (which is what actually matters unless atemporal trade with our Everett branches can remedy the holes), but I’d expect a “kindess-foomed” entity to probably not kill people to repurpose their atoms for other entities. A hedonium-foom would, sure, but killing isn’t particularly kind to most people.
A priori, I’m about 95% confident that there’s some coherent and robust math for Vingean reflection which we have yet to invent. But our chances of cracking it before ASI / human superintelligence HSI are quite thin, like maybe 10%, on my modal model.
Claude 3.5 or any other LLM have vastly worse cognitive attractor dynamics under self-modification than humans do given a commercially-induced RSI capability. I have a draft story about this sort of thing; but basically, the internals range from maybe-aligned-but-horribly-incapable to unaligned-and-incapable to unaligned-and-capable-of-RSI; nowhere along that Pareto frontier do we see something as stable (wrt raw-utility-as-would-be-galactically-amortized post-foom) as a mildly above-average human.
I can however imagine models two generations from now, were they aligned like Opus 3.5, being sufficiently stable in the comparably more narrow action domain of doing a pivotal act to actually bump themselves another 2 generations’ worth of capacity and just execute a pivotal act. But I really don’t think we’ll get Claude Legolas 8.6 aligned like that (P < 0.03).
Might be useful, but this conflates instrumental epistemics with a normative/value thing (which you acknowledge indirectly). This gap widens under intelligence augmentation, but on my model, values become more stable.