I’m mildly skeptical that we can drop consequentialist preferences altogether (I mean, as first-class preferences, I’m not just talking about ‘consequentialist planning as a means-to-an-end for being helpful right now’ and similar). I don’t have any airtight proof, and I really have some uncertainty here, but FWIW I’m partly getting that from a general intuition that the people doing (and figuring out) important and novel things in the world, the kinds of things that would help with the rogue ASI problem, are people really deeply care about simulacrum level 1 rather than 3, and you only get that from having consequentialist preferences as major first-class components of the motivation system.
A different idea is that we can build top-level preferences out of a mix of consequentialist desires and non-consequentialist (e.g. virtue-ethics-y) desires. And yeah, that seems like the obvious “Plan A” to me. The question is whether it would actually work, see §1.2.
I don’t understand how simulacrum levels come into it[1]. Are you taking me to mean ‘virtue signalling’ when I say virtue? No! Adherence to honest and earnest communication, for example, is a commonly recognised virtue.
FWIW I also have never got what is supposedly ordinal about the simulacrum levels beyond 1, the honest one. The other ‘levels’ just look like various orthogonal breeds of fakery, to me. Haven’t scrutinised deeply.
When I think of humans who get difficult things done, or figure difficult things out, they tend to care about accomplishing those things, a lot, and in a direct and explicit way, not just e.g. as a facet of what kind of person they see themselves as. I mean, maybe “what kind of person I see myself as” has something to do with how they originally came to care about those things, but it’s not what they’re explicitly thinking about. They’re thinking directly about the object-level prize at the end of the journey, and how to get that prize.
E.g. plenty of climate change activists think of climate change activism as a good and virtuous thing to do, but I think the subset of climate change activists who are really moving the needle are the ones who are directly thinking about climate change being directly bad, and really want it to stop, and are focused directly on how to make that happen.
E.g. plenty of mathematicians think of math as a good and praiseworthy activity, but I think that the person who will solve the Riemann hypothesis will be a person who is (in addition to being smart etc.) really damn curious about why the Riemann hypothesis is true, and focused directly on figuring that out. Or they’re really damn eager to become famous by solving the Riemann hypothesis, or whatever else.
It seems to me that this is a general pattern—i.e., we need direct-consequentialism not just consequentialism-incidentally-arising-from-virtue to accomplish difficult novel tasks—and my hunch is that this pattern generalizes to brain-like AGI. If so, then we will face the problem of balancing consequentialist direct top-level goals with non-consequentialist direct top-level goals, rather than merely facing the (probably easier) problem of avoiding the former altogether.
FWIW I also have never got what is supposedly ordinal about the simulacrum levels beyond 1, the honest one. The other ‘levels’ just look like various orthogonal breeds of fakery, to me. Haven’t scrutinised deeply.
(Off-topic but fun) I think it’s at least somewhat ordinal, e.g. Zvi’s “Level 1: Symbols describe reality. Level 2: Symbols pretend to describe reality. Level 3: Symbols pretend to pretend to describe reality. Level 4: Symbols need not pretend to describe reality.”
(Agree fun) These have looked post-hoc and overfitted to me, and the variety of apparent but incompatible explanations for the supposed ordering is a smell from my pov. Maybe I should look closer at some point.
I’m mildly skeptical that we can drop consequentialist preferences altogether (I mean, as first-class preferences, I’m not just talking about ‘consequentialist planning as a means-to-an-end for being helpful right now’ and similar). I don’t have any airtight proof, and I really have some uncertainty here, but FWIW I’m partly getting that from a general intuition that the people doing (and figuring out) important and novel things in the world, the kinds of things that would help with the rogue ASI problem, are people really deeply care about simulacrum level 1 rather than 3, and you only get that from having consequentialist preferences as major first-class components of the motivation system.
A different idea is that we can build top-level preferences out of a mix of consequentialist desires and non-consequentialist (e.g. virtue-ethics-y) desires. And yeah, that seems like the obvious “Plan A” to me. The question is whether it would actually work, see §1.2.
I don’t understand how simulacrum levels come into it [1] . Are you taking me to mean ‘virtue signalling’ when I say virtue? No! Adherence to honest and earnest communication, for example, is a commonly recognised virtue.
FWIW I also have never got what is supposedly ordinal about the simulacrum levels beyond 1, the honest one. The other ‘levels’ just look like various orthogonal breeds of fakery, to me. Haven’t scrutinised deeply.
Oh, hmm, good point, thanks. Let me try again:
When I think of humans who get difficult things done, or figure difficult things out, they tend to care about accomplishing those things, a lot, and in a direct and explicit way, not just e.g. as a facet of what kind of person they see themselves as. I mean, maybe “what kind of person I see myself as” has something to do with how they originally came to care about those things, but it’s not what they’re explicitly thinking about. They’re thinking directly about the object-level prize at the end of the journey, and how to get that prize.
E.g. plenty of climate change activists think of climate change activism as a good and virtuous thing to do, but I think the subset of climate change activists who are really moving the needle are the ones who are directly thinking about climate change being directly bad, and really want it to stop, and are focused directly on how to make that happen.
E.g. plenty of mathematicians think of math as a good and praiseworthy activity, but I think that the person who will solve the Riemann hypothesis will be a person who is (in addition to being smart etc.) really damn curious about why the Riemann hypothesis is true, and focused directly on figuring that out. Or they’re really damn eager to become famous by solving the Riemann hypothesis, or whatever else.
It seems to me that this is a general pattern—i.e., we need direct-consequentialism not just consequentialism-incidentally-arising-from-virtue to accomplish difficult novel tasks—and my hunch is that this pattern generalizes to brain-like AGI. If so, then we will face the problem of balancing consequentialist direct top-level goals with non-consequentialist direct top-level goals, rather than merely facing the (probably easier) problem of avoiding the former altogether.
(This is all a lightly-held opinion.)
(Off-topic but fun) I think it’s at least somewhat ordinal, e.g. Zvi’s “Level 1: Symbols describe reality. Level 2: Symbols pretend to describe reality. Level 3: Symbols pretend to pretend to describe reality. Level 4: Symbols need not pretend to describe reality.”
See also Thane’s attempt.
(Agree fun) These have looked post-hoc and overfitted to me, and the variety of apparent but incompatible explanations for the supposed ordering is a smell from my pov. Maybe I should look closer at some point.