First, you wrote “a part of me is actually more scared of many futures in which alignment is solved, than a future where biological life is simply wiped out by a paperclip maximizer.” So, I tried to assuage this fear for a particular class of alignment solutions.
Second… Yes, for a utilitarian this doesn’t mean “much”. But, tbh, who cares? I am not a utilitarian. The vast majority of people are not utilitarians. Maybe even literally no one is an (honest, not self-deceiving) utilitarian. From my perspective, disappointing the imaginary utilitarian is (in itself) about as upsetting as disappointing the imaginary paperclip maximizer.
Third, what I actually want from multi-user alignment is a solution that (i) is acceptable to me personally (ii) is acceptable to the vast majority of people (at least if they think through it rationally and are arguing honestly and in good faith) (iii) is acceptable to key stakeholders (iv) as much as possible, doesn’t leave any Pareto improvements on the table and (v) sufficiently Schelling-pointy to coordinate around. Here, “acceptable” means “a lot better than paperclips and not worth starting an AI race/war to get something better”.
Second… Yes, for a utilitarian this doesn’t mean “much”. But, tbh, who cares? I am not a utilitarian. The vast majority of people are not utilitarians. Maybe even literally no one is an (honest, not self-deceiving) utilitarian. From my perspective, disappointing the imaginary utilitarian is (in itself) about as upsetting as disappointing the imaginary paperclip maximizer.
I’m not a utilitarian either, because I don’t know what my values are or should be. But I do assign significant credence to the possibility that something in the vincinity of utilitarianism is the right values (for me, or period). Given my uncertainties, I want to arrange the current state of the world so that (to the extent possible), whatever I end up deciding my values are, through things like reason, deliberation, doing philosophy, the world will ultimately not turn out to be a huge disappointment according to those values. Unfortunately, your proposed solution isn’t very reassuring to this kind of view.
It’s quite possible that I (and people like me) are simply out of luck, and there’s just no feasible way to do what we want to do, but it sounds like you think I shouldn’t even want what I want, or at least that you don’t want something like this. Is it because you’re already pretty sure what your values are or should be, and therefore think there’s little chance that millennia from now you’ll end up deciding that utilitarianism (or NU, or whatever) is right after all, and regret not doing more in 2021 to push the world in the direction of [your real values, whatever they are]?
I’m moderately sure what my values are, to some approximation. More importantly, I’m even more sure that, whatever my values are, they are not so extremely different from the values of most people that I should wage some kind of war against the majority instead of trying to arrive at a reasonable compromise. And, in the unlikely event that most people (including me) will turn out to be some kind of utilitarians after all, it’s not a problem: value aggregation will then produce a universe which is pretty good for utilitarians.
I’m moderately sure what my values are, to some approximation. More importantly, I’m even more sure that, whatever my values are, they are not so extremely different from the values of most people [...]
Maybe you’re just not part of the target audience of my OP then… but from my perspective, if I determine my values through the kind of process described in the first quote, and most people determine their values through the kind of process described in the second quote, it seems quite likely that the values end up being very different.
[...] that I should wage some kind of war against the majority instead of trying to arrive at a reasonable compromise.
The kind of solution I have in mind is not “waging war” but for example, solving metaphilososphy and building an AI that can encourage philosophical reflection in humans or enhance people’s philosophical abilities.
And, in the unlikely possibility that most people (including me) will turn out to be some kind of utilitarians after all, it’s not a problem: value aggregation will then produce a universe which is pretty good for utilitarians.
What if you turn out to be some kind of utilitarian but most people don’t (because you’re more like the first group in the OP and they’re more like the second group), or most people will eventually turn out to be some kind of utilitarian in a world without AI, but in a world with AI, this will happen?
I don’t think people determine their values through either process. I think that they already have values, which are to a large extent genetic and immutable. Instead, these processes determine what values they pretend to have for game-theory reasons. So, the big difference between the groups is which “cards” they hold and/or what strategy they pursue, not an intrinsic difference in values.
But also, if we do model values as the result of some long process of reflection, and you’re worried about the AI disrupting or insufficiently aiding this process, then this is already a single-user alignment issue and should be analyzed in that context first. The presumed differences in moralities are not the main source of the problem here.
I don’t think people determine their values through either process. I think that they already have values, which are to a large extent genetic and immutable. Instead, these processes determine what values they pretend to have for game-theory reasons. So, the big difference between the groups is which “cards” they hold and/or what strategy they pursue, not an intrinsic difference in values.
This is not a theory that’s familiar to me. Why do you think this is true? Have you written more about it somewhere or can link to a more complete explanation?
But also, if we do model values as the result of some long process of reflection, and you’re worried about the AI disrupting or insufficiently aiding this process, then this is already a single-user alignment issue and should be analyzed in that context first. The presumed differences in moralities are not the main source of the problem here.
This seems reasonable to me. (If this was meant to be an argument against something I said, there may have been anther miscommuncation, but I’m not sure it’s worth tracking that down.)
This is not a theory that’s familiar to me. Why do you think this is true? Have you written more about it somewhere or can link to a more complete explanation?
I considering writing about this for a while, but so far I don’t feel sufficiently motivated. So, the links I posted upwards in the thread are the best I have, plus vague gesturing in the directions of Hansonian signaling theories, Jaynes’ theory of consciousness and Yudkowsky’s belief in belief.
First, you wrote “a part of me is actually more scared of many futures in which alignment is solved, than a future where biological life is simply wiped out by a paperclip maximizer.” So, I tried to assuage this fear for a particular class of alignment solutions.
Second… Yes, for a utilitarian this doesn’t mean “much”. But, tbh, who cares? I am not a utilitarian. The vast majority of people are not utilitarians. Maybe even literally no one is an (honest, not self-deceiving) utilitarian. From my perspective, disappointing the imaginary utilitarian is (in itself) about as upsetting as disappointing the imaginary paperclip maximizer.
Third, what I actually want from multi-user alignment is a solution that (i) is acceptable to me personally (ii) is acceptable to the vast majority of people (at least if they think through it rationally and are arguing honestly and in good faith) (iii) is acceptable to key stakeholders (iv) as much as possible, doesn’t leave any Pareto improvements on the table and (v) sufficiently Schelling-pointy to coordinate around. Here, “acceptable” means “a lot better than paperclips and not worth starting an AI race/war to get something better”.
I’m not a utilitarian either, because I don’t know what my values are or should be. But I do assign significant credence to the possibility that something in the vincinity of utilitarianism is the right values (for me, or period). Given my uncertainties, I want to arrange the current state of the world so that (to the extent possible), whatever I end up deciding my values are, through things like reason, deliberation, doing philosophy, the world will ultimately not turn out to be a huge disappointment according to those values. Unfortunately, your proposed solution isn’t very reassuring to this kind of view.
It’s quite possible that I (and people like me) are simply out of luck, and there’s just no feasible way to do what we want to do, but it sounds like you think I shouldn’t even want what I want, or at least that you don’t want something like this. Is it because you’re already pretty sure what your values are or should be, and therefore think there’s little chance that millennia from now you’ll end up deciding that utilitarianism (or NU, or whatever) is right after all, and regret not doing more in 2021 to push the world in the direction of [your real values, whatever they are]?
I’m moderately sure what my values are, to some approximation. More importantly, I’m even more sure that, whatever my values are, they are not so extremely different from the values of most people that I should wage some kind of war against the majority instead of trying to arrive at a reasonable compromise. And, in the unlikely event that most people (including me) will turn out to be some kind of utilitarians after all, it’s not a problem: value aggregation will then produce a universe which is pretty good for utilitarians.
Maybe you’re just not part of the target audience of my OP then… but from my perspective, if I determine my values through the kind of process described in the first quote, and most people determine their values through the kind of process described in the second quote, it seems quite likely that the values end up being very different.
The kind of solution I have in mind is not “waging war” but for example, solving metaphilososphy and building an AI that can encourage philosophical reflection in humans or enhance people’s philosophical abilities.
What if you turn out to be some kind of utilitarian but most people don’t (because you’re more like the first group in the OP and they’re more like the second group), or most people will eventually turn out to be some kind of utilitarian in a world without AI, but in a world with AI, this will happen?
I don’t think people determine their values through either process. I think that they already have values, which are to a large extent genetic and immutable. Instead, these processes determine what values they pretend to have for game-theory reasons. So, the big difference between the groups is which “cards” they hold and/or what strategy they pursue, not an intrinsic difference in values.
But also, if we do model values as the result of some long process of reflection, and you’re worried about the AI disrupting or insufficiently aiding this process, then this is already a single-user alignment issue and should be analyzed in that context first. The presumed differences in moralities are not the main source of the problem here.
This is not a theory that’s familiar to me. Why do you think this is true? Have you written more about it somewhere or can link to a more complete explanation?
This seems reasonable to me. (If this was meant to be an argument against something I said, there may have been anther miscommuncation, but I’m not sure it’s worth tracking that down.)
I considering writing about this for a while, but so far I don’t feel sufficiently motivated. So, the links I posted upwards in the thread are the best I have, plus vague gesturing in the directions of Hansonian signaling theories, Jaynes’ theory of consciousness and Yudkowsky’s belief in belief.
Isn’t this the main thesis of “The righteous mind”?