Are you suggesting that I should base my morality on whether I’ll be rewarded for adhering to it? That just sounds like selfishness disguised as impersonal ethics.
To be clear, I do have some selfish/non-impartial preferences. I care about my own life and happiness, and the happiness of my friends and family. But I also have some altruistic preferences, and my commentary on AI tends to reflect that.
A moral code is invented[1] by a group of people to benefit the group as a whole, it sometimes demands sacrifice from individuals, but a good one usually has the quality that at some point in a person’s past, they would have voluntarily signed on with it. Redistribution is a good example. If you have a concave utility function, and if you don’t know where you’ll end up in life, you should be willing to sign a pledge to later share your resources with less fortunate people who’ve also signed the pledge, just in case you become one of the less fortunate. The downside of not being covered in that case is much larger than the upside of not having to share in the other case. For convenience, we could decide to make the pledge mandatory and the coverage universal (ie, taxes and welfare) since there aren’t a lot of humans who would decline that deal in good faith. (Perhaps some humans are genuinely convex egoists and wouldn’t sign that deal, but we outnumber them, and accomodating them would be inconvenient, so we ignore them.) If we’re pure of heart, we could make the pledge acausal and implicit and adhere to it without any enforcement mechanisms, and I think that’s what morality usually is or should be in the common sense.
But anyway, it sometimes seems to me that you often advocate a morality regarding AI relations that doesn’t benefit anyone who currently exists, or, the coalition that you are a part of. This seems like a mistake. Or worse.
I wonder if it comes from a place of concern that… if we had public consensus that humans would prefer to retain full control over the lightcone, then we’d end up having stupid and unnecessary conflicts with the AIs over that, while, if we pretend we’re perfectly happy to share, relations will be better? You may feel that as long as we survive and get a piece, it’s not worth fighting for a larger piece? The damages from war would be so bad for both sides that we’d prefer to just give them most of the lightcone now?
And I think stupid wars aren’t possible under ASI-level information technology. If we had the capacity to share information and find out who’d win a war and skip to a surrender deal, doing so always has higher EV for both sides than actually fighting. The reason wars are not skipped that way today is that we still lack the capacity to simultaneously mutually exchange proofs of force capacity, but we’re getting closer to having that every day. Generally, in that era, coexisting under confessed value differences will be pretty easy. Honestly I feel like it already ought to be easy, for humans, if we’d get serious about it.
Though, as Singer says, much of morality is invented only in the same sense as mathematics is invented, being so non-arbitrary that it seems to have a kind of external observer-independent existence and fairly universal truths, which powerful AIs are likely to also discover. But the moralities in that class are much weaker (I don’t think Singer fully recognises the extent of this), and I don’t believe they have anything to say about this issue.
But anyway, it sometimes seems to me that you often advocate a morality regarding AI relations that doesn’t benefit anyone who currently exists, or, the coalition that you are a part of. This seems like a mistake. Or worse.
I dispute this, since I’ve argued for the practical benefits of giving AIs legal autonomy, which I think would likely benefit existing humans. Relatedly, I’ve also talked about how I think hastening the arrival AI could benefit people who currently exist. Indeed, that’s one of the best arguments for accelerating AI. The argument is that, by ensuring AI arrives sooner, we can accelerate the pace of medical progress, among other useful technologies. This could ensure that currently-existing old people who would otherwise die without AI will be saved and live a longer and healthier life than the alternative.
(Of course, this must be weighed against concerns about AI safety. I am not claiming that there is no tradeoff between AI safety and acceleration. Rather, my point is that, despite the risks, accelerating AI could still be the preferable choice.)
However, I do think there is an important distinction here to make between the following groups:
The set of all existing humans
The human species itself, including all potential genetic descendants of existing humans
Insofar as I have loyalty towards a group, I have much more loyalty towards (1) than (2). It’s possible you think that I should see myself as belonging to the coalition comprised of (2) rather than (1), but I don’t see a strong argument for that position.
To the extent it makes sense to think of morality as arising from game theoretic considerations, there doesn’t appear to be much advantage for me in identifying with the coalition of all potential human descendants (group 2) rather than with the coalition of currently existing humans plus potential future AIs (group 1 + AIs) . If we are willing to extend our coalition to include potential future beings, then I would seem to have even stronger practical reasons to align myself with a coalition that includes future AI systems. This is because future AIs will likely be far more powerful than any potential biological human descendants.
I want to clarify, however, that I don’t tend to think of morality as arising from game theoretic considerations. Rather, I mostly think of morality as simply an expression of my personal preferences about the world.
Are you suggesting that I should base my morality on whether I’ll be rewarded for adhering to it? That just sounds like selfishness disguised as impersonal ethics.
To be clear, I do have some selfish/non-impartial preferences. I care about my own life and happiness, and the happiness of my friends and family. But I also have some altruistic preferences, and my commentary on AI tends to reflect that.
A moral code is invented[1] by a group of people to benefit the group as a whole, it sometimes demands sacrifice from individuals, but a good one usually has the quality that at some point in a person’s past, they would have voluntarily signed on with it. Redistribution is a good example. If you have a concave utility function, and if you don’t know where you’ll end up in life, you should be willing to sign a pledge to later share your resources with less fortunate people who’ve also signed the pledge, just in case you become one of the less fortunate. The downside of not being covered in that case is much larger than the upside of not having to share in the other case.
For convenience, we could decide to make the pledge mandatory and the coverage universal (ie, taxes and welfare) since there aren’t a lot of humans who would decline that deal in good faith. (Perhaps some humans are genuinely convex egoists and wouldn’t sign that deal, but we outnumber them, and accomodating them would be inconvenient, so we ignore them.)
If we’re pure of heart, we could make the pledge acausal and implicit and adhere to it without any enforcement mechanisms, and I think that’s what morality usually is or should be in the common sense.
But anyway, it sometimes seems to me that you often advocate a morality regarding AI relations that doesn’t benefit anyone who currently exists, or, the coalition that you are a part of. This seems like a mistake. Or worse.
I wonder if it comes from a place of concern that… if we had public consensus that humans would prefer to retain full control over the lightcone, then we’d end up having stupid and unnecessary conflicts with the AIs over that, while, if we pretend we’re perfectly happy to share, relations will be better? You may feel that as long as we survive and get a piece, it’s not worth fighting for a larger piece? The damages from war would be so bad for both sides that we’d prefer to just give them most of the lightcone now?
And I think stupid wars aren’t possible under ASI-level information technology. If we had the capacity to share information and find out who’d win a war and skip to a surrender deal, doing so always has higher EV for both sides than actually fighting. The reason wars are not skipped that way today is that we still lack the capacity to simultaneously mutually exchange proofs of force capacity, but we’re getting closer to having that every day. Generally, in that era, coexisting under confessed value differences will be pretty easy. Honestly I feel like it already ought to be easy, for humans, if we’d get serious about it.
Though, as Singer says, much of morality is invented only in the same sense as mathematics is invented, being so non-arbitrary that it seems to have a kind of external observer-independent existence and fairly universal truths, which powerful AIs are likely to also discover. But the moralities in that class are much weaker (I don’t think Singer fully recognises the extent of this), and I don’t believe they have anything to say about this issue.
I dispute this, since I’ve argued for the practical benefits of giving AIs legal autonomy, which I think would likely benefit existing humans. Relatedly, I’ve also talked about how I think hastening the arrival AI could benefit people who currently exist. Indeed, that’s one of the best arguments for accelerating AI. The argument is that, by ensuring AI arrives sooner, we can accelerate the pace of medical progress, among other useful technologies. This could ensure that currently-existing old people who would otherwise die without AI will be saved and live a longer and healthier life than the alternative.
(Of course, this must be weighed against concerns about AI safety. I am not claiming that there is no tradeoff between AI safety and acceleration. Rather, my point is that, despite the risks, accelerating AI could still be the preferable choice.)
However, I do think there is an important distinction here to make between the following groups:
The set of all existing humans
The human species itself, including all potential genetic descendants of existing humans
Insofar as I have loyalty towards a group, I have much more loyalty towards (1) than (2). It’s possible you think that I should see myself as belonging to the coalition comprised of (2) rather than (1), but I don’t see a strong argument for that position.
To the extent it makes sense to think of morality as arising from game theoretic considerations, there doesn’t appear to be much advantage for me in identifying with the coalition of all potential human descendants (group 2) rather than with the coalition of currently existing humans plus potential future AIs (group 1 + AIs) . If we are willing to extend our coalition to include potential future beings, then I would seem to have even stronger practical reasons to align myself with a coalition that includes future AI systems. This is because future AIs will likely be far more powerful than any potential biological human descendants.
I want to clarify, however, that I don’t tend to think of morality as arising from game theoretic considerations. Rather, I mostly think of morality as simply an expression of my personal preferences about the world.