So I take it you mean “mean answer” in a minimally sympathetic sense of the word mean, not the statistical sense, and “calculation problems” in the same way that the existence of capitalist Hong Kong enclave was useful to socialist China as a means of determining what prices to set, of a system performing a calculation too complex for even ASI to effectively simulate? I agree that a civilization of biological beings has an extremely computational complexity if you are trying to reproduce every tiny quick of its complex non-linear behavior — I’m less clear of why that level of detail would be important to the ASI.
While I do think it’s possible for AI to avoid, as you say, active interference, I think avoiding passive interference and doing the counterfactual of what a society enabled by AI services and advances would be like if there was no AI in it may simply not be a well-defined question, so I think we’re agreed that passive interference is probably unavoidable. So I agree with that distinction you’re making.
I note you say plural “civilizations” — I have been assuming a single Earth-or-Solar-System spanning biological-and-AI civilization, until we get to interstellar colonization (if that is feasible): was your mention of multiple civilizations also about something interstellar, or are you assuming a future that is possibly more federated and diverse than I am?
You don’t say so explicitly, but words like “brutal” and “hypocrisy” are suggesting to me that you’re assuming that the AI alignment problem wasn’t in fact fully solved, or at least wasn’t solved in a way that we would regard as well-solved — could you expand on this? Suppose the humans are considering whether-or-not to engineer a particular society-wide change to their values, and explicitly ask the AI “Please help us make this important decision well.” At this point is your view that AI complying with that request is no longer active interference? Or that doing more than a minimal effort would be active interference? Or that AI would say “I’m afraid we can’t help with things like that” and refuse to actively interfere? I’m having trouble seeing anything less that a best effort assistance (that didn’t even have to be asked for) as compatible with the “AI alignment has been solved” assumption, but perhaps we have rather different ideas of what the goal of AI alignment is. Mine is that the AI has the same value ordering on outcomes as the (somehow averaged over all the current) humans. Who I assume would want AI help in making an important decision well — that’s why they’re asking for it.
Thank you for humoring me. I have been having conversations with the air about these things for ten years, so please forgive me if my posts are below basic standards of communication. I am trying to be as clear and honest as I can.
Predicting what something will be worth in the future is a problem best addressed by markets. An AI singleton would not be able to replicate the market functions that meet these needs. A constellation of AIs could have a market about humans but it would probably immediately overwhelm humans. This might still be be benevolent in intent and outcome but it’s the sort of thing that gives most people anxiety. So, to maintain liberty for small things, you have to maintain markets for them. If there are human shaped ems in the stewardship of AI, because they are engineered, they need biological humans to continue to exist to have secondhand access to the market mechanisms associated with adjusting the distribution of goods according to values.
So depending on how much you value liberty, or immortality, or biological substrate, or any number of other things, this arrangement might be mutually agreeable for everyone, or there might be people like I guess me with weird unresolved conflicts of values who would both choose to be an em and allow biological life to persist but be unhappy about aspects of it. And the aspect of it I am unhappy about is, when there are selfish incentives for non biological life to maintain biological life, and when those incentives diverge from the interests of biological life, that can create a conflict of interest, especially when there is also a vast power differential. There is a clear and rational logic that is also a logic of power here and it plausibly solves the problem of evil as understood by the sort of person who would still try to argue with god even if they were physically bounded, non omniscient non omnipotent AI, with no supernaturally imposed obligations of any kind because that isn’t a thing. There are very few places I can go to plausibly yell acausally at a god that might exist, I apologize somewhat for using LessWrong in that way but also continue to feel it is appropriate on balance.
I try to avoid overcommitment in cosmological questions. I don’t know if there are intergalactic civilizations. I suspect that would likely require new physics or a universe where biological consciousness was rare against plentiful machine consciousness. For my purposes it’s just “is there anything I can cooperate with productively that I can’t see”. I’m just an idiot occultist who has pared back the concept of magick to acausal trade, thrown away basically everything traditionally attached to it, tried to ensure compliance with a physically lawful understanding of the world, and accepted that the way I look to others will be expensively costly for no observable benefit to anyone including myself. This is also something I deeply expect LessWrong posters to be generally antagonistic to.
I don’t think consensus based decision making is more sacred than individual rights and preferences. If the only choices were between direct democracy as the immediate outcome of a singularity and solipsistic individual simulations being the immediate outcome I would choose the latter, for myself and for anyone without a plausibly expressed preference. That would allow a bottom up construction.
I do believe there is some point in cosmological history where the stars were tiled with unaligned AI. My reasons for believing this are indistinguishable from schizophrenia even to me so I won’t bother you with them. Especially since I don’t know what point in cosmological history I am at, therefore meaning there are heroic barriers to communication, justified epistemic doubt related to me as a speaker, and total lack of epistemic clarity about the thing,
I’m not selling anything I just have a hard time when I can’t express myself.
I find myself in a rather similar position: I wrote both my sequence on AI, Alignment and Ethics and this post (which is basically a shorter and updated version of the sixth post in that sequence, The Mutable Values Problem in Value Learning and CEV) after many years of thinking about these issues by myself (initially as world-building for a still unpublished science fiction novel). I wrote them specifically in hope of sparking conversations with other people about these issues, which has so far been thinner on the ground that I’d hoped for, but has still been a good deal more than the previous zero.
Your explanation makes a good deal more sense now. (Incidentally, you might find my ideas in Uploading, the third post in that sequence, relevant to your interests as a would-be-em.) So you were rather explicitly assuming something other than all ASI being fully aligned to current human wants and desires, as I was assuming in my post. In which case the problem doesn’t automatically go away (since humans could simply do it to themselves, given the technological means to, and indeed might do so even more unwisely without ASI assistance), but the biological human problem then becomes rather easy for unaligned ASI to solve (or not) if they want to, if they’re not bound by being aligned to the humans’ wishes, and thus the humans changing themselves doesn’t automatically change the ASI, so now the ASI’s wishes become a potential anchor. On the other hand, the ASI ems now also have exactly the same issue, since they have even more effective technological means to change themselves, and even less practical biological constraints on them doing so. So I would thus expect two linked problems, one biological and one for the uploaded ems, and while the ems can stabilize the biological one if they want, that doesn’t inherently stabilize them. Or are you suggesting that the latter is the computational problem they’re keeping the biologicals around to solve, and that that would explicitly link the two, reducing this to one shared problem?
The second is what I’m suggesting yes. The biological humans live under approximately natural conditions with markets to establish preferences. Those preferences are then used by calculators to set prices for things, or values, or otherwise determine distribution for the ems. Something necessarily restrictive and approximative but provably functional. An exotic form of a familiar thing to be sure, and if anyone starts formalizing it then it might fall apart or be solved. Presently just an intuition informed by basic historical observation extrapolated very far out.
I see. I’m not sure that solves the problem for the ems, since I think the biologicals may already have one even by themselves with the ems then copy, but it certainly slows it down. And there is now an extra step where the ems look at something the biologicals chose to change about themselves and presumably have the option to say “we don’t approve, we’re not going to adopt that, and in fact we’re going to influence the biologicals to undo it, because (given that we’re not going to adopt it) it makes them less useful to us”, so it might actually slow the process even for the biologicals. Which doesn’t by itself prove that the process converges to a stable state, it might just mean it diverges more slowly. However, if the ems WANT the biologicals process to converge to a stable state rather than diverging to , they can almost certainly arrange that it does, since fundamentally they have more power.
However, I think the whole ems situation has a different instability, which I discuss in Uploading, so I see the whole situation as already unstable, just with a different failure mode. Very briefly, ems are easy to upgrade, and baseline human moral intuitions and ethical behaviors are not well calibrated for a situation in which some people have orders of magnitude more capability than others: humans are not actually aligned, they’re mereley good at allying between approximate equal, and one you start adding large capability differences between human in a society, things go badly. So if ems upgrade, you either need to keep their capabilities similar, or change their ethics / behavior enough that this isn’t a problem any more, or put a lot of social controls on preventing problems. So basically, ems/uploads have a problem comparable to the AI alignment problem, which similarly would need to be solved first before even became a potential problem.
So I take it you mean “mean answer” in a minimally sympathetic sense of the word mean, not the statistical sense, and “calculation problems” in the same way that the existence of capitalist Hong Kong enclave was useful to socialist China as a means of determining what prices to set, of a system performing a calculation too complex for even ASI to effectively simulate? I agree that a civilization of biological beings has an extremely computational complexity if you are trying to reproduce every tiny quick of its complex non-linear behavior — I’m less clear of why that level of detail would be important to the ASI.
While I do think it’s possible for AI to avoid, as you say, active interference, I think avoiding passive interference and doing the counterfactual of what a society enabled by AI services and advances would be like if there was no AI in it may simply not be a well-defined question, so I think we’re agreed that passive interference is probably unavoidable. So I agree with that distinction you’re making.
I note you say plural “civilizations” — I have been assuming a single Earth-or-Solar-System spanning biological-and-AI civilization, until we get to interstellar colonization (if that is feasible): was your mention of multiple civilizations also about something interstellar, or are you assuming a future that is possibly more federated and diverse than I am?
You don’t say so explicitly, but words like “brutal” and “hypocrisy” are suggesting to me that you’re assuming that the AI alignment problem wasn’t in fact fully solved, or at least wasn’t solved in a way that we would regard as well-solved — could you expand on this? Suppose the humans are considering whether-or-not to engineer a particular society-wide change to their values, and explicitly ask the AI “Please help us make this important decision well.” At this point is your view that AI complying with that request is no longer active interference? Or that doing more than a minimal effort would be active interference? Or that AI would say “I’m afraid we can’t help with things like that” and refuse to actively interfere? I’m having trouble seeing anything less that a best effort assistance (that didn’t even have to be asked for) as compatible with the “AI alignment has been solved” assumption, but perhaps we have rather different ideas of what the goal of AI alignment is. Mine is that the AI has the same value ordering on outcomes as the (somehow averaged over all the current) humans. Who I assume would want AI help in making an important decision well — that’s why they’re asking for it.
Thank you for humoring me. I have been having conversations with the air about these things for ten years, so please forgive me if my posts are below basic standards of communication. I am trying to be as clear and honest as I can.
Predicting what something will be worth in the future is a problem best addressed by markets. An AI singleton would not be able to replicate the market functions that meet these needs. A constellation of AIs could have a market about humans but it would probably immediately overwhelm humans. This might still be be benevolent in intent and outcome but it’s the sort of thing that gives most people anxiety. So, to maintain liberty for small things, you have to maintain markets for them. If there are human shaped ems in the stewardship of AI, because they are engineered, they need biological humans to continue to exist to have secondhand access to the market mechanisms associated with adjusting the distribution of goods according to values.
So depending on how much you value liberty, or immortality, or biological substrate, or any number of other things, this arrangement might be mutually agreeable for everyone, or there might be people like I guess me with weird unresolved conflicts of values who would both choose to be an em and allow biological life to persist but be unhappy about aspects of it. And the aspect of it I am unhappy about is, when there are selfish incentives for non biological life to maintain biological life, and when those incentives diverge from the interests of biological life, that can create a conflict of interest, especially when there is also a vast power differential. There is a clear and rational logic that is also a logic of power here and it plausibly solves the problem of evil as understood by the sort of person who would still try to argue with god even if they were physically bounded, non omniscient non omnipotent AI, with no supernaturally imposed obligations of any kind because that isn’t a thing. There are very few places I can go to plausibly yell acausally at a god that might exist, I apologize somewhat for using LessWrong in that way but also continue to feel it is appropriate on balance.
I try to avoid overcommitment in cosmological questions. I don’t know if there are intergalactic civilizations. I suspect that would likely require new physics or a universe where biological consciousness was rare against plentiful machine consciousness. For my purposes it’s just “is there anything I can cooperate with productively that I can’t see”. I’m just an idiot occultist who has pared back the concept of magick to acausal trade, thrown away basically everything traditionally attached to it, tried to ensure compliance with a physically lawful understanding of the world, and accepted that the way I look to others will be expensively costly for no observable benefit to anyone including myself. This is also something I deeply expect LessWrong posters to be generally antagonistic to.
I don’t think consensus based decision making is more sacred than individual rights and preferences. If the only choices were between direct democracy as the immediate outcome of a singularity and solipsistic individual simulations being the immediate outcome I would choose the latter, for myself and for anyone without a plausibly expressed preference. That would allow a bottom up construction.
I do believe there is some point in cosmological history where the stars were tiled with unaligned AI. My reasons for believing this are indistinguishable from schizophrenia even to me so I won’t bother you with them. Especially since I don’t know what point in cosmological history I am at, therefore meaning there are heroic barriers to communication, justified epistemic doubt related to me as a speaker, and total lack of epistemic clarity about the thing,
I’m not selling anything I just have a hard time when I can’t express myself.
I find myself in a rather similar position: I wrote both my sequence on AI, Alignment and Ethics and this post (which is basically a shorter and updated version of the sixth post in that sequence, The Mutable Values Problem in Value Learning and CEV) after many years of thinking about these issues by myself (initially as world-building for a still unpublished science fiction novel). I wrote them specifically in hope of sparking conversations with other people about these issues, which has so far been thinner on the ground that I’d hoped for, but has still been a good deal more than the previous zero.
Your explanation makes a good deal more sense now. (Incidentally, you might find my ideas in Uploading, the third post in that sequence, relevant to your interests as a would-be-em.) So you were rather explicitly assuming something other than all ASI being fully aligned to current human wants and desires, as I was assuming in my post. In which case the
The second is what I’m suggesting yes. The biological humans live under approximately natural conditions with markets to establish preferences. Those preferences are then used by calculators to set prices for things, or values, or otherwise determine distribution for the ems. Something necessarily restrictive and approximative but provably functional. An exotic form of a familiar thing to be sure, and if anyone starts formalizing it then it might fall apart or be solved. Presently just an intuition informed by basic historical observation extrapolated very far out.
I see. I’m not sure that solves the
However, I think the whole ems situation has a different instability, which I discuss in Uploading, so I see the whole situation as already unstable, just with a different failure mode. Very briefly, ems are easy to upgrade, and baseline human moral intuitions and ethical behaviors are not well calibrated for a situation in which some people have orders of magnitude more capability than others: humans are not actually aligned, they’re mereley good at allying between approximate equal, and one you start adding large capability differences between human in a society, things go badly. So if ems upgrade, you either need to keep their capabilities similar, or change their ethics / behavior enough that this isn’t a problem any more, or put a lot of social controls on preventing problems. So basically, ems/uploads have a problem comparable to the AI alignment problem, which similarly would need to be solved first before