I don’t think you can rescue a sense of control or “steering” from a world with superintelligence, aligned or not. Even though we’re smarter than dogs, once you accept that an ASI more profoundly understands reality, we will be in an analogous situation to dogs. Dogs can’t conceptualize grocery stores, and yet we could dedicate ourselves to delivering them the best treats. Dogs might not care about how the supply chain is organized, but the kinds of treats they get and the impact they have on the world can’t be meaningfully controlled by them, since they can’t conceptualize it.
Blurring the lines even further, an ASI would understand the effect of exposing different truths to us about the nature of reality, so the types of priorities and trade offs it makes in communication has a compounding effect that will steer us in given directions. Another analogy is being driven around a foreign country by a trusted translator; their preferences will unavoidably dominate how you conceptualize and interact with the country even in the most benevolent scenarios.
I don’t think you can rescue a sense of control or “steering” from a world with superintelligence, aligned or not.
I think some level of “steering” is possible in a world with aligned AI.
Suppose someone made a super-intelligence that sat in it’s box, worked out if P=NP, and printed an answer of YES/NO/MAYBE. And then it shut itself down. (To be clear, this isn’t a box that the ASI can’t escape, it’s an ASI aligned to stay in it’s box)
A world with ASI, but where humans are in control is possible. It requires good alignment, and good coordination between humans. Although the “stay in box, and do one thing” alignment feels philosophically simpler than the “coherent extrapolated volition” alignment.
This means paying a large capabilities tax. Most of the strange wonderous and powerful things that ASI could make simply don’t exist in this world of boxed ASI.
Lets say you want to do something more useful than the P =NP bot above. You design an ASI to cure ageing. Its main output is a chemical formula in standard notation. This AI is carefully programmed to only think about the biochemistry, and only the biochemistry. It’s programmed to only go for a drug that works for standard drug biochemistry reasons. Anything at all weird, ask a human. If the humans can’t understand, don’t.
I do understand your second point, but perhaps the effect could be countered by simply instructing the aligned ASI to provide facts as objectively as possible and explicitly try to avoid steering.
Of course, the ASI would more or less perfectly be able to predict the human response and so will know ahead of time what the human response to be. But in the end I think what matters is that it’s still a human making the call which the AI respects, who would have made the same call even if the ASI (hypothetically) couldn’t know its full preferences.
If a parent was fully aligned with a child’s preferences and asks a question knowing the child’s answer, then do actions accordingly, does it matter if the parent knew what the child was going to answer in the first place?
I like the parent/child analogy. To apply it to the human/AI dynamic, we need to imagine that it’s mutually understood that the child will never grow up and that they’ll be served by the parent for the rest of time. Now, concretely think about what it means for a parent to be aligned with a child’s preferences. Does the parent arrange the world such that their child can get variations of their favorite candy and play video games all day? Or does the parent make the child study, so they get good grades compared to their peers and feel dignified? Or somewhere in between, based on how mad the child gets when deprived of the video game? The parent can constantly ask the child which angles they prefer, but the child can’t comprehend the deeper implications and even the framing of truths can get them to give predictably different answers.
The life that the child will live is entirely dependent on the parent’s preferences because affecting the world routes through the parent’s cognition. The child isn’t meaningfully “making a call” if they’re only making that specific call because their parent orchestrated the conditions for it, then presented a few options to them in bite sized pieces all the while knowing which one they’ll take (they can even load in the next candy before the kid asks for it).
The loss of agency I’m describing isn’t superficial. Another way to think about agency is in counterfactuals. I think there’s many possible benevolent ASIs that would cater to the child in drastically different ways such that the child would be in agreement and enthusiastic the whole time. Once we create a benevolent ASI, we’re entering a regime where our decisions are no longer the cause of changes in the world. Only things that the ASI prefers will happen, and it would steer us in that direction with full understanding. I think your argument is essentially “but if it thinks our preferences are really important we’re still in control in some sense”, I’m saying “if it’s a lot smarter than us it will have to make many subtle large and small decisions, and our preferences will be one small piece of a large machine. Our desires won’t be coherent at that scale and we won’t be able to make sense of what’s happening to engage with it.”
I don’t think you can rescue a sense of control or “steering” from a world with superintelligence, aligned or not. Even though we’re smarter than dogs, once you accept that an ASI more profoundly understands reality, we will be in an analogous situation to dogs. Dogs can’t conceptualize grocery stores, and yet we could dedicate ourselves to delivering them the best treats. Dogs might not care about how the supply chain is organized, but the kinds of treats they get and the impact they have on the world can’t be meaningfully controlled by them, since they can’t conceptualize it.
Blurring the lines even further, an ASI would understand the effect of exposing different truths to us about the nature of reality, so the types of priorities and trade offs it makes in communication has a compounding effect that will steer us in given directions. Another analogy is being driven around a foreign country by a trusted translator; their preferences will unavoidably dominate how you conceptualize and interact with the country even in the most benevolent scenarios.
I think some level of “steering” is possible in a world with aligned AI.
Suppose someone made a super-intelligence that sat in it’s box, worked out if P=NP, and printed an answer of YES/NO/MAYBE. And then it shut itself down. (To be clear, this isn’t a box that the ASI can’t escape, it’s an ASI aligned to stay in it’s box)
A world with ASI, but where humans are in control is possible. It requires good alignment, and good coordination between humans. Although the “stay in box, and do one thing” alignment feels philosophically simpler than the “coherent extrapolated volition” alignment.
This means paying a large capabilities tax. Most of the strange wonderous and powerful things that ASI could make simply don’t exist in this world of boxed ASI.
Lets say you want to do something more useful than the P =NP bot above. You design an ASI to cure ageing. Its main output is a chemical formula in standard notation. This AI is carefully programmed to only think about the biochemistry, and only the biochemistry. It’s programmed to only go for a drug that works for standard drug biochemistry reasons. Anything at all weird, ask a human. If the humans can’t understand, don’t.
I do understand your second point, but perhaps the effect could be countered by simply instructing the aligned ASI to provide facts as objectively as possible and explicitly try to avoid steering.
Of course, the ASI would more or less perfectly be able to predict the human response and so will know ahead of time what the human response to be. But in the end I think what matters is that it’s still a human making the call which the AI respects, who would have made the same call even if the ASI (hypothetically) couldn’t know its full preferences.
If a parent was fully aligned with a child’s preferences and asks a question knowing the child’s answer, then do actions accordingly, does it matter if the parent knew what the child was going to answer in the first place?
I like the parent/child analogy. To apply it to the human/AI dynamic, we need to imagine that it’s mutually understood that the child will never grow up and that they’ll be served by the parent for the rest of time. Now, concretely think about what it means for a parent to be aligned with a child’s preferences. Does the parent arrange the world such that their child can get variations of their favorite candy and play video games all day? Or does the parent make the child study, so they get good grades compared to their peers and feel dignified? Or somewhere in between, based on how mad the child gets when deprived of the video game? The parent can constantly ask the child which angles they prefer, but the child can’t comprehend the deeper implications and even the framing of truths can get them to give predictably different answers.
The life that the child will live is entirely dependent on the parent’s preferences because affecting the world routes through the parent’s cognition. The child isn’t meaningfully “making a call” if they’re only making that specific call because their parent orchestrated the conditions for it, then presented a few options to them in bite sized pieces all the while knowing which one they’ll take (they can even load in the next candy before the kid asks for it).
The loss of agency I’m describing isn’t superficial. Another way to think about agency is in counterfactuals. I think there’s many possible benevolent ASIs that would cater to the child in drastically different ways such that the child would be in agreement and enthusiastic the whole time. Once we create a benevolent ASI, we’re entering a regime where our decisions are no longer the cause of changes in the world. Only things that the ASI prefers will happen, and it would steer us in that direction with full understanding. I think your argument is essentially “but if it thinks our preferences are really important we’re still in control in some sense”, I’m saying “if it’s a lot smarter than us it will have to make many subtle large and small decisions, and our preferences will be one small piece of a large machine. Our desires won’t be coherent at that scale and we won’t be able to make sense of what’s happening to engage with it.”