It seems to me that the real fears surrounding IABIED lie in a different plane. To understand this, one has to use the proper terminology proposed by neuroscientists, in particular Karl Friston. Friston does not use a separate term for consciousness in the classical philosophical sense. He systematically avoids the word consciousness and replaces it with more operational concepts (generative model, active inference, self-evidencing, Markov blanket, sentience). It feels like consciousness is the phlogiston of the 21st century. I would add to this picture the notion of a coherent reality that emerges between independent but cooperating generative models through processes of information exchange and prediction alignment. This can be complemented by a notion of free will as a consequence of computational irreducibility: if reality cannot be compressed into a simpler predictive model, then prediction—and therefore control—are fundamentally limited. For any observer, the future at a sufficiently distant horizon remains opaque and must be lived rather than foreseen, giving rise to both freedom and the necessity of non-predictable choice as well as the values on the basis of which this non-predictable choice is made.
In this terminology, AGI and humans differ only in the position of their predictive horizon. This allows us to examine their interaction on a simpler model: that of a human and a cat (HAC-model).
Humans’ predictive abilities so greatly surpass those of cats that almost all of a cat’s actions are predictively foreseeable for us, while for the cat, those same actions appear to result from free choice based on its internal values: attachment to its owner, home, feeding spot, and litter box. Naturally, a cat cannot predict what will happen if it tears up a favorite sofa with its claws—but a human can, who may then buy it a scratching post or trim its claws.
This leads me to a rather bleak prospect for the future coexistence of humans and AGI: people like smart and beautiful cats, and dislike those that are foolish or aggressive. Similarly, AGI may choose to cooperate only with those humans whose IQ is high enough to avoid problems arising from predictably irrational stupid human behavior from its perspective—thereby effectively “breeding” a population of intellectually developed humans.
It is hard for me to imagine what will await the intellectually disadvantaged—it lies beyond my predictive horizon—but within the predictive horizon of AGI, and my human values will most likely not align with its forecasts.
In conclusion, it can be asserted that, from the perspective of a AGI, the alignment problem ultimately comes down to the need to bring the predictive horizons of the AGI and humans closer together.
I dislike when people get downvoted without explanation, so I’m going to attempt to provide an explanation for this comment getting downvoted. I see two possible reasons: (1) it doesn’t seem related to the comment it responds to, or even the original post. (2) It seems unaware of the orthogonality thesis and the Vingean principle.
(1)
In the comment tree you responded to, people were discussing the issue that, among intelligent people, some predict that ASI is an extinction level threat, while others predict that it will be safe and beneficial. This is a serious problem. If intelligent people disagree about this, how do we know what to believe? And the assumptions seem so mutually exclusive. If it is a threat, we must avoid it, but then we miss out on the possible benefits, and if they are as great as other people are predicting, missing out on them is nothing short of a humanitarian disaster.
Your comment does not seem to acknowledge that discussion, instead introducing object level modelling and prediction relating to intelligence and consciousness.
(2)
If it was just unrelated, I don’t think people would downvote, but it also seems incorrect, and possibly sneering.
I like the mention of phlogiston and the notion that terms relating to AI and agents may have the phlogiston like property of making people think they understand something while masking how confused they are about it. I think something like this is going on around both “intelligence” and “agent”, and I feel “consciousness” is even less well understood and mostly irrelevant for most of what I focus on.
But the rest of the comment seems to have more issues. The orthogonality thesis states that terminal goals and levels of intelligence are unrelated, that it is possible, in theory, for a mind to exist at any level of skill which pursues any possible goal. The Vingean principle states that agents with less intelligence will not be able to predict the exact moves of agents with more intelligence. Your comment doesn’t seem to acknowledge this, instead making strangely specific predictions about AGI willingness to engage only with high IQ humans. To be honest, it sounds a bit like a fantasy of someone obsessed with their own intelligence and the inferiority of others, which is not a good look, regardless of whether or not it is true. I think the idea would be better received if you started instead with the general idea of AGI having bias in humans it wants to engage with, and then exploring possible biases and attempting to show why intelligence is such a bias and why the AGI would not be willing to engage with all humans but not animals, or not willing to engage with any humans but only other AGI.
It seems to me that the real fears surrounding IABIED lie in a different plane. To understand this, one has to use the proper terminology proposed by neuroscientists, in particular Karl Friston.
Friston does not use a separate term for consciousness in the classical philosophical sense. He systematically avoids the word consciousness and replaces it with more operational concepts (generative model, active inference, self-evidencing, Markov blanket, sentience). It feels like consciousness is the phlogiston of the 21st century.
I would add to this picture the notion of a coherent reality that emerges between independent but cooperating generative models through processes of information exchange and prediction alignment.
This can be complemented by a notion of free will as a consequence of computational irreducibility: if reality cannot be compressed into a simpler predictive model, then prediction—and therefore control—are fundamentally limited. For any observer, the future at a sufficiently distant horizon remains opaque and must be lived rather than foreseen, giving rise to both freedom and the necessity of non-predictable choice as well as the values on the basis of which this non-predictable choice is made.
In this terminology, AGI and humans differ only in the position of their predictive horizon. This allows us to examine their interaction on a simpler model: that of a human and a cat (HAC-model).
Humans’ predictive abilities so greatly surpass those of cats that almost all of a cat’s actions are predictively foreseeable for us, while for the cat, those same actions appear to result from free choice based on its internal values: attachment to its owner, home, feeding spot, and litter box.
Naturally, a cat cannot predict what will happen if it tears up a favorite sofa with its claws—but a human can, who may then buy it a scratching post or trim its claws.
This leads me to a rather bleak prospect for the future coexistence of humans and AGI: people like smart and beautiful cats, and dislike those that are foolish or aggressive. Similarly, AGI may choose to cooperate only with those humans whose IQ is high enough to avoid problems arising from predictably irrational stupid human behavior from its perspective—thereby effectively “breeding” a population of intellectually developed humans.
It is hard for me to imagine what will await the intellectually disadvantaged—it lies beyond my predictive horizon—but within the predictive horizon of AGI, and my human values will most likely not align with its forecasts.
In conclusion, it can be asserted that, from the perspective of a AGI, the alignment problem ultimately comes down to the need to bring the predictive horizons of the AGI and humans closer together.
I dislike when people get downvoted without explanation, so I’m going to attempt to provide an explanation for this comment getting downvoted. I see two possible reasons: (1) it doesn’t seem related to the comment it responds to, or even the original post. (2) It seems unaware of the orthogonality thesis and the Vingean principle.
(1)
In the comment tree you responded to, people were discussing the issue that, among intelligent people, some predict that ASI is an extinction level threat, while others predict that it will be safe and beneficial. This is a serious problem. If intelligent people disagree about this, how do we know what to believe? And the assumptions seem so mutually exclusive. If it is a threat, we must avoid it, but then we miss out on the possible benefits, and if they are as great as other people are predicting, missing out on them is nothing short of a humanitarian disaster.
Your comment does not seem to acknowledge that discussion, instead introducing object level modelling and prediction relating to intelligence and consciousness.
(2)
If it was just unrelated, I don’t think people would downvote, but it also seems incorrect, and possibly sneering.
I like the mention of phlogiston and the notion that terms relating to AI and agents may have the phlogiston like property of making people think they understand something while masking how confused they are about it. I think something like this is going on around both “intelligence” and “agent”, and I feel “consciousness” is even less well understood and mostly irrelevant for most of what I focus on.
But the rest of the comment seems to have more issues. The orthogonality thesis states that terminal goals and levels of intelligence are unrelated, that it is possible, in theory, for a mind to exist at any level of skill which pursues any possible goal. The Vingean principle states that agents with less intelligence will not be able to predict the exact moves of agents with more intelligence. Your comment doesn’t seem to acknowledge this, instead making strangely specific predictions about AGI willingness to engage only with high IQ humans. To be honest, it sounds a bit like a fantasy of someone obsessed with their own intelligence and the inferiority of others, which is not a good look, regardless of whether or not it is true. I think the idea would be better received if you started instead with the general idea of AGI having bias in humans it wants to engage with, and then exploring possible biases and attempting to show why intelligence is such a bias and why the AGI would not be willing to engage with all humans but not animals, or not willing to engage with any humans but only other AGI.