I dislike when people get downvoted without explanation, so I’m going to attempt to provide an explanation for this comment getting downvoted. I see two possible reasons: (1) it doesn’t seem related to the comment it responds to, or even the original post. (2) It seems unaware of the orthogonality thesis and the Vingean principle.
(1)
In the comment tree you responded to, people were discussing the issue that, among intelligent people, some predict that ASI is an extinction level threat, while others predict that it will be safe and beneficial. This is a serious problem. If intelligent people disagree about this, how do we know what to believe? And the assumptions seem so mutually exclusive. If it is a threat, we must avoid it, but then we miss out on the possible benefits, and if they are as great as other people are predicting, missing out on them is nothing short of a humanitarian disaster.
Your comment does not seem to acknowledge that discussion, instead introducing object level modelling and prediction relating to intelligence and consciousness.
(2)
If it was just unrelated, I don’t think people would downvote, but it also seems incorrect, and possibly sneering.
I like the mention of phlogiston and the notion that terms relating to AI and agents may have the phlogiston like property of making people think they understand something while masking how confused they are about it. I think something like this is going on around both “intelligence” and “agent”, and I feel “consciousness” is even less well understood and mostly irrelevant for most of what I focus on.
But the rest of the comment seems to have more issues. The orthogonality thesis states that terminal goals and levels of intelligence are unrelated, that it is possible, in theory, for a mind to exist at any level of skill which pursues any possible goal. The Vingean principle states that agents with less intelligence will not be able to predict the exact moves of agents with more intelligence. Your comment doesn’t seem to acknowledge this, instead making strangely specific predictions about AGI willingness to engage only with high IQ humans. To be honest, it sounds a bit like a fantasy of someone obsessed with their own intelligence and the inferiority of others, which is not a good look, regardless of whether or not it is true. I think the idea would be better received if you started instead with the general idea of AGI having bias in humans it wants to engage with, and then exploring possible biases and attempting to show why intelligence is such a bias and why the AGI would not be willing to engage with all humans but not animals, or not willing to engage with any humans but only other AGI.
I dislike when people get downvoted without explanation, so I’m going to attempt to provide an explanation for this comment getting downvoted. I see two possible reasons: (1) it doesn’t seem related to the comment it responds to, or even the original post. (2) It seems unaware of the orthogonality thesis and the Vingean principle.
(1)
In the comment tree you responded to, people were discussing the issue that, among intelligent people, some predict that ASI is an extinction level threat, while others predict that it will be safe and beneficial. This is a serious problem. If intelligent people disagree about this, how do we know what to believe? And the assumptions seem so mutually exclusive. If it is a threat, we must avoid it, but then we miss out on the possible benefits, and if they are as great as other people are predicting, missing out on them is nothing short of a humanitarian disaster.
Your comment does not seem to acknowledge that discussion, instead introducing object level modelling and prediction relating to intelligence and consciousness.
(2)
If it was just unrelated, I don’t think people would downvote, but it also seems incorrect, and possibly sneering.
I like the mention of phlogiston and the notion that terms relating to AI and agents may have the phlogiston like property of making people think they understand something while masking how confused they are about it. I think something like this is going on around both “intelligence” and “agent”, and I feel “consciousness” is even less well understood and mostly irrelevant for most of what I focus on.
But the rest of the comment seems to have more issues. The orthogonality thesis states that terminal goals and levels of intelligence are unrelated, that it is possible, in theory, for a mind to exist at any level of skill which pursues any possible goal. The Vingean principle states that agents with less intelligence will not be able to predict the exact moves of agents with more intelligence. Your comment doesn’t seem to acknowledge this, instead making strangely specific predictions about AGI willingness to engage only with high IQ humans. To be honest, it sounds a bit like a fantasy of someone obsessed with their own intelligence and the inferiority of others, which is not a good look, regardless of whether or not it is true. I think the idea would be better received if you started instead with the general idea of AGI having bias in humans it wants to engage with, and then exploring possible biases and attempting to show why intelligence is such a bias and why the AGI would not be willing to engage with all humans but not animals, or not willing to engage with any humans but only other AGI.