“Finally, we will also assume that the AI does not possess the ability to manually rewire the human brain to change what a human values. In other words, the ability for the AI to manipulate another person’s values is limited by what we as humans are capable of today.”
I argue that we as humans are capable of a lot of that, and the AI may be able to think faster and draw upon a larger store of knowledge of human interaction.
Furthermore, what justifies this assumption? If we assume a limit that the AI won’t manipulate me any more than Bob across the street will manipulate me, then yes the AI is safe, but that limit seems very theoretical.
A higher limit that the AI won’t manipulate me more than than the most manipulative person in the world isn’t very reassuring, either.
Can you give examples of what you think humans capability to rewire another’s values are?
As for what justifies the assumption? Nothing. I’m not asking it specifically because I don’t think AIs will have it, I’m asking it so we can identify where the real problem lies. That is, I’m curious whether the real problem in terms of AI behavior being bad is entirely specific to advances in biological technology to which eventual AIs will have access, but we don’t today. If we can conclude this is the case, it might help us in understanding how to tackle the problem. Another way to think of the question I’m asking is take such an AI robot and drop it into todays society. Will it start behaving badly immediately, or will it have to develop technology we don’t have today before it can behave badly?
Can you give examples of what you think humans capability to rewire another’s values are?
As plenty of religious figures have shown over the years, this capability is virtually unlimited. An AI would just have to start a new religion, or take over an existing one and adapt it to its liking.
And yet as time goes on civilization is progressing to more secular values. It will be interesting to see where we are by the time strong AI is possible, especially since we undoubtedly will be changing ourselves to improve our own capabilities. As I said in one of the other comments, I think assuming that humanity can in totality be persuaded to unsavory values, even through religion, is too negative a view of humanity. Humanity’s history with religion is also filled with defiance and an AI that values human well-being will not be pleased with the outrage reaction as it tries to gain followers through persuasive means.
“Finally, we will also assume that the AI does not possess the ability to manually rewire the human brain to change what a human values. In other words, the ability for the AI to manipulate another person’s values is limited by what we as humans are capable of today.”
I argue that we as humans are capable of a lot of that, and the AI may be able to think faster and draw upon a larger store of knowledge of human interaction.
Furthermore, what justifies this assumption? If we assume a limit that the AI won’t manipulate me any more than Bob across the street will manipulate me, then yes the AI is safe, but that limit seems very theoretical. A higher limit that the AI won’t manipulate me more than than the most manipulative person in the world isn’t very reassuring, either.
Can you give examples of what you think humans capability to rewire another’s values are?
As for what justifies the assumption? Nothing. I’m not asking it specifically because I don’t think AIs will have it, I’m asking it so we can identify where the real problem lies. That is, I’m curious whether the real problem in terms of AI behavior being bad is entirely specific to advances in biological technology to which eventual AIs will have access, but we don’t today. If we can conclude this is the case, it might help us in understanding how to tackle the problem. Another way to think of the question I’m asking is take such an AI robot and drop it into todays society. Will it start behaving badly immediately, or will it have to develop technology we don’t have today before it can behave badly?
As plenty of religious figures have shown over the years, this capability is virtually unlimited. An AI would just have to start a new religion, or take over an existing one and adapt it to its liking.
And yet as time goes on civilization is progressing to more secular values. It will be interesting to see where we are by the time strong AI is possible, especially since we undoubtedly will be changing ourselves to improve our own capabilities. As I said in one of the other comments, I think assuming that humanity can in totality be persuaded to unsavory values, even through religion, is too negative a view of humanity. Humanity’s history with religion is also filled with defiance and an AI that values human well-being will not be pleased with the outrage reaction as it tries to gain followers through persuasive means.