Some abstractions are heavily determined by the territory. The concept of trees is pretty heavily determined by the territory. Whereas the concept of betrayal is determined by the way that human minds function, which is determined by other people’s abstractions. So while it seems reasonably likely to me that an AI “naturally thinks” in terms of the same low-level abstractions as humans, it thinking in terms of human high-level abstractions seems much less likely, absent some type of safety intervention. Which is particularly important because most of the key human values are very high-level abstractions.
My guess is that if you have to deal with humans, as at least early AI systems will have to do, then abstractions like “betrayal” are heavily determined.
I agree that if you don’t have to deal with humans, then things like “betrayal” may not arise; similarly if you don’t have to deal with Earth, then “trees” are not heavily determined abstractions.
Some abstractions are heavily determined by the territory. The concept of trees is pretty heavily determined by the territory. Whereas the concept of betrayal is determined by the way that human minds function, which is determined by other people’s abstractions. So while it seems reasonably likely to me that an AI “naturally thinks” in terms of the same low-level abstractions as humans, it thinking in terms of human high-level abstractions seems much less likely, absent some type of safety intervention. Which is particularly important because most of the key human values are very high-level abstractions.
My guess is that if you have to deal with humans, as at least early AI systems will have to do, then abstractions like “betrayal” are heavily determined.
I agree that if you don’t have to deal with humans, then things like “betrayal” may not arise; similarly if you don’t have to deal with Earth, then “trees” are not heavily determined abstractions.