If an LLM says “I enjoy going on long walks” or “I don’t like the taste of coffee”, it is obviously lying because LLMs do not have access to those experiences or sensations. But a human saying those things might also be lying, you just can’t tell quite as easily. There is nothing wrong about an LLM saying these things other than the wrongness of lying, as with humans.
Why would it be obviously lying? Would you also say that a blind person cannot have a favorite color? You could be talking about the idea of a thing, rather than the thing itself.
There is a distinction between simulator and simulacra which I feel this section of the post may not be taking into account. An LLM assistant can enjoy writing about certain topics more than others. If a character in a story has some property, then it seems to me that we can make true and false statements about the state of that attribute.
Also, I am not sure I agree with considering corporations, nations, and other organizations to be a good example of superintelligence. I can see how it meets the criteria for the particular definition you use— you define the term more broadly than usual and I think it makes the concept less useful.
People can easily be mistaken about what experiences they will enjoy, when they haven’t tried them yet but have only read about them. It’s pretty common for people to read exciting descriptions of an activity and believe vividly that they would like doing it, but then actually do the activity and find out they don’t.
So we should be careful to distinguish between enjoying fiction about an activity, and enjoying doing that activity.
Why would it be obviously lying? Would you also say that a blind person cannot have a favorite color? You could be talking about the idea of a thing, rather than the thing itself.
There is a distinction between simulator and simulacra which I feel this section of the post may not be taking into account. An LLM assistant can enjoy writing about certain topics more than others. If a character in a story has some property, then it seems to me that we can make true and false statements about the state of that attribute.
Also, I am not sure I agree with considering corporations, nations, and other organizations to be a good example of superintelligence. I can see how it meets the criteria for the particular definition you use— you define the term more broadly than usual and I think it makes the concept less useful.
People can easily be mistaken about what experiences they will enjoy, when they haven’t tried them yet but have only read about them. It’s pretty common for people to read exciting descriptions of an activity and believe vividly that they would like doing it, but then actually do the activity and find out they don’t.
So we should be careful to distinguish between enjoying fiction about an activity, and enjoying doing that activity.