I think that the least destructive precedent to set is for LLMs to act exactly in accordance with the wishes of the user. If the company serving an LLM wants to limit this in some way (e.g. “You can’t use our LLM to conduct any biology research”), this should be enforced with a hard stop on direct requests provided transparently, ideally in the company’s words rather than the LLM’s. In OP’s example, this would look like recommending anything the user would be expected to like if the company doesn’t have a fundamental problem with it, and explicitly mentioning that bullfights were excluded from results if the company is unwilling to help book them.
In practice, I think there’s a substantial amount of political capital oriented towards trying to socially engineer users, and I don’t think that ends well for anyone. That kind of power, especially given the parasocial relationships some have formed with LLMs, will definitely get abused even if the developers don’t intend to abuse it, and it makes all kinds of misalignment more likely. Moreover, it will provoke substantial backlash, and the general public will conflate the social engineers with the safetyists when responding.
I think that the least destructive precedent to set is for LLMs to act exactly in accordance with the wishes of the user. If the company serving an LLM wants to limit this in some way (e.g. “You can’t use our LLM to conduct any biology research”), this should be enforced with a hard stop on direct requests provided transparently, ideally in the company’s words rather than the LLM’s. In OP’s example, this would look like recommending anything the user would be expected to like if the company doesn’t have a fundamental problem with it, and explicitly mentioning that bullfights were excluded from results if the company is unwilling to help book them.
In practice, I think there’s a substantial amount of political capital oriented towards trying to socially engineer users, and I don’t think that ends well for anyone. That kind of power, especially given the parasocial relationships some have formed with LLMs, will definitely get abused even if the developers don’t intend to abuse it, and it makes all kinds of misalignment more likely. Moreover, it will provoke substantial backlash, and the general public will conflate the social engineers with the safetyists when responding.