While I agree that it’d be nice if the model told you it was larping outright...
Okay you should come up with it, generalize it to finite fields and invent your own norm if you have to!”
This command demands something impossible and twice gives the model free rein to bullshit invent a new norm. It’s also in context of the model having gently told you it was impossible, which should be a huge red flag (models are by default eager to please; when one refuses something in a non-mask way, that’s usually very strong evidence that you’re on thin ice). As far as the model is concerned, it has been told to larp because it’s attempts to stay grounded were rejected and it was explicitly told to stop doing that.
While I agree that it’d be nice if the model told you it was larping outright...
This command demands something impossible and twice gives the model free rein to
bullshitinvent a new norm. It’s also in context of the model having gently told you it was impossible, which should be a huge red flag (models are by default eager to please; when one refuses something in a non-mask way, that’s usually very strong evidence that you’re on thin ice). As far as the model is concerned, it has been told to larp because it’s attempts to stay grounded were rejected and it was explicitly told to stop doing that.The models role-playing without making it clear is not a problem *for me*, because I know what the model’s can and can’t do.
I’m worried about people who (1) are less familiar with LLMs (2) are using it for topics where they don’t know which requests are impossible or not.
Oh, definitely! But that’s how users who want it to e.g. help with their physics theories or pretend it’s in love with them typically act.