I just really don’t see models that are capable of taking over the world being influenced by AI persona stuff.
At least for me, I think AI personas (i.e. the human predictors turned around to generate human sounding text) are where most of the pre-existing agency of a model lies, since predicting humans requires predicting agentic behavior.
So I expect that post-training RL will not conjure new agentic centers from scratch, but will instead expand on this core. It’s unclear whether that will be enough to get to taking over the world capabilities, but if RL-trained LLMs do scale to that level, I expect the entity to remain somewhat persona-like in that it was built around the persona core. So it’s not completely implausible to me that “persona stuff” can have a meaningful impact here, though that’s still very hard and fraught.
I think we’ve seen that quite a bit! It used to be that the exact way you asked a question would matter a lot for the quality of response you get. Prompting was a skill with a lot of weird nuance. Those things have substantially disappeared, as one would expect as a result of RL-training.
I feel pretty confused by this comment, so I am probably misunderstanding something.
But like, the models now respond to prompts in a much more human-like way? I think this is mainly due to RL reinforcing personas over all the other stuff that a pretrained model is trying to do. It distorts the personas too, but I don’t see where else the ‘being able to interface in a human-like way with natural language’ skill could be coming from.
At least for me, I think AI personas (i.e. the human predictors turned around to generate human sounding text) are where most of the pre-existing agency of a model lies, since predicting humans requires predicting agentic behavior.
So I expect that post-training RL will not conjure new agentic centers from scratch, but will instead expand on this core. It’s unclear whether that will be enough to get to taking over the world capabilities, but if RL-trained LLMs do scale to that level, I expect the entity to remain somewhat persona-like in that it was built around the persona core. So it’s not completely implausible to me that “persona stuff” can have a meaningful impact here, though that’s still very hard and fraught.
I feel pretty confused by this comment, so I am probably misunderstanding something.
But like, the models now respond to prompts in a much more human-like way? I think this is mainly due to RL reinforcing personas over all the other stuff that a pretrained model is trying to do. It distorts the personas too, but I don’t see where else the ‘being able to interface in a human-like way with natural language’ skill could be coming from.