I see. Is it fair to describe your perspective as descriptive rather than prescriptive on this matter (outside of suggesting things to avoid bad interventions)?
Jakub Janiak
Karma: 4
Bump on this as I’m genuinely curious as well
“There is no utility or reward function, it’s just prediction all the way down[4].” This was a very illuminating statement to better understand your model. It seems as though from this, we’re able to shape LLMs into particular self-prediction via system prompt, persona vectors, etc.
Do you have a perspective on what kind of “self” we ought to shape LLMs into? Does it make sense to give LLMs a sense of self vs shaping them to be more tool-like?
“I focus on a scenario where an international agency enforces drastic limits to AI development for two years, starting at the beginning of 2028”
What is your p value for this occurring?
I largely view markets as pooled public bounty for information. Not only does it help in delivering the elicitation reward to the person with relevant information, but it also pools resources to pay that person. I don’t think AI superforecasters will know exactly who to reach out to for information.
The lack of improved rationality, purely from a betting and capabilities perspective, would be surprising to me in future AI systems. But broadly agree it’s a large brush stroke assumption.
It’s not clear to me where in your link the formulation of general-purpose prediction markets can be attributed to anyone else. Particular use cases for betting on papacy or a political election isn’t necessarily obvious to me.
imo training multi-agent RL feels like it’s the most likely driver of this. That said, most multi-agent RL in LLM environments I’m aware of is non adversarial.