One of my worries here is that a lot of results on LLMs are affected fairly deeply by data contamination, because LLMs essentially have the ~entire internet in their heads, meaning it’s very easy for them to state near perfect answers without having the ability to generalize very well.
To be clear, for the purposes of coherence theorems, behavior is enough, so this is enough evidence to say that LLMs are at least reasonably coherent towards expected utility maximization, and suggests coherence theorems have real-life consequences.
But I do think it matters a little whether it’s internally represented for real-life AI, because of generalization issues.
One of my worries here is that a lot of results on LLMs are affected fairly deeply by data contamination, because LLMs essentially have the ~entire internet in their heads, meaning it’s very easy for them to state near perfect answers without having the ability to generalize very well.
Thane Ruthenis describes it here:
https://www.lesswrong.com/posts/RDG2dbg6cLNyo6MYT/thane-ruthenis-s-shortform#vxHcdFb3KLWEQbmR
To be clear, for the purposes of coherence theorems, behavior is enough, so this is enough evidence to say that LLMs are at least reasonably coherent towards expected utility maximization, and suggests coherence theorems have real-life consequences.
But I do think it matters a little whether it’s internally represented for real-life AI, because of generalization issues.