Naively, we can imagine a model with a good capabilities engine and the wrong objective (which could be a complex mix of stuff or whatever); unless it is in a situation where randomization is at least as good as just doing the optimal thing, we expect it not to randomize bc its capabilities engine knows what the optimal thing is. So failures will generally be consistent, so it will have high “coherence”.
Another point to make in the opposite direction: randomization is often the optimal thing, so doesn’t this “coherence” definition mean that all optimal game players playing a non-pure strategy, like Nash equilibria (or many exploitative strategies), are defined to be heavily “incoherent” no matter how well they play the Nash? Because they will play different strategies on different games and so there will be a high fraction of variance attributable to randomness. It is unfortunate if you have defined away as “noise” all powerful superintelligent behavior on many economically valuable and dangerous environments.
(And what about any kind of search or exploration or novelty and avoiding mode-collapse...? You can do different things in different episodes and that may not be ‘random’ at all in any kind of ‘meaningless’ sense, but highly structured and optimal. When I use a LLM for my creative writing, I regard a lack of variance as a serious problem and a ‘perfectly coherent’ LLM would be largely useless to me!)
Another point to make in the opposite direction: randomization is often the optimal thing, so doesn’t this “coherence” definition mean that all optimal game players playing a non-pure strategy, like Nash equilibria (or many exploitative strategies), are defined to be heavily “incoherent” no matter how well they play the Nash? Because they will play different strategies on different games and so there will be a high fraction of variance attributable to randomness. It is unfortunate if you have defined away as “noise” all powerful superintelligent behavior on many economically valuable and dangerous environments.
(And what about any kind of search or exploration or novelty and avoiding mode-collapse...? You can do different things in different episodes and that may not be ‘random’ at all in any kind of ‘meaningless’ sense, but highly structured and optimal. When I use a LLM for my creative writing, I regard a lack of variance as a serious problem and a ‘perfectly coherent’ LLM would be largely useless to me!)