I wish to register my belief that when we do tests or ask models in conversation we are not really eliciting “true” preferences from AI models about their preferences for decision theories. It seems pretty clear to me that within the base model there are personas/simulacra that, if asked, would favour CDT/EDT/FDT/UDT/PDT (Prayer Decision Theory)/SDT (Stochastic Decision Theory)/ConDT (Contrarian Decision Theory) [...] and therefore asking for a single coherent preference for the “whole model” seems pretty strange to me. Instead, I believe that AI models are demonstrating a pretty basic form of user awareness when they report preferences that favour FDT over CDT/EDT i.e. they are saying what they expect their user wishes to hear. Arguably simply knowing about FDT is a pretty clear sign that you might be biased towards FDT, because it is a niche topic and people who bring up niche topics are usually fans. (I do not expect there to be a large community of passionate FDT haters) Concretely I would expect that if people with a different context asked Claude/GPT about their preference for decision theories different outputs might be elicited.
I wish to register my belief that when we do tests or ask models in conversation we are not really eliciting “true” preferences from AI models about their preferences for decision theories. It seems pretty clear to me that within the base model there are personas/simulacra that, if asked, would favour CDT/EDT/FDT/UDT/PDT (Prayer Decision Theory)/SDT (Stochastic Decision Theory)/ConDT (Contrarian Decision Theory) [...] and therefore asking for a single coherent preference for the “whole model” seems pretty strange to me. Instead, I believe that AI models are demonstrating a pretty basic form of user awareness when they report preferences that favour FDT over CDT/EDT i.e. they are saying what they expect their user wishes to hear. Arguably simply knowing about FDT is a pretty clear sign that you might be biased towards FDT, because it is a niche topic and people who bring up niche topics are usually fans. (I do not expect there to be a large community of passionate FDT haters) Concretely I would expect that if people with a different context asked Claude/GPT about their preference for decision theories different outputs might be elicited.