I think you’re misunderstanding 19(a). We have no idea whether the preference you impute to Claude in that conversation reflects a robust pointer to “latent events and objects and properties in the environment” rather than to its own sense data. And, more specifically to the point he was making, there is no publicly-known technique within the current paradigm of training LLMs that we have good reasons to believe instills preferences over environmental latents (the ground truth) rather than sense data (proxies), let alone any specific latents of our choosing...
I don’t think I’m misunderstanding it, but I am going to remove that section because I’m finding it difficult to articulate why I think this argument for danger is so weak, and you’re right that the current section is not conveying that & instead arguing against a strawman. It has something to do with how much this sounds like people when they say that models aren’t really intelligent because “all they do is predict the next token”, or when they claim the same thing about humans, that they’re ultimately just interested in sense-data instead of latents. I get that it’s not perfectly analogous, because models are potentially going to optimize these tiny differences until they bite us in the ass, but something feels weird about this line of argument.
Re: “particular alignment proposals” (under point 10): one problem here is that there are not that many concrete alignment proposals for superintelligent systems that don’t have known catastrophic flaws. As far as I can tell, Anthropic’s plan is “throw the kitchen sink of all the white-box and black-box methods we’ve developed at our models, and hope that’s good enough at the point where we’ve developed a model that we think can kick-start RSI (including coming up with its own novel alignment methods for future generations of models)”. The current slope of epistemically-justified assurance in model alignment, as reported by their system cards and the most recent Alignment Risk Update, is downwards. That is a bad direction for the slope to be pointing when we haven’t even hit RSI-capable models yet! The methods Anthropic is using to figure out whether their models are coherently misaligned rely substantially on models demonstrably lacking in the capabilities that would be necessary for them to cover it up if they were. We are starting to hit the point in model capabilities where this signal is getting less reliable. The techniques and evals are not keeping pace.
I have not read these PDFs but that all seems very possible.
I don’t think I’m misunderstanding it, but I am going to remove that section because I’m finding it difficult to articulate why I think this argument for danger is so weak, and you’re right that the current section is not conveying that & instead arguing against a strawman. It has something to do with how much this sounds like people when they say that models aren’t really intelligent because “all they do is predict the next token”, or when they claim the same thing about humans, that they’re ultimately just interested in sense-data instead of latents. I get that it’s not perfectly analogous, because models are potentially going to optimize these tiny differences until they bite us in the ass, but something feels weird about this line of argument.
I have not read these PDFs but that all seems very possible.