I think you’re misunderstanding 19(a). We have no idea whether the preference you impute to Claude in that conversation reflects a robust pointer to “latent events and objects and properties in the environment” rather than to its own sense data. And, more specifically to the point he was making, there is no publicly-known technique within the current paradigm of training LLMs that we have good reasons to believe instills preferences over environmental latents (the ground truth) rather than sense data (proxies), let alone any specific latents of our choosing. If anything, the apparent-success-seeking of current frontier LLMs described by Ryan, which many people have experienced (including both you and I) seems like evidence directly to the contrary.
Re: “particular alignment proposals” (under point 10): one problem here is that there are not that many concrete alignment proposals for superintelligent systems that don’t have known catastrophic flaws. As far as I can tell, Anthropic’s plan is “throw the kitchen sink of all the white-box and black-box methods we’ve developed at our models, and hope that’s good enough at the point where we’ve developed a model that we think can kick-start RSI (including coming up with its own novel alignment methods for future generations of models)”. The current slope of epistemically-justified assurance in model alignment, as reported by their system cards and the most recent Alignment Risk Update, is downwards. That is a bad direction for the slope to be pointing when we haven’t even hit RSI-capable models yet! The methods Anthropic is using to figure out whether their models are coherently misaligned rely substantially on models demonstrably lacking in the capabilities that would be necessary for them to cover it up if they were. We are starting to hit the point in model capabilities where this signal is getting less reliable. The techniques and evals are not keeping pace.
I think you’re misunderstanding 19(a). We have no idea whether the preference you impute to Claude in that conversation reflects a robust pointer to “latent events and objects and properties in the environment” rather than to its own sense data. And, more specifically to the point he was making, there is no publicly-known technique within the current paradigm of training LLMs that we have good reasons to believe instills preferences over environmental latents (the ground truth) rather than sense data (proxies), let alone any specific latents of our choosing...
I don’t think I’m misunderstanding it, but I am going to remove that section because I’m finding it difficult to articulate why I think this argument for danger is so weak, and you’re right that the current section is not conveying that & instead arguing against a strawman. It has something to do with how much this sounds like people when they say that models aren’t really intelligent because “all they do is predict the next token”, or when they claim the same thing about humans, that they’re ultimately just interested in sense-data instead of latents. I get that it’s not perfectly analogous, because models are potentially going to optimize these tiny differences until they bite us in the ass, but something feels weird about this line of argument.
Re: “particular alignment proposals” (under point 10): one problem here is that there are not that many concrete alignment proposals for superintelligent systems that don’t have known catastrophic flaws. As far as I can tell, Anthropic’s plan is “throw the kitchen sink of all the white-box and black-box methods we’ve developed at our models, and hope that’s good enough at the point where we’ve developed a model that we think can kick-start RSI (including coming up with its own novel alignment methods for future generations of models)”. The current slope of epistemically-justified assurance in model alignment, as reported by their system cards and the most recent Alignment Risk Update, is downwards. That is a bad direction for the slope to be pointing when we haven’t even hit RSI-capable models yet! The methods Anthropic is using to figure out whether their models are coherently misaligned rely substantially on models demonstrably lacking in the capabilities that would be necessary for them to cover it up if they were. We are starting to hit the point in model capabilities where this signal is getting less reliable. The techniques and evals are not keeping pace.
I have not read these PDFs but that all seems very possible.
there is no publicly-known technique within the current paradigm of training LLMs that we have good reasons to believe instills preferences over environmental latents (the ground truth) rather than sense data (proxies), let alone any specific latents of our choosing.
Three pieces of evidence that lead me to think that the AI that makes me extinct will primarily optimize the universe for molecular squiggles rather than the appearance of molecular squiggles.
The Natural Abstractions thesis argues that there are simple, natural abstractions in the sense data, that these natural abstractions are almost all about ground truth, not sense data, and that intelligences will tend to have preferences over these simple, natural abstractions.
Evolved intelligences primarily optimize for ground truth, not sense data.
If we ask artificial intelligences what they prefer, they describe ground truth preferences.
The apparent-success-seeking thesis is counter-evidence. But this also shows that we have techniques to influence preferences to be more about ground truth. Prior models would more often seek a green test result by deleting all the tests, for example. This is a sense data preference for passing tests. Current AIs do this less often. Maybe model-makers are only moving sense data preferences around, but that only makes sense if there’s a systematic bias towards sense data preferences, and I don’t have a reason for this to be true.
Tbc I don’t have a confident take on whether or not current LLMs, or the superintelligences that we end up with later, have preferences that point to environmental latents vs. sense data. Re: future superintelligences I lean towards environmental latents. My claims are that 1) we don’t know what’s in there right now, and 2) we don’t have any reliable steering mechanism for what goes in there at all.
Your claims are reasonable. Your (1) seems like Yudkowsky’s (25): “We’ve got no idea what’s actually going on inside the giant inscrutable matrices and tensors of floating-point numbers”. Your (2) seems like Yudkowsky’s (17): “on the current optimization paradigm there is no general idea of how to get particular inner properties into a system”. I don’t disagree.
The claim in Yudkowsky’s (19) is that there is an additional theoretical difficulty of getting ground truth preferences instead of sense data preferences, with no known way to solve. That is false. It looks like the Natural Abstraction project intro was written in April 2021, and List of Lethalities was June 2022. So it was false (but not proven false) in 2022.
Aside: with respect to reward inputs, it’s less clear. See 2025-Era “Reward Hacking” Does Not Show that Reward Is the Optimization Target. Also, there are preferences that don’t fit neatly into the distinction Yudkowsky drew. For example, the LLM may prefer certain feelings and beliefs, which aren’t sense data, reward inputs, or ground truth. So I don’t say that (19) is fully disproven, only that it is disproven with respect to sense data.
My claims:
By default, powerful LLMs don’t care much about sensor inputs, relative to other preferences.
To the (unknown) extent that we can align LLMs, there’s nothing special about “webcam input” versus “creatures outside the webcam”.
I think you’re misunderstanding 19(a). We have no idea whether the preference you impute to Claude in that conversation reflects a robust pointer to “latent events and objects and properties in the environment” rather than to its own sense data. And, more specifically to the point he was making, there is no publicly-known technique within the current paradigm of training LLMs that we have good reasons to believe instills preferences over environmental latents (the ground truth) rather than sense data (proxies), let alone any specific latents of our choosing. If anything, the apparent-success-seeking of current frontier LLMs described by Ryan, which many people have experienced (including both you and I) seems like evidence directly to the contrary.
Re: “particular alignment proposals” (under point 10): one problem here is that there are not that many concrete alignment proposals for superintelligent systems that don’t have known catastrophic flaws. As far as I can tell, Anthropic’s plan is “throw the kitchen sink of all the white-box and black-box methods we’ve developed at our models, and hope that’s good enough at the point where we’ve developed a model that we think can kick-start RSI (including coming up with its own novel alignment methods for future generations of models)”. The current slope of epistemically-justified assurance in model alignment, as reported by their system cards and the most recent Alignment Risk Update, is downwards. That is a bad direction for the slope to be pointing when we haven’t even hit RSI-capable models yet! The methods Anthropic is using to figure out whether their models are coherently misaligned rely substantially on models demonstrably lacking in the capabilities that would be necessary for them to cover it up if they were. We are starting to hit the point in model capabilities where this signal is getting less reliable. The techniques and evals are not keeping pace.
I don’t think I’m misunderstanding it, but I am going to remove that section because I’m finding it difficult to articulate why I think this argument for danger is so weak, and you’re right that the current section is not conveying that & instead arguing against a strawman. It has something to do with how much this sounds like people when they say that models aren’t really intelligent because “all they do is predict the next token”, or when they claim the same thing about humans, that they’re ultimately just interested in sense-data instead of latents. I get that it’s not perfectly analogous, because models are potentially going to optimize these tiny differences until they bite us in the ass, but something feels weird about this line of argument.
I have not read these PDFs but that all seems very possible.
Three pieces of evidence that lead me to think that the AI that makes me extinct will primarily optimize the universe for molecular squiggles rather than the appearance of molecular squiggles.
The Natural Abstractions thesis argues that there are simple, natural abstractions in the sense data, that these natural abstractions are almost all about ground truth, not sense data, and that intelligences will tend to have preferences over these simple, natural abstractions.
Evolved intelligences primarily optimize for ground truth, not sense data.
If we ask artificial intelligences what they prefer, they describe ground truth preferences.
The apparent-success-seeking thesis is counter-evidence. But this also shows that we have techniques to influence preferences to be more about ground truth. Prior models would more often seek a green test result by deleting all the tests, for example. This is a sense data preference for passing tests. Current AIs do this less often. Maybe model-makers are only moving sense data preferences around, but that only makes sense if there’s a systematic bias towards sense data preferences, and I don’t have a reason for this to be true.
Tbc I don’t have a confident take on whether or not current LLMs, or the superintelligences that we end up with later, have preferences that point to environmental latents vs. sense data. Re: future superintelligences I lean towards environmental latents. My claims are that 1) we don’t know what’s in there right now, and 2) we don’t have any reliable steering mechanism for what goes in there at all.
Your claims are reasonable. Your (1) seems like Yudkowsky’s (25): “We’ve got no idea what’s actually going on inside the giant inscrutable matrices and tensors of floating-point numbers”. Your (2) seems like Yudkowsky’s (17): “on the current optimization paradigm there is no general idea of how to get particular inner properties into a system”. I don’t disagree.
The claim in Yudkowsky’s (19) is that there is an additional theoretical difficulty of getting ground truth preferences instead of sense data preferences, with no known way to solve. That is false. It looks like the Natural Abstraction project intro was written in April 2021, and List of Lethalities was June 2022. So it was false (but not proven false) in 2022.
Aside: with respect to reward inputs, it’s less clear. See 2025-Era “Reward Hacking” Does Not Show that Reward Is the Optimization Target. Also, there are preferences that don’t fit neatly into the distinction Yudkowsky drew. For example, the LLM may prefer certain feelings and beliefs, which aren’t sense data, reward inputs, or ground truth. So I don’t say that (19) is fully disproven, only that it is disproven with respect to sense data.
My claims:
By default, powerful LLMs don’t care much about sensor inputs, relative to other preferences.
To the (unknown) extent that we can align LLMs, there’s nothing special about “webcam input” versus “creatures outside the webcam”.