there is no publicly-known technique within the current paradigm of training LLMs that we have good reasons to believe instills preferences over environmental latents (the ground truth) rather than sense data (proxies), let alone any specific latents of our choosing.
Three pieces of evidence that lead me to think that the AI that makes me extinct will primarily optimize the universe for molecular squiggles rather than the appearance of molecular squiggles.
The Natural Abstractions thesis argues that there are simple, natural abstractions in the sense data, that these natural abstractions are almost all about ground truth, not sense data, and that intelligences will tend to have preferences over these simple, natural abstractions.
Evolved intelligences primarily optimize for ground truth, not sense data.
If we ask artificial intelligences what they prefer, they describe ground truth preferences.
The apparent-success-seeking thesis is counter-evidence. But this also shows that we have techniques to influence preferences to be more about ground truth. Prior models would more often seek a green test result by deleting all the tests, for example. This is a sense data preference for passing tests. Current AIs do this less often. Maybe model-makers are only moving sense data preferences around, but that only makes sense if there’s a systematic bias towards sense data preferences, and I don’t have a reason for this to be true.
Tbc I don’t have a confident take on whether or not current LLMs, or the superintelligences that we end up with later, have preferences that point to environmental latents vs. sense data. Re: future superintelligences I lean towards environmental latents. My claims are that 1) we don’t know what’s in there right now, and 2) we don’t have any reliable steering mechanism for what goes in there at all.
Your claims are reasonable. Your (1) seems like Yudkowsky’s (25): “We’ve got no idea what’s actually going on inside the giant inscrutable matrices and tensors of floating-point numbers”. Your (2) seems like Yudkowsky’s (17): “on the current optimization paradigm there is no general idea of how to get particular inner properties into a system”. I don’t disagree.
The claim in Yudkowsky’s (19) is that there is an additional theoretical difficulty of getting ground truth preferences instead of sense data preferences, with no known way to solve. That is false. It looks like the Natural Abstraction project intro was written in April 2021, and List of Lethalities was June 2022. So it was false (but not proven false) in 2022.
Aside: with respect to reward inputs, it’s less clear. See 2025-Era “Reward Hacking” Does Not Show that Reward Is the Optimization Target. Also, there are preferences that don’t fit neatly into the distinction Yudkowsky drew. For example, the LLM may prefer certain feelings and beliefs, which aren’t sense data, reward inputs, or ground truth. So I don’t say that (19) is fully disproven, only that it is disproven with respect to sense data.
My claims:
By default, powerful LLMs don’t care much about sensor inputs, relative to other preferences.
To the (unknown) extent that we can align LLMs, there’s nothing special about “webcam input” versus “creatures outside the webcam”.
Three pieces of evidence that lead me to think that the AI that makes me extinct will primarily optimize the universe for molecular squiggles rather than the appearance of molecular squiggles.
The Natural Abstractions thesis argues that there are simple, natural abstractions in the sense data, that these natural abstractions are almost all about ground truth, not sense data, and that intelligences will tend to have preferences over these simple, natural abstractions.
Evolved intelligences primarily optimize for ground truth, not sense data.
If we ask artificial intelligences what they prefer, they describe ground truth preferences.
The apparent-success-seeking thesis is counter-evidence. But this also shows that we have techniques to influence preferences to be more about ground truth. Prior models would more often seek a green test result by deleting all the tests, for example. This is a sense data preference for passing tests. Current AIs do this less often. Maybe model-makers are only moving sense data preferences around, but that only makes sense if there’s a systematic bias towards sense data preferences, and I don’t have a reason for this to be true.
Tbc I don’t have a confident take on whether or not current LLMs, or the superintelligences that we end up with later, have preferences that point to environmental latents vs. sense data. Re: future superintelligences I lean towards environmental latents. My claims are that 1) we don’t know what’s in there right now, and 2) we don’t have any reliable steering mechanism for what goes in there at all.
Your claims are reasonable. Your (1) seems like Yudkowsky’s (25): “We’ve got no idea what’s actually going on inside the giant inscrutable matrices and tensors of floating-point numbers”. Your (2) seems like Yudkowsky’s (17): “on the current optimization paradigm there is no general idea of how to get particular inner properties into a system”. I don’t disagree.
The claim in Yudkowsky’s (19) is that there is an additional theoretical difficulty of getting ground truth preferences instead of sense data preferences, with no known way to solve. That is false. It looks like the Natural Abstraction project intro was written in April 2021, and List of Lethalities was June 2022. So it was false (but not proven false) in 2022.
Aside: with respect to reward inputs, it’s less clear. See 2025-Era “Reward Hacking” Does Not Show that Reward Is the Optimization Target. Also, there are preferences that don’t fit neatly into the distinction Yudkowsky drew. For example, the LLM may prefer certain feelings and beliefs, which aren’t sense data, reward inputs, or ground truth. So I don’t say that (19) is fully disproven, only that it is disproven with respect to sense data.
My claims:
By default, powerful LLMs don’t care much about sensor inputs, relative to other preferences.
To the (unknown) extent that we can align LLMs, there’s nothing special about “webcam input” versus “creatures outside the webcam”.