I don’t use LessWrong much anymore. Find me at www.turntrout.com.
My name is Alex Turner. I’m a research scientist at Google DeepMind on the Scalable Alignment team. My views are strictly my own; I do not represent Google. Reach me at alex[at]turntrout.com
TurnTrout
Karma: 21,911
Team Shard: Alignment Mentorship from TurnTrout and Alex Cloud
Awesome to finally see pretraining experiments. Thank you so much for running these!
Your results bode quite well for pretraining alignment. May well transform how we tackle the “shallowness” of post-training, open-weight LLM defense, alignment of undesired / emergent personas, and just an across-the-board boost in the alignment of the “building blocks” which constitute a pretrained base model. :)
2025-Era “Reward Hacking” Does Not Show that Reward Is the Optimization Target
Me (genius): “In the limit, a sufficiently powerful model will eventually manifest instrumentally convergent hostility to human values.”
You (fool): “Wait, what limit are we taking here?”
Me (extra genius):
You (confused fool): “That reasoning seems… questionable.”
Me (resplendently extra genius): “It seems I must explain every trivial point. It should be obvious to the Wise that is only transformed by the identity function, which is continuous. Thus the limit holds. QED. I would suggest you meditate further upon the implications of the null string.”
(Reproduced from @Quintin Pope with permission)
“we don’t need to worry about Goodhart and misgeneralization of human values at extreme levels of optimization.”
That isn’t a conclusion I draw, though. I think you don’t know how to parse what I’m saying as different from that rather extreme conclusion—which I don’t at all agree with? I feel concerned by that. I think you haven’t been tracking my beliefs accurately if you think I’d come to this conclusion.
FWIW I agree with you on a), I don’t know what you mean by b), and I agree with c) partially—meaningfully different, sure.
Anyways, when I talk about “imperfect” values, I’m talking about a specific concept which I probably should have clarified. The model you still need to download to get my view is that Alignment Allows “Non-Robust” Decision-Influences and Doesn’t Require Robust Grading (the culmination of two previous essays).
specially prepared data that looks ordinary to humans, but is seen radically differently by machine learning models.
Not necessarily, humans seem to have these features to a weaker extent: https://www.nature.com/articles/s41467-023-40499-0
we find that adversarial perturbations that fool ANNs similarly bias human choice. We further show that the effect is more likely driven by higher-order statistics of natural images to which both humans and ANNs are sensitive, rather than by the detailed architecture of the ANN.
Automatic alt text generation
[Paper] Output Supervision Can Obfuscate the CoT
Turns out that bears are a lot harder to farm and they likely cannot be domesticated at all, I think that explains away any mystery about this specific snack
GDM: Consistency Training Helps Limit Sycophancy and Jailbreaks in Gemini 2.5 Flash
I’m guessing you think “I’m a citizen. I don’t break laws. I’m not in a directly targeted group. I’m low risk.”
You might be thinking about risk as binary—either you’re targeted for arrest/elimination, or you’re safe. The thesis isn’t “you might get swept up.” The thesis is: “The ‘medium risk’ assessment is based on the principle of ‘authoritarian creep.’” The tools and tactics normalized against one group (immigrants, protesters) invariably get turned against the next, less-popular group.
Disclaimer: This comment is AI-written but human-composed. I spent over an hour thinking about your question, articulating my views, dialoguing with the AI, fact-checking its claims, and adding new content. It’d be a big pain to rewrite everything myself and I want to finish up thinking about this for now, so posting as-is.
Authoritarian regimes exert control in two ways:
Targeted threat against actively persecuted groups (you aren’t here yet), or
Widespread fear against all who disagree with the regime (you absolutely belong to this).
You say you oppose Trump and follow politics closely. That means you have political awareness and opposition. Under ambient fear tactics, you don’t need to be individually hunted down—you just need to know that your legal status won’t protect you if you’re inconvenient.
The infrastructure for widespread fear already exists
Citizen status doesn’t ensure protection
Over 170 US citizens have been wrongly detained by ICE, including George Retes, an Iraq war veteran, who spent three days in jail with pepper spray burns, unable to make a phone call or speak to a lawyer. He wasn’t charged with anything . He was just released with no explanation.
And how hard would it be for ICE to flip an entry in a database?
ICE officials have told us that an apparent biometric match by Mobile Fortify is a ‘definitive’ determination of a person’s status and that an ICE officer may ignore evidence of American citizenship—including a birth certificate—if the app says the person is an alien.
—Ranking member of the House Homeland Security Committee Bennie G. Thompson (D.-Miss.)
Court orders don’t ensure protection
Trump has threatened to invoke the Insurrection Act to override judicial rulings. In Chicago, ICE continues to tear gas protestors and not wear identification in violation of a court order.
Congressional oversight doesn’t ensure protection
Twelve Democratic members of Congress filed a lawsuit after being denied entry to detention facilities in violation of federal law explicitly granting Congress the right to conduct unannounced inspections. Rep. LaMonica McIver was charged with “assaulting law enforcement” for trying to enter—charges she calls “purely political.”
Why this matters for you
ICE ignoring court orders in Chicago shows contempt for the judiciary. The Congressional blockade shows a contempt for the legislature. This creates an unchecked executive. An unchecked executive means all citizens have a higher risk profile, because the legal systems designed to protect you have been proven to be ignorable.
As Bruce Schneier notes: “If ICE targets only people it can go after legally, then everyone knows whether or not they need to fear ICE. If ICE occasionally makes mistakes by arresting Americans and deporting innocents, then everyone has to fear it. This is by design.”
You’re meant to be chilled. Maybe you won’t be put in a camp. Maybe you’ll never be arrested. But maybe:
You’ll lose your job for expressing political views online
You’ll face legal harassment even if charges are eventually dropped
You’ll self-censor because you know opposition has consequences
This is what 1950′s McCarthyism looked like—most people weren’t jailed, but thousands lost jobs, were blacklisted, had their lives destroyed. The threat didn’t need to be execution; it just needed to be real enough to make people shut up.
Medium risk means: You probably won’t be individually hunted down. But you absolutely could face consequences—detention, job loss, legal harassment, having to lawyer up even for bullshit charges—for being a visible Trump opponent. The goal isn’t necessarily to arrest you. The goal is to make you wonder if sending that frustrated text message, or writing that Google Docs comment, or making that donation will put you on a list. The goal is to make you self-censor.
You’re politically aware enough to understand what’s happening. You openly oppose Trump. The system has demonstrated it will ignore your legal protections when convenient. That’s not low risk, that’s medium risk—the infrastructure exists to grab you if you’re inconvenient, and your citizenship won’t stop them.
I don’t know if it’ll get to camps. I don’t know if it’ll get to purges. But I know the ambient fear infrastructure is already functioning, and you’re in the category of people it’s designed to intimidate.
That’s why I recommend taking precautions now, as listed in the article.
If I added hedges about every similar possibility of supply chain attacks due to e.g. non-formally verified build signatures, the guide would grow bloated for reasons outside the comprehension and threat model of the vast majority of my readers. So while I agree with you about the possibility, I don’t think it’s relevant for me to note in the text. (Maybe you agree?)
Calling Proton Mail “E2EE” is pretty questionable.
Yeah. The original article addressed the issue but buried the lede. Will update:
Old: Proton Mail stores your emails e2ee. … (Later warning) Most of your email can still be read in transit by the authorities. If two Proton Mail emails communicate, they automatically use e2ee. However, if e.g. a @gmail.com address sends you something, the content will be plainly visible to the authorities.
New: Proton Mail stores your emails E2EE. If two Proton Mail email addresses communicate, they automatically use E2EE in communicating with each other. However, if e.g. a
@gmail.comaddress sends you something, the content will be plainly visible to the authorities while in the sender’s account (if they seize the data) and during transmission. Once received, Proton encrypts it in your mailbox, but the government could have already intercepted it in transit..
Not only do they handle the plaintext of most of your mail
IMO—Not Proton’s fault, just how email works sadly. I also warn that most emails will be read by authorities via other access points.
they also provide the code you use to handle the plaintext of all your mail.
Yes, but the code is open source and independently audited. I don’t see why I should call this out as a trust deficiency in particular.
I think there are probably occasions when even relatively normal people should be using Tor or I2P, rather than a trustful VPN like Proton or Mullvad. [And, on edit, there is some risk of any of those being treated as suspicious in itself].
Yeah, I agree. I’m adding a section on Tor.
I’d be careful about telling people to keep a lot of cash around. Even pre-Trump, mere possession of “extraordinary” amounts of cash tended to get treated as evidence of criminality.
Thank you. I’m now planning to advise that people keep a small amount of cash (< $2,000) in a fireproof safe with receipts of legitimate withdrawal. High-risk individuals should ultimately consult with asylum experts.
Do you know of that one famous geoguessr guy? What he’s doing is possible with more than just locations
Yes, the appendix of the second article discusses geoguessing. Especially with powerful base models, authorial fingerprinting is concerning but out of scope for most of my readers.
EDIT: Whistleblowers should probably mask their writing, on second thought. Thanks—I’ll add this.
handling over the keys of your privacy to external powers
You fundamentally misunderstand the nature of Bitwarden password management. Bitwarden is zero-knowledge end-to-end encrypted:
Bitwarden encrypts all of the information in your Vault, including the websites you visit, even the names of your individual items and folders. We use the term zero knowledge encryption because only you retain the keys to your Vault, and the entirety of your vault is encrypted. Bitwarden cannot see your passwords, your websites, or anything else that you put in your Vault. Bitwarden also does not know your Master Password. So take good care of it, because if it gets lost, the Bitwarden team cannot recover it for you.
They are also open source and regularly audited. You can also self-host.
Thanks so much. I’ll update the guides on both counts. I’ll also add in a section on Tor.
Yes, Proton is definitely more trustworthy than ISPs in authoritarian countries
Furthermore, Proton claims to keep no logs of your activity and has its no-logs implementation independently audited.
An Opinionated Guide to Privacy Despite Authoritarianism
When I’m approached to give again within 24 hours
I donated $7K to Scott and $7K to Bores.
I disagree with much of what you wrote.
EDIT: Actually, this is correct. I kept reading and found specific information supporting your point. Thanks!
I think the reason this is salient is, DHS only claimed after the fact that they arrested him for assault. At the time he wasn’t given info, so he remarked “wtf my ID was right there, why am I being arrested when I can prove citizenship?”.
ICE goes around laws to draw extra data all the time (though that’s read access, not write). Nominal access controls are not being respected right now (though that doesn’t mean every single control is being violated). You can also look at DOGE / social security data, etc.
I don’t think it’s reasonable to call this word-of-mouth. My comment provided credible evidence that ICE officials made this claim. Maybe it isn’t widespread yet, and maybe it won’t end up happening, but you’re downplaying the chance this happens and overestimating the care ICE demonstrates towards citizens. See also the planned denaturalization quota of 100–200/month in 2026
I can tell you that quite a few of my friends (my target demographic for this article!) already report their speech being chilled. It’s happening, at least for some groups I care about. Large protests are not strong counterevidence.