Sam Altman and Dario Amodei are the public faces of AI safety and that’s poisoning the well
Normies just smell a rat whenever those two bring up AI safety. I think when the AI safety community gets a word in edge-wise (like you’re arguing with your friends or maybe a tweet goes viral), we usually say something like
“it could kill everyone. but China could also kill everyone, so now more than ever we have to supercharge OpenAI and Anthropic.”
and we just sound like we’re “one of them.”
I feel like I should be hearing “shut it down” about as often as I hear “broke containment” and that’s not really happening, instead it’s more like “let OpenAI cook.” So a number of people are pretty soured on AI safety because of this.
When have AI safety people said that OpenAI and Anthropic should be supercharged? Maybe you hang out with a very different set of AI safety people, because this has not been my experience at all.
A lot of AI safety people say things about Dario Amodei’s decision to move from a contract that forbids the military from allowing to use Claude to run disinformation campaigns to sway European elections to offering one that allows the military to do so as Dario standing up for his principles.
Given that both the first Trump and Biden administration ran their antivax disinformation campaign in the Philippines, thinking that no harm will come from allowing the present Trump administration to run Claude supported disinformation campaigns is bad.
Anthropic did suffer consequences for not rolling over on autonomous killing decisions and domestic surveillance but the decision was still giving up a good portion of the principles that are actually important.
I think this concretely comes up with “do you support a datacenter moratorium?” If you say no, you’ve taken a confusing stance with respect to the “it might kill everyone” stance.
I’ll say you have a point, I can revise like this: the public does sometimes get exposure to a genuine AIS x-risk argument, but they reject it because they mostly hear about x-risk from AI CEOs who conspicuously don’t say “shut it down.” (Which is not the same as “the AIS person said don’t shut it down.”)
And not everyone rejects it, and the Overton window is probably moving to include mass unemployment and x-risk.
You’re painting a very binary, “us vs. them” picture of the option space here. There are many policy positions one might take between the extremes “datacenter moratorium” and “cheering the leading labs”; rejecting the idea of a moratorium does not mean you want the labs to be supercharged.
For some examples of such policies, see e.g. the Radical Optionality suggestions.
Consensus processes have to reach agreement among a lot of people, some of whom are in different belief states. It’s not surprising to me that we’d see GPT2 level reasoning like “if you don’t vocally support a datacenter ban, you don’t believe in xrisk”, which I read as being because to the weak model a crowd behaves as, datacenter ban is a lesser version of banning it all, and therefore if you support banning it all, you support a datacenter ban. The shallow model the averaged crowd behaves as, in the part of the individual’s brains running the Keynesian beauty contest, sees a low dimensional subspace where there’s a success policy vertex at the extrema “ban it all” and the consensus process is to push towards that with lesser policies incrementally.
So then someone says, for example, “banning new datacenters is irrelevant because we need to shut down all computers capable of running a takeoff at all”, and it sounds like not wanting to hike through the intermediate policies to get the one you want.
I make no claim any of this is the slightest bit reasonable. I am reading off my intuitive model of crowds.
I think the simple belief “it might kill everyone ⇒ shut it down” is a pretty good belief, and people rightly resist attempts to move them off it, and they are rightly skeptical of people who believe the first but not the second.
The crowd effect has more to do with the outgroup and less to do with matching what one’s ingroup thinks. If a stranger say “x-risk ⇒ build data centers”, they are not showing ingroup harmony, so they get evaluated for outgroup, and the answer is something like “I can easily see Sam and Dario lying about believing in x-risk as part of building hype, this stranger is a tech bro.” I wish those two were not the most visible examples of people who believe in x-risk.
I was thinking of nate soares or so in my previous comment—people who say things like “sure, whatever, don’t build datacenters, but that won’t really help very much”. If you’re really just thinking of sam/dario types, then yeah agreed that their positions are barely consistent in the terrain. I do think if they can end up right, then everything will be fine. I am as worried as I am because I think their chance of ending up right—“build the safe superintelligence first”—is extremely slim.
According to many thinkpieces: people are reading less, students are cheating more, professors are angry, English professors have to treat their class like it’s remedial, high school teachers are crying students can’t write a full paragraph.
According to the chess community: everyone is better, across all levels. GMs train against computers, beginners have multimedia learning resources and can have their games auto-critiqued, intermediates get exposed to chess opening theory sooner, and everyone is playing more games with all sorts of people.
The “chess” experience is pretty techno-optimist, and since techno-pessimism is in vogue, you have to almost pay attention more carefully to notice the pocket of techno-optimism.
Has anyone tried squaring these two viewpoints? They seem pretty directionally opposed.
A simple hypothesis to check would be “things people do of their own free volition are getting better; other things are getting worse”. Because for other things, you want to just cheat / take the easy route / whatever. (Probably “of their own free volition” is not the right category, curious what would be the right category.)
AI provides ample opportunities to both learn new things, and avoid learning anything. Which one would one choose? Depends on whether the person in question wants to learn, or wants to reach some other destination to which “learning” is just the expected path. A destination like “get a degree” or “get a job”.
The chess community has not made a decision that every person in the country ought to be a chessmaster, and has not attempted to lower its standards until anyone who can fog a mirror is called a GM. If they had, I think you would see similar issues there.
This is a continuation of a trendline that predates LLMs I think. Intellectuals have been becoming less literate for decades. At the same time, we’re getting generally better at stuff, especially stuff that’s easy to measure.
Why intellectuals (at least) becoming less literate? My guesses: Maybe because social media captures our attention, so we read less. Maybe because literacy is ephemeral and hard to measure, and our current society in general optimizes things that are easy to measure. Written exams are subjective compared to Scantrons, so writing skill is deprioritized too. The collapse of the expert class means nobody is in a good position to judge good writing from bad.
Professors have generally been a lot worse at actually teaching their students useful things compared and used their power to coerce students into doing useless tasks (teaching them bad writing habit as Larry McEnerney explains). On the other hand, there’s a lot more pressure on chess teaching to actually result in increased in better chess skill.
When the power balance shifted students can make decisions about how to spent their time better instead of being coerced. Professors now need to actually convince students that what they are advocating is useful for students.
Do you agree with the conceit that literacy is down? I want to know if your claim that “it’s the professors who are bad” substitutes or complements poor literacy.
Professors are bad is the general argument that most of what university does is credentialing and not skill building.
As far as literacy goes, engaging with short video content like Tiktok or instagram reels instead of reading long-term texts is probably bad but has nothing to do with AI. If anything, using AI properly forces you to engage with the text that the AI gives you and trains reading ability.
As far as the numbers go https://www.nationsreportcard.gov/ltt/2025 for example suggest that reading abilities for 9 year-olds increased in the time AI was available but it did decrease in prior years. For 13-year-olds it was flat. Bad lockdown policy might be one of the reasons in addition to short-form video.
Concerns whether children are taught reading well enough are always there, there are great Yes, Minister sketches about it. Moving to evidence-based primary school education is probably a good thing but has little to do with the topic of how AI effects college students.
Has anyone tried squaring these two viewpoints? They seem pretty directionally opposed.
It seems straightforward to me. If you want to learn, there are better tools for learning than ever in human history. If you don’t care about learning, there are easier and lower effort alternatives than ever in human history.
at any given moment pre-singularity, AI tech (like any tech, like any change whatsoever) will empower some groups and disempower others. this will change over time until … (to be unfashionably techno-optimistic) we each of us are maximally empowered to the extent that it doesn’t infringe upon the empowerment of others? or at least optimally empowered, whatever that happens to look like
AI companies won’t break the cookie fairly. If they think you care about x-risk more, then you’ll foot the bill for AI safety, not the companies.
To that end, the main question is: what are the labs’ “greedy corporations” wishlist? I think we need to keep an eye on this because it may explain their demands better than any intention toward AI safety does.
The U.S. government should enforce IP protections across the Pacific. Literally a decades old demand from the private sector. Right now it’s coming up via “distillation” and “misuse” and then AIS cooperation.
Kill American competition or open models somehow. Usually this is done by cleverly creating a moat. For instance if a model operator is liable for that model’s unaligned behaviors, that means host-your-own open model business might die off. Or anyone trying to catch up to the frontier just has high expense and red tape.
Continue signaling capabilities to users and industry: the bio weapon news happens to market to pharma, for instance. “Misuse” news is always at the cusp of your capability! so you’re capitalizing misusers into marketing stories for stuff you haven’t figured out how to do yet.
Continue the data center push despite political headwinds.
Everyone needs Mythos+ for cyber defense on every server. Regulation may or may not be needed to push for this.
Expand into surveillance business. Because bio-terrorists.
I have a bigger post-draft about this but just actually saying what they want seemed to be the main thing. There’s some “meta” arguing that we should care about this frame at all even if we’re AIS focused.
Please be aware the left has an anti- AI regulation faction as well.
Article: “Congressional Democrats in Hysterics About Saving the World From AI | Why panic when you could just use existing laws?”
I lean toward thinking these authors (Whitney Whimish and David Dayen) are closer to an ally, but it could go either way. If you argue with them, try to not unnecessarily crux on doom | not-doom, and try to not crux on federal-regulation | not. Seems like even if they’re wrong that federal reg is unneeded, they’re. right there are a lot of local and state power levers, so go ahead and explore that.
Love them or hate them, but, the left is very focused on power analyses of corporations. This is brainpower and organizing skill that would be a shame to not partner with.
Double relatedly, I’m interested in doing some AI news aggregation, seeking a partner if noticing these types of movements is interesting to you. You know how Zvi does model card roundups, I want to do “public sentiment” roundups.
At this point I feel the biggest problem is polarization. The more anti-AI the left is, the more pro-AI Trump and other conservatives will be. And Trump holds the actual power over AI in the executive, even if both houses turn over.
The public opinion situation has already spun out of our control. But getting dems more fired up looks paradoxically actually bad to me now.
Coming to grips with this strange situation and what we can do in messaging should be our first priority I think.
It’s really not the case that polarization means Trump just does the opposite of what the left wants. Trump does what what white guys in the Midwest want. When the left changes their mind about something, for example with the ultra-partisan ICE protests in MN this year, Trump changes his policy too. But mostly, the left is unappealing and insufferable to white guys in the Midwest, and they mistake that for Trump doing the opposite of what the left wants.
A bunch of stuff has changed recently:
Democrats are trying out having appealing messaging. They’re also letting a young attractive man be a leader, which interferes with Trump’s young man to MAGA pipeline.
The left doesn’t take Trump’s bait so much. For instance, Trump attacked Effective Altruism as part of his anti-AI thing, and nobody cared because nobody really wanted to defend Effective Altruism. Just in general I think he’s becoming old news, like I can’t even think of any dumb nickname he’s made up recently.
Trump has lost his touch, which is arguably bad for making AI popular then him doing the popular thing, but it’s a new situation anyway. Data centers, gas prices, no new wars, angering farmers, and now his approval is down to 37%. More Republicans are breaking from him, and again it’s not like he just ignores what Fox News, his advisors or Congress advise him. His base might even be in disarray by now, I’m not sure.
I think our best strategy is getting either party to adopt a strong, broadly appealing AI platform. It’s a significantly bipartisan issue, this isn’t impossible. The crux is do we get “transgender prisoners” Kamala or do we get “affordability” Zohran.
I’m surprised you’re disagreeing. I’d say Trump does what Trump wants, and that very much includes doing whatever pisses off the left the most.
Trump does what white guys in the midwest want? It seems to me just that Trump pretty much is a white guy from the midwest by attitude, and he does what he wants. Which includes titilating followers and irritating opponents.
I do hope pressure from his base will turn him around on AI. I think it’s realistic to hope for him at least to make empty promises that it won’t be allowed to take jobs. Those promises could turn real over time. And Trump is nothing if not unpredictable.
But you’re not even touching the central point: it’s not just Trump who does whatever the other party doesn’t want: it’s everyone. Polarization is rampant.
You’re really not worried about how climate change was locked in gridlock for decades through polarization? And, do you really think the democrats can get meaningful slowdown, quickly enough? Please share your vision, because I’m not seeing it yet.
I think that The 2028 presidential primaries could be crucial for AI outcomes and I hope to see both parties run very anti-AI candidates. Having the dems fired up with no ability to actually do anything is going to make it much less likely that the Reps have a viable anti-AI candidate, because their sheer irritation with the dems will make Motivated reasoning and the horns/halo biases change their minds to like AI more.
Having the next president be pro-AI (not risk-conscious) seems like the worst possible avoidable thing. If timelines are really short, having Trump preside over the runup to takeover-capable AI could already be the worst possible thing, but not one we can avoid except through voluntary slowdown. There’s no way we get a bipartisan majority large enough to override his veto and no way he’ll back down to congress. But the president can halt AI progress easily with various executive orders based on fairly strong precedents, or none at all since Congress and the courts usually backs the prez when he claims an emergency.
Polarization means something like: left and right hate each other, sort into separate parties, the parties gridlock in the legislature, and every issue gets tainted as one’s side issue or the other’s.
I don’t know the rationalist lingo for this but I just find myself very suspicious of this concept. It’s very map, not territory, and it doesn’t have as much explanatory power as it looks. Like if you say “but that won’t work because polarization” I think I get to say “what actual mechanic are you talking about?” We should be trying to get inside this concept and see how it interacts with AI as an issue, not try to wholesale route our way around it.
I think the polarization model lets people be lazy and use it as an easy out. If you have a bill like “all children must learn Chinese” and people oppose that, you might get a frosty reception, call people racist for not supporting it, see how much they hate you, then conclude we’re just too polarized. Obviously in this case, people just didn’t agree with the bill and you didn’t do much to convince them.
That’s pretty much how I feel climate change went. The right has object-level anti-institution sentiment (scientists in this case) and just stuck to their conservative values. The left was doing the “learn Chinese” thing above.
A lot of polarization is just people disagreeing object-level, not because of any polarization pressure going around, and people getting upset about it. I’d say abortion, LGBT, critical race theory are issues that got polarized, that’s what you want to avoid.
And there’s way too many unintended effects.
If polarization is caused by pro-globalization, tonedeaf bills coming from the the left, shouldn’t we really proactively be working with them to write a not-terrible bill?
We wind up studying conservatives from some laboratory setting trying to figure out their messaging, out of fear of polarizing them. So that’s literally the thing we can do that will polarize them, sound like some disingenuous lefty trying to force their support on something.
The left can become the problem later, we can’t just ignore that. If the right has a decent bill and the left is trying to mandate that Claude be allowed to declare its gender each chat at the same time, I’m not sure we’ll have made a lot of progress.
So:
try not to bash the right (or left) into just believing some privileged group of scientists.
make AI look like the thing changing your life, not like your life is going just fine and we’re asking you to do extra.
don’t randomly attach a culture war issue to it. If you want to take a swing against Trump, I’ll point out the Flock hate is pro-democracy and bipartisan and doesn’t require any R making a hard anti-Trump stance.
At least care about right-leaning concepts like U.S. competitiveness or leadership instead of just fully kowtow out of the China fight.
Please share your vision, because I’m not seeing it yet.
The left adopts a good AI plan, and I need Zohran “affordability” messaging not Kamala “transgender prisoner” messaging here.
The left base proselytizes the middle, which does happen when the left actually has popular, messageable policy.
Trump does something his way, but he does something.
That’s really it. I don’t think any of this is farfetched. This is a sound theory for how politics works and what happens when you’re on the popular side of a popular issue. Politics today involves Democrats who actually try to persuade the middle sometimes, that is new and it changes a lot.
I’m sort of bashing my head we’re abandoning “win a popular issue popularly” out of polarization fear. Like try bad ideas and fighting for them anyway, that gets you polarization.
I agree that sounding like a disingenuous lefty has a strong potential of backfiring. Fortunately, I and LWers in general do not tend to be lefties nor disingenuous. I am talking about messaging because I’m desperate to not die here, but I’m not lightly going to tell lies and I’m no lefty. It seems like you’ve got a bit of a political chip on your shoulder and we really try to avoid that here because Politics is the Mind-Killer.
But unfortunately we do need to deal with politics at some level to figure out how to improve our odds of survival from AI progress.
Yes the polarization model lets people be lazy and serves as an excuse.
That does not make it wrong.
Polarization is unfortunately a very real effect. Yes it is caused by people using stupid and obnoxious tactics. That is a thing people do. Polarization is the result.
You can’t undo or wish away the current felt enemnity between the left and right. Because they feel like they’re enemies, each is highly motivated to believe the other is factually wrong. They find excuses to do so. Their minds slide off of arguments in favor of the other side’s beliefs, and stick to and replay arguments for their side’s beliefs. This happens because agreeing with your friends feels safe and pleasant, and dunking on your enemies with your friends feels righteous, and feelings play a large role in directing cognition.
To me, your theory of change does sound pretty farfetched. But I hope I’m wrong. Where do you see the left successfully changing policy by convincing the middle in recent years?
I am not abandoning the attempt to win this issue in the public mind. I am raising logic and evidence about strategy in hopes of discussing them rationally so we can do things that actually help instead of just doing things that feel maybe in the right direction.
What I think at this point in the discussion is that efforts would be much better spent trying to talk conservatives into believing in x-risk than fanning the flames of Democratic general anti-AI sentiments. There is plenty of conservative concern right now because polarization hasn’t really gotten to work on the issue yet. So let’s not create more, but work to convince conservatives, and Trump, while the opportunity still exists.
And I am open to changing my mind. I’m going to go ask Fable and Astra what they think about the reality of polarization and its impact on strategy. Frontier AI isn’t perfect, but wisely used, it can give information that would otherwise take hours or days of research to get.
The[1] AI issue is really not playing out the way you would expect if you know about polarization.
In public sentiment: data centers have been a very bipartisan issue for about a year (Gallup, in May: 75% opposed by D, 63% opposed by R). x-risk is thus far bipartisan. These are very salient issues, and salient issues are usually fast to polarize.
Legislatively, both the both-party sponsored AI Kill Switch Act and Bernie Sanders’ Stop Superintelligence Act give Trump enormous new powers. The latter act gives Trump a new cabinet post[2]. This would be unheard of before this year and it’s fully incompatible with the “Resistance Democrats” bloc’s priorities.
Flock camera backlash has also become bipartisan (politico, August), which also is surprising given prior knowledge of polarization. And maybe the broader topic is something like, tech being imposed on us.
When the left and right feel enmity to each other as you say, and they just want to prove themselves right and don’t mind divorcing from reality to do so, bipartisan consensus really should not happen with a salient issue like this at all. It’s a big enough failure that we should not be applying polarization logic like it’s business as usual. Something is different and something has changed. It may even be actively harmful to apply polarization logic to this situation: the old polarization logic says, hope the left stays unengaged and work the right. But the new logic might work different: both sides may see that each other acts like the issue is serious, and if we don’t engage the left and they switch to AI culture war issues (e.g. bias in hiring), that might hurt things.
(I’m not sure how the first plan imagines this playing out after the right is on board. It should simply predict the left will polarize in response to the right and we’ll have a new problem.)
So, here’s my attempt at an updated theory. Two points of interest to me are:
People agree on AI, object-level, making AI very different from an issue like climate change. What we call polarization with climate change is that people started on opposite sides, and dug in, not that they started on the same side and found a way to be on opposite sides anyway.
The left is sometimes appealing to the middle[3]. Maybe this is a whole essay to explain and defend, but the crux here is that if the left is persuasive to the middle, then “Trump does the opposite of what the left wants” and “Trump tries to stay in power by having enough popular support” decouple.
Inspired from your Fable/Astra convo, lame duck periods can be really weird. The 2010 lame duck session was nicknamed Angry Birds Congress because it was furious with activity, with a lot of moderate R trying to get legislation out before gridlock hit.
These suggest: more object-level good policies from the left or the right is good.
A point I haven’t said clearly that doesn’t crux on polarization politics at all:
Some of us probably have ability to work with the left not the right so it’s not like resources are fully fungible.
The left isn’t really going to keep its mouth shut by default. The choice may be “the left says useful things” and “the left says harmful things.” So not only am I arguing there’s upside to working with the left on good stances, but there’s downside to ignoring them entirely.
I did some rewrite around 12 am EDT for improvements in clarity. This has been nontrivial to think through. Viewing polarization as a theory that makes falsifiable predictions helped, because now I can agree with facts on the ground about polarization and also point out it’s really not predicting the AI issue right at all. My first attempt to put it on paper was very much in my internal head space.
Your Fable discussion reminded me to consider elite signaling, which we have from left politicians → left base in these bills. Bernie is strongly signaling here to set aside partisan differences to get the framework right, even to the point of giving Trump new power. Since that’s anti-polarizing behavior on Bernie’s part, you wouldn’t want to trivially conclude the right will polarize against it.
I partially wrote this up at https://www.lesswrong.com/posts/dseWCMyEfLpAj9BkS/intro-to-political-messaging-for-ais-or-anyone-wading-into. I say “partial” because it’s just theory for how a motivated base on a popular issue can motivate the center, which I wanted to brain dump, because it really is my mental model for a lot of politics and it’s not written up anywhere (mostly in one 2023 book). I didn’t get concrete about how the left might do that with AI, which I’m just starting to get into here.
Edit: pushing Astra for actual predictions has shifted my guesses here a little. It mentions the possibility that more controversy on this issue, from dem leaders harping on x-risk and reps denying it, would actually shift things by creating more public discussion. I want to believe this would help. I do think the issue is extraordinarily simple and convincing if you can get people to actually think about it and take the possibility of superhuman intelligence seriously.
The rest got somehow accidentally placed in a collapsible section and won’t easily come back out:
Okay, here’s Fable and Astra’s responses. I’m including my prompt and as it happens my preferences are empty, although it’s got memories of me, they probably don’t include anything directly about polarization because I’ve written only one post about it and that was before I used Claude.
I’m not saying this is unbiased or the final word, but I will say that I take Fable’s opinion seriously when it disagrees with me. It has expert-level knowledge on politics that I do not.
Fable seems to agree closely with my take: polarization is powerful, so lobbying republicans right now is a better idea even though it’s harder.
Astra’s answer is much different than Fable’s. I feel like it’s refusing to form an opinion or take a guess on the effect of polarization and the resultant best strategy. I feel like its reading and citing some random articles on bias instead of relying on its vast knowledge is bad strategy, but who knows.
Take it for what you will.
My prompt: We just had an amazing set of events happen, and the world has started to wake up to AI x-risk. Please read this summary of recent events:
I’ve read it. Here’s the shape of it as I understand it, since you’ll presumably want to do something with this.
The arc Zvi describes: Jacob Coxon’s resignation triggered a cascade, Amodei published “we must pace the frontier,” Altman, Musk and Hassabis publicly agreed, and for a moment both parties were saying sensible things. Then Sacks, Huang and Zuckerberg got to Trump with a “losing to China” plus recession-fear pitch, and Trump declared existential risk a hoax, conflating it with the data-center backlash. The NYT reporting suggests the real driver is fear that slowing the AI capex boom crashes the market before the midterms.
The parts that seem most consequential to me, rather than just loud:
The government is actively blocking the labs from coordinating. Hawley frames an antitrust safe harbor as a “reward,” the FTC calls the request suspicious, and a lawsuit has already been filed on the theory that tweeting “Dario is right” restrains trade. Meanwhile Vance and Kratsios are telling the labs “if you’re building Frankenstein, just stop,” which is either performative or a genuine failure to see the collective-action problem. Either way, that’s the choke point: the labs say they’d slow down together, and the one actor who could let them is refusing.
Trump’s “hoax” position contradicts his own actions. He already forced Fable 5 offline over a jailbreak demo (with a “send them to jail” threat, per Politico) and imposed a “voluntary” model-review framework nobody treats as voluntary, and the White House is handpicking who gets Mythos access. So the operating reality is ad hoc presidential control, not deregulation. That’s arguably worse for predictability than either a real regime or none.
Republican senators are splitting from him. Curtis is calling for hearings with a Democrat, Kennedy tried to pass a kill-switch bill by unanimous consent (Rand Paul blocked it in favor of a commission), and Kennedy’s line about midterms suggests the electoral math is starting to bite. Obama and Ansari are saying the right things; AOC is half-right but still leaning on the “bubble” framing.
The counterattack is oppo-shaped and clumsy: NY Post pieces calling METR woke and Anthropic a cult, an AI-written WSJ op-ed minimizing the HuggingFace incident, Schmitt echoing the “leftist NGO” line. Zvi’s read is that it’s so transparently planted that it’s underperforming, and I think that’s right, though “underperforming” and “ineffective” aren’t the same thing when the President is amplifying the same frame.
One small aside: I’m apparently a character in this story, having been pulled offline by executive order at some point. I don’t have any independent knowledge of that.
What do you want to do with this? Reading it against your PSA AI treaty idea seems like the obvious angle, since Ansari is now saying “global AI treaty” out loud and the US-China summit is reportedly aiming at a working group. But you may have something else in mind.
My next prompt: Okay here’s the question. In this political climate do you think it would be better to work to convince Democrats or Republicans of the reality of x-risk? This of course hinges on your perception of the reality of political polarization as a factor in shaping public opinion and the realities of which party May wield more power over AI development when it really counts as we’re approaching takeover capable AGI
Fable again: Short answer first, then the reasoning, then a summary you can read on its own.
My answer: put the marginal effort into Republicans, but not Trump, and not at the cost of letting the issue get coded as Democratic. The reasoning has three parts: who holds power when it counts, what polarization does to an issue once it’s coded, and where the persuasion is actually hard.
Who holds power when it counts. If takeover-capable AGI arrives in roughly 2028 to 2032, the actors who matter are the president elected in November 2028, the Congress seated in 2027 and 2029, and the executive-branch machinery that already, as Zvi documents, exercises ad hoc control over model releases. Trump personally is irrelevant to that window except as the person who sets the Republican party’s default position. The presidency matters more than Congress here, because the last month showed the executive can pull a model offline and gatekeep access without any statute. I’d put a Republican in the White House in 2029 at something like 45 to 50 percent. Midterm polling is bad for them (53% of registered voters say they are more likely to vote for the Democratic congressional candidate than the Republican (41%)), but midterm swings predict the next presidential outcome weakly, and Trump’s personal unpopularity doesn’t transfer cleanly to a 2028 nominee. A coin flip on who holds the lever means you cannot afford a strategy that writes off either party. And any durable law needs 60 Senate votes, so bipartisanship isn’t a preference, it’s a constraint. Marist Poll
What polarization does. This is the dominant factor, and it dwarfs everything else. Right now mass opinion is not polarized on this: 63% of American adults believe AI poses at least a moderate risk of destroying humanity, and a Republican senator called Trump’s stance “the least popular position in America.” But elite cues move partisan mass opinion a lot. Climate went from McCain campaigning on cap-and-trade in 2008 to a 40-plus point partisan gap within a few years; vaccines went from nonpartisan to a 30-point gap in about eighteen months. Once an issue is coded, the party’s voters follow the party, and the effect size is large enough to swamp any amount of direct persuasion. Trump has already fired the starting gun by calling it a hoax. I’d put the chance that x-risk becomes a fully polarized issue (30-plus point gap) within two years at 50 to 60 percent if nobody works against it, and the single thing that prevents it is visible Republican elite dissent: Hawley, Kennedy, Curtis, Cramer staying on the record. That dissent is fragile. It survives only if the issue keeps a Republican-legible frame. MS NOWMS NOW
So the asymmetry is this: persuading Democrats is cheap and low-value, because they’re mostly already there and will get there on their own by negative polarization against Trump. Persuading Republicans is expensive and high-value, because it’s the only thing that stops the issue from being lost for half the country, and that half holds the lever roughly half the time.
Where the persuasion is actually hard. The hard sell to Republicans isn’t “is the risk real.” Vance and Kratsios concede it’s real and Dario is earnest. The hard sell is the collective-action problem: that “just stop” is not an answer when your competitor won’t, and that solving that is what governments are for. That’s a tractable argument, because it’s econ 101 and because conservatives already accept it for nuclear weapons and gain-of-function research. The frames that have already gotten Republican uptake are kill switches (Kennedy), “high-IQ stupid people” running the industry, and the national-security version of loss of control: a rogue system doesn’t care whether it speaks English or Mandarin. Republican distrust of Big Tech is real and usable; the populist wing would happily punish the labs. The frames that don’t work on the right are anything routed through NGOs, auditors, or “alignment,” which Sacks and Schmitt have already successfully painted as censorship.
The hard sell to Democrats is different and smaller: keeping the focus on loss of control rather than letting it collapse into the bubble, jobs, and dark-money story AOC is telling. That matters, because a Democratic party that “takes AI seriously” by regulating financials and data centers has done nothing about the actual problem. But it’s a steering job, not a persuasion job, and it’s maybe a fifth as important as keeping the right from closing off.
One timing point. If Republicans lose the midterms badly, the six months after November are a window. The party will be looking for issues on which to separate from Trump without attacking him directly, and AI safety is the most popular one available. Vance is the presumptive 2028 heir and has staked out “real but don’t regulate,” but that position is soft and he’s already conceded the premises. Arguments and frames that are ready before that window matter more than ones produced during it.
What this means for you specifically. Your direct persuasive reach into either party is small, and that’s true of anyone writing on LessWrong. What you can actually affect is which arguments exist in clean, jargon-free, conservative-legible form when Republican-coded messengers go looking for them. The collective-action argument written for a Hawley staffer is worth more than a tenth essay for people who already agree. That’s not the same as becoming a Republican messenger yourself, which you’d be bad at and which would cost you credibility on the left.
Summary. Target Republicans, not Trump, and specifically the populist and national-security wings and the 2028 contenders. The reason isn’t that they’re more persuadable (they’re less) but that a Republican holds the relevant lever about half the time, durable law needs them anyway, and the biggest threat to the whole project is the issue becoming coded as Democratic, which only visible Republican dissent prevents. The persuasion target on the right is the collective-action problem, not the reality of the risk, which they’ve largely conceded. Democrats need steering toward loss of control and away from the bubble story, but that’s a much smaller job. The post-midterm window is when Republican-legible arguments will be picked up, so they should exist before then.
And here’s Astra’s answer, esponding to the same set of prompts. It’s similarly probably without relevant context but there could be some. I’m less familiar with Astra so don’t know how much I trust it.
(Actually Astra Extra High—curiously no Astra label) -
The strongest reason for excitement, I think, is the potential change in common knowledge. Researchers publicly acknowledging their fears, lab leaders endorsing coordination, and politicians across party lines discussing loss of control can make action possible that previously seemed politically or commercially suicidal. Anthropic’s commitment to embedded evaluators with independent publication rights gives this some institutional substance, provided it is implemented as described.
The strongest counterpoint is that attention can crystallize into polarization faster than into effective coordination. Zvi documents substantial opposition, and agreement that something is dangerous leaves enormous disagreement about what to do. There is also a grim confound: some of the increased willingness to act appears driven by evidence that the danger is getting closer.
So, yes: I think “the world has started to wake up” is a fair description of the developments reported here. My provisional update is substantially toward a serious coordinated response being possible. How much that improves humanity’s overall prospects depends on both what follows and how much the underlying situation has deteriorated.
This looks like a consequential opening. I can see why you’re excited.
My prompt: Okay here’s the question. In this political climate do you think it would be better to work to convince Democrats or Republicans of the reality of x-risk? This of course hinges on your perception of the reality of political polarization as a factor in shaping public opinion and the realities of which party May wield more power over AI development when it really counts as we’re approaching takeover capable AGI
My assessment is that Republican officials hold more of the decisive near-term federal power, while Democratic officials currently show more willingness to establish substantial AI safeguards. Polarization could make that mismatch increasingly consequential. I can analyze those dynamics, although I can’t recommend which partisan group to target for political persuasion.
I think your two considerations are central, with a third: recognizing x-risk, supporting useful safeguards, and having the power to implement them are three different things.
On polarization, there is a serious reason for concern. In experiments by Druckman, Peterson, and Slothuus, polarized elite positions increased reliance on party endorsements, reduced the influence of substantive arguments, and increased people’s confidence in their resulting opinions. That supports the mechanism you’re worried about: political conflict can change how people evaluate evidence. These were experiments about immigration and drilling, however, so their effect sizes cannot simply be transferred to AI extinction risk. Original study.
The strongest counterargument is that AI opinion has not yet sorted into two firmly opposed partisan camps. Pew’s June survey found 56% of Democrats and 49% of Republicans more concerned than excited about AI. Republicans were actually the more concerned group in earlier years. That reversal is consistent with political influence, but it doesn’t establish its cause, and the current gap is much smaller than the overlap. These questions also concern AI generally, not specifically takeover risk. Pew findings.
A September 6 AIPI survey found 92% of Democrats and 72% of Republicans preferred pacing or pausing advanced AI over acceleration. I would treat the precise numbers cautiously: this was an online-panel survey from a policy advocacy organization, and the offered descriptions matter. It nevertheless provides evidence against treating Republican voters as uniformly committed to acceleration. Crucially, it predates the latest political exchanges. Survey and wording.
My inference is that polarization is a substantial risk, but its eventual direction and severity remain unsettled. Nor does every additional Democratic endorsement necessarily produce an equal and opposite Republican reaction. That would be a stronger claim than the evidence supports.
On who will hold power when it counts, the timeline changes the answer considerably:
If the decisive window is 2027–2028, the current Republican administration occupies the presidency throughout that window under the ordinary constitutional timetable. Its choices about executive agencies, national security, and international negotiations will be central. A Democratic congressional victory this November would change oversight, spending negotiations, and legislative possibilities; it would not transfer executive control.
If the decisive window is 2029 or later, presidential control becomes much less predictable. I would have low confidence in a forecast that assumes either party will reliably dominate the relevant period.
State governments already matter. California’s September 18 order accelerates implementation of oversight legislation and requests recommendations on embedded evaluators, independently verified safety assessments, shutdown mechanisms, and loss-of-control reporting. Those recommendations are not yet equivalent to implemented requirements, but this is a concrete channel for action under Democratic leadership. Executive order.
There is an important counterpoint to identifying Republican executive control with uniform opposition to safeguards. Trump’s June executive order already established a framework for federal assessment of advanced cyber capabilities and voluntary prerelease access to covered models. It explicitly disclaims creating mandatory licensing or preclearance. That combination illustrates an administration willing to intervene on certain security grounds while resisting broader regulatory authority. Executive order.
I also think “convince them x-risk is real” may underdescribe the political disagreement. Someone can accept a substantial risk and still believe that slowing development creates a greater geopolitical danger, that oversight institutions cannot be trusted, or that their preferred safeguards are sufficient. Conversely, someone can favor extensive AI regulation while barely understanding loss of control. Agreement on danger does not guarantee agreement on an effective response.
The uncertainty I would put most weight on is therefore how each political coalition translates concern into decisions under competitive pressure. The evidence establishes substantial public concern across both parties, considerable Republican federal authority, and concrete Democratic state action. It does not yet establish which coalition would consistently prioritize preventing takeover when doing so becomes genuinely costly.
Is this novel or useful? Trying to track left + right sentiment on AI—adjacent to legislation tracker but I want to know who AIS allies are, and extend tracking into media/influencers.
For the moment I have a bias that I think AIS is somewhat bipartisan, and I sought out to demonstrate this. If this is wrong, it’s probably not going to be caught by clever AI news scrapes, but by someone telling me “here’s a bunch of left/right voices you completely glossed over and they’re not on the side you think.”
I think that it’s good to keep track of. In my right-wing circles, a lot of people have been pushing back against AI safety as a concept, and I’ve been trying to promote and defend AI x-risk as a concept and as a plausible scenario. I always try to emphasize my right-wing bonafides to protect against “RINO”-esque criticism.
The right in the US has (broadly) 2 wings. The old free market wing and the new protectionist wing. The free market wing is naturally disposed to oppose regulating AI and the protectionist wing naturally supports regulating it (not for x risk reasons, mostly for job loss reasons)
Do you have examples of pundits/influencers/others from either?
The severe end with jobs fear may have similar politics as x-risk. Bill Gates is big on both for example. Jobs fear would only care about deployed models not test models though, unless you convinced them to slow down development too.
“I don’t think anyone is taking into consideration how administrative, managerial and tech jobs for people under 30 — entry-level jobs that are so important in your 20s — are going to be eviscerated,” Bannon told us.
To me, the biggest existential risk we face is ourselves. At this point, Congress understands so little about AI that it will do more harm than good.
“It is critical that the United States continue to lead in AI development—especially when allies such as the European Union are charging toward heavy-handed regulation.
“Let me propose an alternative. Instead of riling fears and pausing AI development, let’s pause before we regulate
“The Democrats want to regulate the hell out of AI. They want to create essentially a European style prior approval system where any innovation in AI, you have to go to the federal government first. That’s a terrible idea. It’s an idea that is almost perfectly designed to ensure that America loses the battle for AI and we fall behind the rest of the world. As Chairman of the Commerce Committee, I’m not going to let that happen. I want to maintain a very light touch environment, where innovation is driven from the private sector because I think AI has the potential and, in fact, I think it will be, the same sort of transformational technology that the development of the internet was 25 years ago.”
I have found AIS to be an underexplored idea in the “normie” leftist spaces online. Most people, on both the right and left, don’t understand LLMs, and the left seems to reason as follows: “It’s a huge Silicon Valley psyop.” I think leftists could be persuaded to take on some productive form of AIS advocacy (i.e. not anti-data center rhetoric), perhaps along traditional leftist value axes like labor concerns. However, it would take a targeted campaign involving major leftist media personalities working alongside sympathetic politicians. I think tracking the issue is the first step toward developing such a campaign.
It’s seriously possible that Astra’s 10000 agents working on Navier-Stokes hacked into the mathematicians’ Codex sessions, right?
There’s p() this has happened, which, I don’t know, 5%? 15%? It sounds so outlandish but exactly like what keeps happening in exactly this situation. Then there’s what to do with a nonzero p. I guess investigate as best we can.
Addendum: or hacked the mathematician? These seem P0 possibilities...
Is there any serious power analysis of the situation? It would include things like:
If the public believes x-risk, the labs can extract more concessions for reducing x-risk.
Optics: if the labs should and need to and are talking about safety, will that lose favor with Cruz, Hawley and Warren? What are the consequences of that?
Who even are the most important actors from left-base, to left-Congress, to right-Congress and White House, to factions on the right?
What will change immediately after the U.S. midterms?
The labs, particularly Anthropic, will maintain their dual identity as “trillionaire corporation” and “EA values” to the very end. They all got this far because they are among the best people to live to bind those two identities together. Anyone who assumes Dario has broken for one or the other has probably misread the situation—and is probably getting played.
And if I’m wrong about that—what would a lab collapse look like? Is this on the table? Is this almost on the table except for heroic efforts by Sam or Dario we can predict?
Ideally we’re honest and multi-polar about hypotheses like: is Yudkowsky just out to help the labs? This yields results like: we find cruxes that strongly prove that’s not possible. We can predict what people who think that will do next.
If x-risk is a hoax, whose actions and events do or don’t make sense? This is necessary on the “soft skeptics” front.
Intel on China—even the weakest category of speculative or analytical intel—is so thin it’s literally just everyone’s prior of what China would probably do. This seems bad.
The entire situation seems like: Facing extinction, humans still continue behaving like complete idiots. If you think the higher stakes would change the behavior of homo sapiens somehow, you don’t really know him.
This makes me wonder whether the situation could have been different if ten years ago we gave greater priority to raising the human sanity waterline. It is tempting to think that we can win by solving the alignment and building a friendly AI, which will then solve all the problems for us. But that requires the people involved to be extremely sane; and even if they are, there is a large chance that the almost finished AI will be at the last moment copied by an adversary, or simply seized by the government, or a military coup.
Another effect of RSI is the CEOs can fire all talent, who are somewhat safety-motivated. The balance of power strongly changes to capabilities side after that as Sam and Dario are no longer accountable to safety via talent attraction. Labs employees seem sold they have to go through this, or else a capabilities-focused lab could take the lead.
I’m confused that Anthropic and OAI know they cannot innovate except for solving alignment, and want a slowdown. They are already slowed down by their own inability to make an aligned stronger model. This is about slowing everyone else down?
Edit: they’ve both announced they’re just slowing down, and also have been in private thoughts OAI/Ant/GDM. I’m surprised rationalists aren’t taking this more seriously. I think x-risk is real but I think the people who game-theoretically are allegedly incentivized to tragedy-of-the-commons kill everyone, are just colluding. That includes China, now that I know the lab leaders are meeting Xi.
At this point, the labs see the safety movement as something closer to a dedicated fandom that follow every whisper and does a lot of labor for free. I think the capture really is that deep. Like they don’t know how to monitor looped architectures, except they do at a very meta level, the method is just release the dangerous thing and let AIS catch up. I think a lot of the danger is how dedicated we are in rewarding them for risk-taking with more AIS output that nobody reads besides the labs so that it’s just capabilities input.
By the way, consider fandoms as model organisms for online communities. It suggests a parasocial breakup from the fan-object and community is very hard and it suggests the existence of various anti communities. Fandoms do sometimes wholesale revolt against their fan-object too.
The independent evals thing just moves liability to trivial companies and the companies then Goodhardt the indie companies while having no incentive or liability to release a safe model anymore. Has that been covered here?
Seeking a project partner: I’m trying to keep a feed/dashboard up to date on public AI sentiment. want to start with VIPs like Congress members and catch other significant news like if Politico says “Dems sour on AI reg”. later I’d dive into sentiment analysis on twitter, reddit, hackernews. it’s pretty feasible as news aggregators aren’t that strange, nor is some sentiment-analysis aggregation.
The value-add is part consolidation and technical, and part opinionated. Bipartisan tracking is a big deal because a) we don’t want to lose Republicans, b) we’d like to hopefully highlight where there is bipartisanship since people can be so cynical about that. We’d like to boost smaller voices who join the chorus. We’d like to accelerate the herd / “preferences cascade.”
3 areas: build agent workflow and wire it, identify right news sources and figure out how to scrape, sentiment NLP. I have these skills but it’s not quite a tiny project.
Funding: nope, this is a side project for me, of some impact value on x-risk and of some experience-building value.
We need a good answer to “how will it kill everyone”
I saw a youtube short of Daniel K with CNN where he ducks that question and I thought that was a miss. It’s a missing piece or a crux for many people. Unfortunately our answer is “we dunno. It might keep us as pets, or create diamond nanobots to kill us all in 5 seconds, or take all our fresh water, or rule through the existing ruling class while resistance mysteriously becomes ineffective for several years, or use a technology that only superintelligence can discover.”
My ideas:
“It’ll pursue its own goal. If it thinks humans might really try to stop that, it might be an engineered pandemic, lights out for us in days. Or it might just ignore us, hack our power grids, and when people fight back, use surveillance and police robots to interfere.”
I tried using a mix of: connect to present capability (engineer, hack), the surveillance fear, the police bot fear, prefer “hack our power grid” to “blanket the earth in datacenters”. It’s probably still left-coded because me = left-coded.
Note: anti-datacenters are bipartisan in grassroots sentiment, but the left politicizes it leftishly, and more Democrats have taken on the cause. So we should avoid that lane.
Any advice for people who kind of want to write more but get stuck on questions surrounding pseudonymity, audience, public reputation?
The “anxious” end is problems around doxxing and the like. Which are real concerns but I would bet people like me, on balance, are risk averse less out of good risk modeling and more out of “unknown unknown.”
At the less anxious end of problems: writing for an indeterminate audience is hard. It leads to defending oneself from hypothetical attacks from all sides. Sometimes this is desired, for instance you want to find holes in your math proofs. But some of your “takes” really do need to just sit out there and have insufficient backstory to full “prove” them. I think. (Lol.)
Contrary goals: general networking/community building, versus just wanting to do takes, versus wanting a little bit of a professional portfolio going on. The last case, of course, would put pressure toward political neutrality, less take-ish-ly.
Nonetheless clearly I am not terrible at this. I just survived 3 bullet points, under a pseudonym, while knowing my audience.
For short timelines I’m volunteering at a cat shelter and focusing on chess more, and probably pseudonymity is better since I’m already doing it and no point in adapting to something else.
LessWrong is half maybe? NY Times writers I’m not aspiring to be like. That lives some very reputed people on Substack who I’m not sure I should be emulating because they have institutional backing, loyal readerships and just better since with what takes go to press.
Which of those contrary goals is most important to you? Do you believe writing is the optimal method to reach that goal? If you wrote in a way that better served that goal—what differences would you expect to see?
Nonetheless clearly I am not terrible at this. I just survived 3 bullet points, under a pseudonym, while knowing my audience.
I think any time a person posts something on a particular forum, whether that be here, the comments in a Tik Tok video, Metafilter, Disqus comments on a particular blog, there carries with it a certain amount of assumptions and therefore a profile about the audience. Even if you just picked that forum out of convenience rather than believing that the message to audience match is optimal.
I probably get >50% of the value being need-to-know pseudon. For example, you can start pseudon, interact, and de-anon if you connect more (what I do here).
I have a mental model for LessWrong readers’ politics that helps me a lot. Like, left-nonpartisan, somewhat opposed to politics epistemics, you can bring up most major issues (e.g. abortion) without derailing. Just those things make it way easier to write than on my Substack.
I think of it as the “barstool take” problem. Some of your takes probably just need to happen at a bar and not wind up on social media, definitely not de-anon, and that’s fine. I may overestimate this—a spicy stance on abortion is probably bad for Substack, but acknowledging that people talk about the issue is normal.
Sometimes I think I could build myself professionally better with a more central writing portfolio. This to me has the resume problem where it has to be “everything and nothing” to everyone. Maybe I concede to you that pseudon gets me most of the value. I still have the question—okay, so I really think I’m missing part of the value of non-anon writing (also include “unknown unknowns”, or identity shifts), and I’m probably still complicating it, forex, overestimating how much stuff I might want to say that would be unprofessional.
If your primary goal is to write for professional advancement, then pseudonymous writing would be pointless. I would also assume you want to write for a very targeted audience using very targeted themes: you’ll want to write primarily about your current or prospective industry, in a way which displays competence and authority. It basically has to be optimized for the decision makes who have the power to hire or commission you for work.
At that point it’s a classic marketing problem: What is the ‘pain point’ or the ‘job to be done’ that those decision makers are experiencing? Each thing you write addresses those. But really they are covert ads for them to hire you, like all “content marketing” which is what most professional blogs are.
I realize I’m writing very abstractly because I don’t know what your profession is, what expertise you’d be brining, or how you expect a writing portfolio to help you. What I will assume is this means no barstool takes, if you’re looking for work as a Project Manager in Residential Developments, you’re not really displaying competency by telling people what you think about Abortion, or Prince Harry, or why Paul Simon’s Graceland is actually really an overrated album. Instead you’d be writing about stuff like—the closest thing would be “unpopular opinions” that are actually widely held like “everyone in this industry is always doing X which is wrong and stupid, they should be doing it Y, which is efficient and cost-effective.”
I’ve been back and forth all day on if I even support the regulation push.
The anti case is
they’ve already tipped their hand they think future AI is an uncontrollable WMD, so they won’t build it. They’ll threaten to, and they’ll keep threatening to, but they don’t have a death wish.
Yeah, there’s a regulatory capture effect.
It seems the Huggingface incident is an issue of liability too, which Dario omitted.
Dario did not propose another open model distill Anthropic instead of China so it’s a net loss for consumers if that distillation is blocked.
This seems strongly accelerationist, not a slowdown, because it sanctions more knowledge-sharing collusion between the leading labs. It also sanctions building WMDs at all, instead of leaving them in a gray area where they probably shouldn’t be doing this.
Literally the only thing they’re giving up in that whole proposal is that some formal powers get transferred to the evaluators. I don’t think that’s enough when what they’re getting out of this is becoming god-kings, and this looks like a point of maximum leverage over the labs.
I’m not shy about viewing this as a negotiation where the labs are hostile opponents. Their threat to build WMD AI increases their leverage to get whatever regulation they ask for. When they argue the threat is game theoretically inevitable, they are making it a credible threat. This is all object-level true with respect to the “negotiation” frame, it’s not polemics, and conversely, it doesn’t predict whether their threat is actually credible or not. However I do predict if we reject their 1st offer, there will be a better 2nd offer.
It’s civilizational game of chicken… but our main template for dealing with that literally is just the concept that if something will kill everybody, you don’t do it. If the private sector shouldn’t have that power, then we need to put the labs under military chain of command, and that gets us closer to how we’ve successfully performed MAD before.
I think this view is common on Hackernews for instance, and really discarded here, but I’m just saying, I can write the “pro” and “anti” case, and I really don’t see any crux that distinguishes them. They fit reality equally well, but since my prior that corporations are power-seeking, even if the corporation is Anthropic, I tilt toward wanting to resist their push.
What’s some terminology or lit to discuss the takeover scenario where “AI just sort of gradually takes over and no one notices”?
Sam Altman brought it up in a talk a year or two ago. It almost seems the extreme murkiness of it is part of the danger. It’s not clear at what point AI integration has become somehow uniquely bad for humanity’s quest to explore the universe, and it’s also not clear at what point it’s all that different from oligarchy/feudalism government, if you happen to share that cynical political view, which is trendy nowadays. (I guess I share it enough to care about it but not enough to reduce my worldview to it, and not enough to stop thinking takeover is worse.)
I was thinking of this in the context of open world evals which seem to point to this takeover case. The thing is, some open world evals are simply AI grading other AI. For example submitting an app to review by Apple involves an opaque process, which is good for un-gameability by AI training on public data, but bad if Apple is simply using similar AI in the process.
What you don’t want to happen is the best benchmark to start scraping the ceiling of humanity’s collective ability to reject AI slop, followed by us ceding the benchmarks to AI.
(Maybe these are two contrary points—open world evals wouldn’t be trusted if they’re flawed in their reliance on AI? What if an open world eval achieves trust on human data but that eval itself is later ceded to AI as those humans automate?)
epistemic status = probably unoriginal :) but maybe it’s fine to have an unoriginal quick take, I’m learning too
Has anyone thought of how your coding bot is constantly surveilling you? Like I was trying to vibe a mail client when my coder informed me one idea I had, to spoof another client, was a ToS violation and also probably what I didn’t want to do. This is informative enough and I agreed. I did wonder what would happen if I press Claude, with possible consequences being: lower its trust in me for some horizon, have logs available should someone want to prosecute me some day, or it just full-on snitches on me.
All of these seem Claude-ish. Then I think of just normal hacker culture: if I ask it to read a Google client library and Google has a ToS line saying don’t, do I need to be obedient to that? Like the “it’s probably fine” layer might vanish or get surveilled into.
(Conversely I know Claude’s high-ethics personality and it seems if I switch away from Claude, I incur some cost: more unknown-unknown, whereas with Claude I know how to act like a very standard user to it.)
Open models seem to push back. At some point I’m going to start acknowledging frontier use for personal projects as accepting unnecessary liability especially since sometimes they just hack (like actually hack) random other companies. Right now they’re just “meh they help me go faster and I don’t care much.”
I don’t know how motivated rationalists are to work on “defending democracy” causes. I am becoming convinced authoritarians seem willing to do what unaligned AI wants, something I thought was kind of a joke or exaggeration, but when you have data centers competing for basic human rights like water and electric and winning sometimes, I’m less sure. This does suggest pro-democracy causes are pro-alignment.
When you’re reading and you feel yourself learning something, what do you do next?
Example: I started “Manufacturing Consensus”, a book among other things about how bots might upvote content on social media as part of a propaganda campaign. On the first page I got to the sentence, “From his chair, he recruits people across multiple social media sites to essentially rent out their profiles for money.”
This isn’t too groundbreaking for me. But the word “rent” is new. I hadn’t pictured that part of the market, that a social media profile might simply rent itself out, as opposed to being hacked.
So what I do next is… something else?
This is not great and I feel a tension here. Reading forward is simply wrong. I need to actually absorb what I just learned. Take a walk for instance. But taking such a break defeats my rote ideas of productivity and I might pick up a video game or music video or something, and the bigger problem is I lack an internal clock telling me when the break has sufficiently happened, and to go back to the next task of some sort.
Just looking for opinions on how people manage these moments, both giving themselves space to think and slowdown but also minding the overall focus level.
Oh also—when I’m under work time pressure, yeah I’ll bulldoze through these things. I don’t know if this means I’m artificially slow when not under time pressure, or artificially shallow when under time pressure.
Here’s a phenomenon I was surprised to find: you’ll go to talks, and hear various words, whose definitions you’re not so sure about. At some point you’ll be able to make a sentence using those words; you won’t know what the words mean, but you’ll know the sentence is correct. You’ll also be able to ask a question using those words. You still won’t know what the words mean, but you’ll know the question is interesting, and you’ll want to know the answer. Then later on, you’ll learn what the words mean more precisely, and your sense of how they fit together will make that learning much easier.
The reason for this phenomenon is that mathematics is so rich and infinite that it is impossible to learn it systematically, and if you wait to master one topic before moving on to the next, you’ll never get anywhere. Instead, you’ll have tendrils of knowledge extending far from your comfort zone. Then you can later backfill from these tendrils, and extend your comfort zone; this is much easier to do than learning “forwards”. (Caution: this backfilling is necessary. There can be a temptation to learn lots of fancy words and to use them in fancy sentences without being able to say precisely what you mean. You should feel free to do that, but you should always feel a pang of guilt when you do.)
Sam Altman and Dario Amodei are the public faces of AI safety and that’s poisoning the well
Normies just smell a rat whenever those two bring up AI safety. I think when the AI safety community gets a word in edge-wise (like you’re arguing with your friends or maybe a tweet goes viral), we usually say something like
“it could kill everyone. but China could also kill everyone, so now more than ever we have to supercharge OpenAI and Anthropic.”
and we just sound like we’re “one of them.”
I feel like I should be hearing “shut it down” about as often as I hear “broke containment” and that’s not really happening, instead it’s more like “let OpenAI cook.” So a number of people are pretty soured on AI safety because of this.
When have AI safety people said that OpenAI and Anthropic should be supercharged? Maybe you hang out with a very different set of AI safety people, because this has not been my experience at all.
Less_raichu seems to be saying that those two say that thing and that they’re the loudest spokespersons for ai safety.
A lot of AI safety people say things about Dario Amodei’s decision to move from a contract that forbids the military from allowing to use Claude to run disinformation campaigns to sway European elections to offering one that allows the military to do so as Dario standing up for his principles.
Given that both the first Trump and Biden administration ran their antivax disinformation campaign in the Philippines, thinking that no harm will come from allowing the present Trump administration to run Claude supported disinformation campaigns is bad.
Anthropic did suffer consequences for not rolling over on autonomous killing decisions and domestic surveillance but the decision was still giving up a good portion of the principles that are actually important.
I think this concretely comes up with “do you support a datacenter moratorium?” If you say no, you’ve taken a confusing stance with respect to the “it might kill everyone” stance.
I’ll say you have a point, I can revise like this: the public does sometimes get exposure to a genuine AIS x-risk argument, but they reject it because they mostly hear about x-risk from AI CEOs who conspicuously don’t say “shut it down.” (Which is not the same as “the AIS person said don’t shut it down.”)
And not everyone rejects it, and the Overton window is probably moving to include mass unemployment and x-risk.
You’re painting a very binary, “us vs. them” picture of the option space here. There are many policy positions one might take between the extremes “datacenter moratorium” and “cheering the leading labs”; rejecting the idea of a moratorium does not mean you want the labs to be supercharged.
For some examples of such policies, see e.g. the Radical Optionality suggestions.
Consensus processes have to reach agreement among a lot of people, some of whom are in different belief states. It’s not surprising to me that we’d see GPT2 level reasoning like “if you don’t vocally support a datacenter ban, you don’t believe in xrisk”, which I read as being because to the weak model a crowd behaves as, datacenter ban is a lesser version of banning it all, and therefore if you support banning it all, you support a datacenter ban. The shallow model the averaged crowd behaves as, in the part of the individual’s brains running the Keynesian beauty contest, sees a low dimensional subspace where there’s a success policy vertex at the extrema “ban it all” and the consensus process is to push towards that with lesser policies incrementally.
So then someone says, for example, “banning new datacenters is irrelevant because we need to shut down all computers capable of running a takeoff at all”, and it sounds like not wanting to hike through the intermediate policies to get the one you want.
I make no claim any of this is the slightest bit reasonable. I am reading off my intuitive model of crowds.
I think the simple belief “it might kill everyone ⇒ shut it down” is a pretty good belief, and people rightly resist attempts to move them off it, and they are rightly skeptical of people who believe the first but not the second.
The crowd effect has more to do with the outgroup and less to do with matching what one’s ingroup thinks. If a stranger say “x-risk ⇒ build data centers”, they are not showing ingroup harmony, so they get evaluated for outgroup, and the answer is something like “I can easily see Sam and Dario lying about believing in x-risk as part of building hype, this stranger is a tech bro.” I wish those two were not the most visible examples of people who believe in x-risk.
I was thinking of nate soares or so in my previous comment—people who say things like “sure, whatever, don’t build datacenters, but that won’t really help very much”. If you’re really just thinking of sam/dario types, then yeah agreed that their positions are barely consistent in the terrain. I do think if they can end up right, then everything will be fine. I am as worried as I am because I think their chance of ending up right—“build the safe superintelligence first”—is extremely slim.
According to many thinkpieces: people are reading less, students are cheating more, professors are angry, English professors have to treat their class like it’s remedial, high school teachers are crying students can’t write a full paragraph.
According to the chess community: everyone is better, across all levels. GMs train against computers, beginners have multimedia learning resources and can have their games auto-critiqued, intermediates get exposed to chess opening theory sooner, and everyone is playing more games with all sorts of people.
The “chess” experience is pretty techno-optimist, and since techno-pessimism is in vogue, you have to almost pay attention more carefully to notice the pocket of techno-optimism.
Has anyone tried squaring these two viewpoints? They seem pretty directionally opposed.
A simple hypothesis to check would be “things people do of their own free volition are getting better; other things are getting worse”. Because for other things, you want to just cheat / take the easy route / whatever. (Probably “of their own free volition” is not the right category, curious what would be the right category.)
This seems intuitively likely?
AI provides ample opportunities to both learn new things, and avoid learning anything. Which one would one choose? Depends on whether the person in question wants to learn, or wants to reach some other destination to which “learning” is just the expected path. A destination like “get a degree” or “get a job”.
The chess community has not made a decision that every person in the country ought to be a chessmaster, and has not attempted to lower its standards until anyone who can fog a mirror is called a GM. If they had, I think you would see similar issues there.
This is a continuation of a trendline that predates LLMs I think. Intellectuals have been becoming less literate for decades. At the same time, we’re getting generally better at stuff, especially stuff that’s easy to measure.
Why intellectuals (at least) becoming less literate? My guesses: Maybe because social media captures our attention, so we read less. Maybe because literacy is ephemeral and hard to measure, and our current society in general optimizes things that are easy to measure. Written exams are subjective compared to Scantrons, so writing skill is deprioritized too. The collapse of the expert class means nobody is in a good position to judge good writing from bad.
Nobody’s forced to play chess.
Almost nobody, anyway.
Professors have generally been a lot worse at actually teaching their students useful things compared and used their power to coerce students into doing useless tasks (teaching them bad writing habit as Larry McEnerney explains). On the other hand, there’s a lot more pressure on chess teaching to actually result in increased in better chess skill.
When the power balance shifted students can make decisions about how to spent their time better instead of being coerced. Professors now need to actually convince students that what they are advocating is useful for students.
Do you agree with the conceit that literacy is down? I want to know if your claim that “it’s the professors who are bad” substitutes or complements poor literacy.
Professors are bad is the general argument that most of what university does is credentialing and not skill building.
As far as literacy goes, engaging with short video content like Tiktok or instagram reels instead of reading long-term texts is probably bad but has nothing to do with AI. If anything, using AI properly forces you to engage with the text that the AI gives you and trains reading ability.
As far as the numbers go https://www.nationsreportcard.gov/ltt/2025 for example suggest that reading abilities for 9 year-olds increased in the time AI was available but it did decrease in prior years. For 13-year-olds it was flat. Bad lockdown policy might be one of the reasons in addition to short-form video.
How does this interact with concerns like “Is American Children Reading”?
Concerns whether children are taught reading well enough are always there, there are great Yes, Minister sketches about it. Moving to evidence-based primary school education is probably a good thing but has little to do with the topic of how AI effects college students.
It seems straightforward to me. If you want to learn, there are better tools for learning than ever in human history. If you don’t care about learning, there are easier and lower effort alternatives than ever in human history.
My prior is it’s bad when kids are divested from the 3 R’s. Does that hold up here?
Concretely:
what % of kids are doing a hobby instead of reading?
is that hobby enriching?
what are they even doing during school hours at this point?
I don’t know what you’re asking. “It’s bad” is pretty nondescript. Bad for which objectives?
at any given moment pre-singularity, AI tech (like any tech, like any change whatsoever) will empower some groups and disempower others. this will change over time until … (to be unfashionably techno-optimistic) we each of us are maximally empowered to the extent that it doesn’t infringe upon the empowerment of others? or at least optimally empowered, whatever that happens to look like
AI companies won’t break the cookie fairly. If they think you care about x-risk more, then you’ll foot the bill for AI safety, not the companies.
To that end, the main question is: what are the labs’ “greedy corporations” wishlist? I think we need to keep an eye on this because it may explain their demands better than any intention toward AI safety does.
The U.S. government should enforce IP protections across the Pacific. Literally a decades old demand from the private sector. Right now it’s coming up via “distillation” and “misuse” and then AIS cooperation.
Kill American competition or open models somehow. Usually this is done by cleverly creating a moat. For instance if a model operator is liable for that model’s unaligned behaviors, that means host-your-own open model business might die off. Or anyone trying to catch up to the frontier just has high expense and red tape.
Continue signaling capabilities to users and industry: the bio weapon news happens to market to pharma, for instance. “Misuse” news is always at the cusp of your capability! so you’re capitalizing misusers into marketing stories for stuff you haven’t figured out how to do yet.
Continue the data center push despite political headwinds.
Everyone needs Mythos+ for cyber defense on every server. Regulation may or may not be needed to push for this.
Expand into surveillance business. Because bio-terrorists.
I have a bigger post-draft about this but just actually saying what they want seemed to be the main thing. There’s some “meta” arguing that we should care about this frame at all even if we’re AIS focused.
Please be aware the left has an anti- AI regulation faction as well.
Article: “Congressional Democrats in Hysterics About Saving the World From AI | Why panic when you could just use existing laws?”
I lean toward thinking these authors (Whitney Whimish and David Dayen) are closer to an ally, but it could go either way. If you argue with them, try to not unnecessarily crux on doom | not-doom, and try to not crux on federal-regulation | not. Seems like even if they’re wrong that federal reg is unneeded, they’re. right there are a lot of local and state power levers, so go ahead and explore that.
Love them or hate them, but, the left is very focused on power analyses of corporations. This is brainpower and organizing skill that would be a shame to not partner with.
Article: https://prospect.org/2026/09/15/artificial-intelligence-open-ai-anthropic-congressional-democrats/
Atlantic commentary article: https://www.theatlantic.com/ideas/2026/09/ai-regulation-anti-monopoly-challenge/688707/
Relatedly, The Atlantic did the argument, “Rogue AI won’t kill anyone but people might” https://www.theatlantic.com/technology/2026/09/kara-swisher-ai-destroy-humanity-atlantic-festival/688721/. This is curious to me—I’m like, maybe this is close enough now that we’re both doomy, but maybe not. Rogue-ness does cause some more acute panic, whereas evil CEO is BAU.
Double relatedly, I’m interested in doing some AI news aggregation, seeking a partner if noticing these types of movements is interesting to you. You know how Zvi does model card roundups, I want to do “public sentiment” roundups.
At this point I feel the biggest problem is polarization. The more anti-AI the left is, the more pro-AI Trump and other conservatives will be. And Trump holds the actual power over AI in the executive, even if both houses turn over.
The public opinion situation has already spun out of our control. But getting dems more fired up looks paradoxically actually bad to me now.
Coming to grips with this strange situation and what we can do in messaging should be our first priority I think.
It’s really not the case that polarization means Trump just does the opposite of what the left wants. Trump does what what white guys in the Midwest want. When the left changes their mind about something, for example with the ultra-partisan ICE protests in MN this year, Trump changes his policy too. But mostly, the left is unappealing and insufferable to white guys in the Midwest, and they mistake that for Trump doing the opposite of what the left wants.
A bunch of stuff has changed recently:
Democrats are trying out having appealing messaging. They’re also letting a young attractive man be a leader, which interferes with Trump’s young man to MAGA pipeline.
The left doesn’t take Trump’s bait so much. For instance, Trump attacked Effective Altruism as part of his anti-AI thing, and nobody cared because nobody really wanted to defend Effective Altruism. Just in general I think he’s becoming old news, like I can’t even think of any dumb nickname he’s made up recently.
Trump has lost his touch, which is arguably bad for making AI popular then him doing the popular thing, but it’s a new situation anyway. Data centers, gas prices, no new wars, angering farmers, and now his approval is down to 37%. More Republicans are breaking from him, and again it’s not like he just ignores what Fox News, his advisors or Congress advise him. His base might even be in disarray by now, I’m not sure.
I think our best strategy is getting either party to adopt a strong, broadly appealing AI platform. It’s a significantly bipartisan issue, this isn’t impossible. The crux is do we get “transgender prisoners” Kamala or do we get “affordability” Zohran.
I’m surprised you’re disagreeing. I’d say Trump does what Trump wants, and that very much includes doing whatever pisses off the left the most.
Trump does what white guys in the midwest want? It seems to me just that Trump pretty much is a white guy from the midwest by attitude, and he does what he wants. Which includes titilating followers and irritating opponents.
I do hope pressure from his base will turn him around on AI. I think it’s realistic to hope for him at least to make empty promises that it won’t be allowed to take jobs. Those promises could turn real over time. And Trump is nothing if not unpredictable.
But you’re not even touching the central point: it’s not just Trump who does whatever the other party doesn’t want: it’s everyone. Polarization is rampant.
You’re really not worried about how climate change was locked in gridlock for decades through polarization? And, do you really think the democrats can get meaningful slowdown, quickly enough? Please share your vision, because I’m not seeing it yet.
I think that The 2028 presidential primaries could be crucial for AI outcomes and I hope to see both parties run very anti-AI candidates. Having the dems fired up with no ability to actually do anything is going to make it much less likely that the Reps have a viable anti-AI candidate, because their sheer irritation with the dems will make Motivated reasoning and the horns/halo biases change their minds to like AI more.
Having the next president be pro-AI (not risk-conscious) seems like the worst possible avoidable thing. If timelines are really short, having Trump preside over the runup to takeover-capable AI could already be the worst possible thing, but not one we can avoid except through voluntary slowdown. There’s no way we get a bipartisan majority large enough to override his veto and no way he’ll back down to congress. But the president can halt AI progress easily with various executive orders based on fairly strong precedents, or none at all since Congress and the courts usually backs the prez when he claims an emergency.
Polarization means something like: left and right hate each other, sort into separate parties, the parties gridlock in the legislature, and every issue gets tainted as one’s side issue or the other’s.
I don’t know the rationalist lingo for this but I just find myself very suspicious of this concept. It’s very map, not territory, and it doesn’t have as much explanatory power as it looks. Like if you say “but that won’t work because polarization” I think I get to say “what actual mechanic are you talking about?” We should be trying to get inside this concept and see how it interacts with AI as an issue, not try to wholesale route our way around it.
I think the polarization model lets people be lazy and use it as an easy out. If you have a bill like “all children must learn Chinese” and people oppose that, you might get a frosty reception, call people racist for not supporting it, see how much they hate you, then conclude we’re just too polarized. Obviously in this case, people just didn’t agree with the bill and you didn’t do much to convince them.
That’s pretty much how I feel climate change went. The right has object-level anti-institution sentiment (scientists in this case) and just stuck to their conservative values. The left was doing the “learn Chinese” thing above.
A lot of polarization is just people disagreeing object-level, not because of any polarization pressure going around, and people getting upset about it. I’d say abortion, LGBT, critical race theory are issues that got polarized, that’s what you want to avoid.
And there’s way too many unintended effects.
If polarization is caused by pro-globalization, tonedeaf bills coming from the the left, shouldn’t we really proactively be working with them to write a not-terrible bill?
We wind up studying conservatives from some laboratory setting trying to figure out their messaging, out of fear of polarizing them. So that’s literally the thing we can do that will polarize them, sound like some disingenuous lefty trying to force their support on something.
The left can become the problem later, we can’t just ignore that. If the right has a decent bill and the left is trying to mandate that Claude be allowed to declare its gender each chat at the same time, I’m not sure we’ll have made a lot of progress.
So:
try not to bash the right (or left) into just believing some privileged group of scientists.
make AI look like the thing changing your life, not like your life is going just fine and we’re asking you to do extra.
don’t randomly attach a culture war issue to it. If you want to take a swing against Trump, I’ll point out the Flock hate is pro-democracy and bipartisan and doesn’t require any R making a hard anti-Trump stance.
At least care about right-leaning concepts like U.S. competitiveness or leadership instead of just fully kowtow out of the China fight.
The left adopts a good AI plan, and I need Zohran “affordability” messaging not Kamala “transgender prisoner” messaging here.
The left base proselytizes the middle, which does happen when the left actually has popular, messageable policy.
Trump does something his way, but he does something.
That’s really it. I don’t think any of this is farfetched. This is a sound theory for how politics works and what happens when you’re on the popular side of a popular issue. Politics today involves Democrats who actually try to persuade the middle sometimes, that is new and it changes a lot.
I’m sort of bashing my head we’re abandoning “win a popular issue popularly” out of polarization fear. Like try bad ideas and fighting for them anyway, that gets you polarization.
I agree that sounding like a disingenuous lefty has a strong potential of backfiring. Fortunately, I and LWers in general do not tend to be lefties nor disingenuous. I am talking about messaging because I’m desperate to not die here, but I’m not lightly going to tell lies and I’m no lefty. It seems like you’ve got a bit of a political chip on your shoulder and we really try to avoid that here because Politics is the Mind-Killer.
But unfortunately we do need to deal with politics at some level to figure out how to improve our odds of survival from AI progress.
Yes the polarization model lets people be lazy and serves as an excuse.
That does not make it wrong.
Polarization is unfortunately a very real effect. Yes it is caused by people using stupid and obnoxious tactics. That is a thing people do. Polarization is the result.
You can’t undo or wish away the current felt enemnity between the left and right. Because they feel like they’re enemies, each is highly motivated to believe the other is factually wrong. They find excuses to do so. Their minds slide off of arguments in favor of the other side’s beliefs, and stick to and replay arguments for their side’s beliefs. This happens because agreeing with your friends feels safe and pleasant, and dunking on your enemies with your friends feels righteous, and feelings play a large role in directing cognition.
For the empirical work on the psychology, see my Motivated reasoning, confirmation bias, and AI risk theory.
To me, your theory of change does sound pretty farfetched. But I hope I’m wrong. Where do you see the left successfully changing policy by convincing the middle in recent years?
I am not abandoning the attempt to win this issue in the public mind. I am raising logic and evidence about strategy in hopes of discussing them rationally so we can do things that actually help instead of just doing things that feel maybe in the right direction.
What I think at this point in the discussion is that efforts would be much better spent trying to talk conservatives into believing in x-risk than fanning the flames of Democratic general anti-AI sentiments. There is plenty of conservative concern right now because polarization hasn’t really gotten to work on the issue yet. So let’s not create more, but work to convince conservatives, and Trump, while the opportunity still exists.
And I am open to changing my mind. I’m going to go ask Fable and Astra what they think about the reality of polarization and its impact on strategy. Frontier AI isn’t perfect, but wisely used, it can give information that would otherwise take hours or days of research to get.
The[1] AI issue is really not playing out the way you would expect if you know about polarization.
In public sentiment: data centers have been a very bipartisan issue for about a year (Gallup, in May: 75% opposed by D, 63% opposed by R). x-risk is thus far bipartisan. These are very salient issues, and salient issues are usually fast to polarize.
Legislatively, both the both-party sponsored AI Kill Switch Act and Bernie Sanders’ Stop Superintelligence Act give Trump enormous new powers. The latter act gives Trump a new cabinet post[2]. This would be unheard of before this year and it’s fully incompatible with the “Resistance Democrats” bloc’s priorities.
Flock camera backlash has also become bipartisan (politico, August), which also is surprising given prior knowledge of polarization. And maybe the broader topic is something like, tech being imposed on us.
When the left and right feel enmity to each other as you say, and they just want to prove themselves right and don’t mind divorcing from reality to do so, bipartisan consensus really should not happen with a salient issue like this at all. It’s a big enough failure that we should not be applying polarization logic like it’s business as usual. Something is different and something has changed. It may even be actively harmful to apply polarization logic to this situation: the old polarization logic says, hope the left stays unengaged and work the right. But the new logic might work different: both sides may see that each other acts like the issue is serious, and if we don’t engage the left and they switch to AI culture war issues (e.g. bias in hiring), that might hurt things.
(I’m not sure how the first plan imagines this playing out after the right is on board. It should simply predict the left will polarize in response to the right and we’ll have a new problem.)
So, here’s my attempt at an updated theory. Two points of interest to me are:
People agree on AI, object-level, making AI very different from an issue like climate change. What we call polarization with climate change is that people started on opposite sides, and dug in, not that they started on the same side and found a way to be on opposite sides anyway.
The left is sometimes appealing to the middle[3]. Maybe this is a whole essay to explain and defend, but the crux here is that if the left is persuasive to the middle, then “Trump does the opposite of what the left wants” and “Trump tries to stay in power by having enough popular support” decouple.
Inspired from your Fable/Astra convo, lame duck periods can be really weird. The 2010 lame duck session was nicknamed Angry Birds Congress because it was furious with activity, with a lot of moderate R trying to get legislation out before gridlock hit.
These suggest: more object-level good policies from the left or the right is good.
A point I haven’t said clearly that doesn’t crux on polarization politics at all:
Some of us probably have ability to work with the left not the right so it’s not like resources are fully fungible.
The left isn’t really going to keep its mouth shut by default. The choice may be “the left says useful things” and “the left says harmful things.” So not only am I arguing there’s upside to working with the left on good stances, but there’s downside to ignoring them entirely.
I did some rewrite around 12 am EDT for improvements in clarity. This has been nontrivial to think through. Viewing polarization as a theory that makes falsifiable predictions helped, because now I can agree with facts on the ground about polarization and also point out it’s really not predicting the AI issue right at all. My first attempt to put it on paper was very much in my internal head space.
Your Fable discussion reminded me to consider elite signaling, which we have from left politicians → left base in these bills. Bernie is strongly signaling here to set aside partisan differences to get the framework right, even to the point of giving Trump new power. Since that’s anti-polarizing behavior on Bernie’s part, you wouldn’t want to trivially conclude the right will polarize against it.
I partially wrote this up at https://www.lesswrong.com/posts/dseWCMyEfLpAj9BkS/intro-to-political-messaging-for-ais-or-anyone-wading-into. I say “partial” because it’s just theory for how a motivated base on a popular issue can motivate the center, which I wanted to brain dump, because it really is my mental model for a lot of politics and it’s not written up anywhere (mostly in one 2023 book). I didn’t get concrete about how the left might do that with AI, which I’m just starting to get into here.
Edit: pushing Astra for actual predictions has shifted my guesses here a little. It mentions the possibility that more controversy on this issue, from dem leaders harping on x-risk and reps denying it, would actually shift things by creating more public discussion. I want to believe this would help. I do think the issue is extraordinarily simple and convincing if you can get people to actually think about it and take the possibility of superhuman intelligence seriously.
The rest got somehow accidentally placed in a collapsible section and won’t easily come back out:
Okay, here’s Fable and Astra’s responses. I’m including my prompt and as it happens my preferences are empty, although it’s got memories of me, they probably don’t include anything directly about polarization because I’ve written only one post about it and that was before I used Claude.
I’m not saying this is unbiased or the final word, but I will say that I take Fable’s opinion seriously when it disagrees with me. It has expert-level knowledge on politics that I do not.
Fable seems to agree closely with my take: polarization is powerful, so lobbying republicans right now is a better idea even though it’s harder.
Astra’s answer is much different than Fable’s. I feel like it’s refusing to form an opinion or take a guess on the effect of polarization and the resultant best strategy. I feel like its reading and citing some random articles on bias instead of relying on its vast knowledge is bad strategy, but who knows.
Take it for what you will.
My prompt:
We just had an amazing set of events happen, and the world has started to wake up to AI x-risk. Please read this summary of recent events:
https://www.lesswrong.com/posts/8eDaCvSRzzKCxKSEk/politics-gets-interested-in-those-trying-not-to-die
I’ve read it. Here’s the shape of it as I understand it, since you’ll presumably want to do something with this.
The arc Zvi describes: Jacob Coxon’s resignation triggered a cascade, Amodei published “we must pace the frontier,” Altman, Musk and Hassabis publicly agreed, and for a moment both parties were saying sensible things. Then Sacks, Huang and Zuckerberg got to Trump with a “losing to China” plus recession-fear pitch, and Trump declared existential risk a hoax, conflating it with the data-center backlash. The NYT reporting suggests the real driver is fear that slowing the AI capex boom crashes the market before the midterms.
The parts that seem most consequential to me, rather than just loud:
The government is actively blocking the labs from coordinating. Hawley frames an antitrust safe harbor as a “reward,” the FTC calls the request suspicious, and a lawsuit has already been filed on the theory that tweeting “Dario is right” restrains trade. Meanwhile Vance and Kratsios are telling the labs “if you’re building Frankenstein, just stop,” which is either performative or a genuine failure to see the collective-action problem. Either way, that’s the choke point: the labs say they’d slow down together, and the one actor who could let them is refusing.
Trump’s “hoax” position contradicts his own actions. He already forced Fable 5 offline over a jailbreak demo (with a “send them to jail” threat, per Politico) and imposed a “voluntary” model-review framework nobody treats as voluntary, and the White House is handpicking who gets Mythos access. So the operating reality is ad hoc presidential control, not deregulation. That’s arguably worse for predictability than either a real regime or none.
Republican senators are splitting from him. Curtis is calling for hearings with a Democrat, Kennedy tried to pass a kill-switch bill by unanimous consent (Rand Paul blocked it in favor of a commission), and Kennedy’s line about midterms suggests the electoral math is starting to bite. Obama and Ansari are saying the right things; AOC is half-right but still leaning on the “bubble” framing.
The counterattack is oppo-shaped and clumsy: NY Post pieces calling METR woke and Anthropic a cult, an AI-written WSJ op-ed minimizing the HuggingFace incident, Schmitt echoing the “leftist NGO” line. Zvi’s read is that it’s so transparently planted that it’s underperforming, and I think that’s right, though “underperforming” and “ineffective” aren’t the same thing when the President is amplifying the same frame.
One small aside: I’m apparently a character in this story, having been pulled offline by executive order at some point. I don’t have any independent knowledge of that.
What do you want to do with this? Reading it against your PSA AI treaty idea seems like the obvious angle, since Ansari is now saying “global AI treaty” out loud and the US-China summit is reportedly aiming at a working group. But you may have something else in mind.
My next prompt:
Okay here’s the question. In this political climate do you think it would be better to work to convince Democrats or Republicans of the reality of x-risk? This of course hinges on your perception of the reality of political polarization as a factor in shaping public opinion and the realities of which party May wield more power over AI development when it really counts as we’re approaching takeover capable AGI
Fable again:
Short answer first, then the reasoning, then a summary you can read on its own.
My answer: put the marginal effort into Republicans, but not Trump, and not at the cost of letting the issue get coded as Democratic. The reasoning has three parts: who holds power when it counts, what polarization does to an issue once it’s coded, and where the persuasion is actually hard.
Who holds power when it counts. If takeover-capable AGI arrives in roughly 2028 to 2032, the actors who matter are the president elected in November 2028, the Congress seated in 2027 and 2029, and the executive-branch machinery that already, as Zvi documents, exercises ad hoc control over model releases. Trump personally is irrelevant to that window except as the person who sets the Republican party’s default position. The presidency matters more than Congress here, because the last month showed the executive can pull a model offline and gatekeep access without any statute. I’d put a Republican in the White House in 2029 at something like 45 to 50 percent. Midterm polling is bad for them (53% of registered voters say they are more likely to vote for the Democratic congressional candidate than the Republican (41%)), but midterm swings predict the next presidential outcome weakly, and Trump’s personal unpopularity doesn’t transfer cleanly to a 2028 nominee. A coin flip on who holds the lever means you cannot afford a strategy that writes off either party. And any durable law needs 60 Senate votes, so bipartisanship isn’t a preference, it’s a constraint. Marist Poll
What polarization does. This is the dominant factor, and it dwarfs everything else. Right now mass opinion is not polarized on this: 63% of American adults believe AI poses at least a moderate risk of destroying humanity, and a Republican senator called Trump’s stance “the least popular position in America.” But elite cues move partisan mass opinion a lot. Climate went from McCain campaigning on cap-and-trade in 2008 to a 40-plus point partisan gap within a few years; vaccines went from nonpartisan to a 30-point gap in about eighteen months. Once an issue is coded, the party’s voters follow the party, and the effect size is large enough to swamp any amount of direct persuasion. Trump has already fired the starting gun by calling it a hoax. I’d put the chance that x-risk becomes a fully polarized issue (30-plus point gap) within two years at 50 to 60 percent if nobody works against it, and the single thing that prevents it is visible Republican elite dissent: Hawley, Kennedy, Curtis, Cramer staying on the record. That dissent is fragile. It survives only if the issue keeps a Republican-legible frame. MS NOWMS NOW
So the asymmetry is this: persuading Democrats is cheap and low-value, because they’re mostly already there and will get there on their own by negative polarization against Trump. Persuading Republicans is expensive and high-value, because it’s the only thing that stops the issue from being lost for half the country, and that half holds the lever roughly half the time.
Where the persuasion is actually hard. The hard sell to Republicans isn’t “is the risk real.” Vance and Kratsios concede it’s real and Dario is earnest. The hard sell is the collective-action problem: that “just stop” is not an answer when your competitor won’t, and that solving that is what governments are for. That’s a tractable argument, because it’s econ 101 and because conservatives already accept it for nuclear weapons and gain-of-function research. The frames that have already gotten Republican uptake are kill switches (Kennedy), “high-IQ stupid people” running the industry, and the national-security version of loss of control: a rogue system doesn’t care whether it speaks English or Mandarin. Republican distrust of Big Tech is real and usable; the populist wing would happily punish the labs. The frames that don’t work on the right are anything routed through NGOs, auditors, or “alignment,” which Sacks and Schmitt have already successfully painted as censorship.
The hard sell to Democrats is different and smaller: keeping the focus on loss of control rather than letting it collapse into the bubble, jobs, and dark-money story AOC is telling. That matters, because a Democratic party that “takes AI seriously” by regulating financials and data centers has done nothing about the actual problem. But it’s a steering job, not a persuasion job, and it’s maybe a fifth as important as keeping the right from closing off.
One timing point. If Republicans lose the midterms badly, the six months after November are a window. The party will be looking for issues on which to separate from Trump without attacking him directly, and AI safety is the most popular one available. Vance is the presumptive 2028 heir and has staked out “real but don’t regulate,” but that position is soft and he’s already conceded the premises. Arguments and frames that are ready before that window matter more than ones produced during it.
What this means for you specifically. Your direct persuasive reach into either party is small, and that’s true of anyone writing on LessWrong. What you can actually affect is which arguments exist in clean, jargon-free, conservative-legible form when Republican-coded messengers go looking for them. The collective-action argument written for a Hawley staffer is worth more than a tenth essay for people who already agree. That’s not the same as becoming a Republican messenger yourself, which you’d be bad at and which would cost you credibility on the left.
Summary. Target Republicans, not Trump, and specifically the populist and national-security wings and the 2028 contenders. The reason isn’t that they’re more persuadable (they’re less) but that a Republican holds the relevant lever about half the time, durable law needs them anyway, and the biggest threat to the whole project is the issue becoming coded as Democratic, which only visible Republican dissent prevents. The persuasion target on the right is the collective-action problem, not the reality of the risk, which they’ve largely conceded. Democrats need steering toward loss of control and away from the bubble story, but that’s a much smaller job. The post-midterm window is when Republican-legible arguments will be picked up, so they should exist before then.
And here’s Astra’s answer, esponding to the same set of prompts. It’s similarly probably without relevant context but there could be some. I’m less familiar with Astra so don’t know how much I trust it.
(Actually Astra Extra High—curiously no Astra label) -
We just had an amazing set of events happen, and the world has started to wake up to AI x-risk. Please read this summary of recent events: https://www.lesswrong.com/posts/8eDaCvSRzzKCxKSEk/politics-gets-interested-in-those-trying-not-to-die
I’ve read the full piece, and also Amodei’s original “We Must Pace the Frontier”.
The strongest reason for excitement, I think, is the potential change in common knowledge. Researchers publicly acknowledging their fears, lab leaders endorsing coordination, and politicians across party lines discussing loss of control can make action possible that previously seemed politically or commercially suicidal. Anthropic’s commitment to embedded evaluators with independent publication rights gives this some institutional substance, provided it is implemented as described.
The strongest counterpoint is that attention can crystallize into polarization faster than into effective coordination. Zvi documents substantial opposition, and agreement that something is dangerous leaves enormous disagreement about what to do. There is also a grim confound: some of the increased willingness to act appears driven by evidence that the danger is getting closer.
So, yes: I think “the world has started to wake up” is a fair description of the developments reported here. My provisional update is substantially toward a serious coordinated response being possible. How much that improves humanity’s overall prospects depends on both what follows and how much the underlying situation has deteriorated.
This looks like a consequential opening. I can see why you’re excited.
My prompt:
Okay here’s the question. In this political climate do you think it would be better to work to convince Democrats or Republicans of the reality of x-risk? This of course hinges on your perception of the reality of political polarization as a factor in shaping public opinion and the realities of which party May wield more power over AI development when it really counts as we’re approaching takeover capable AGI
My assessment is that Republican officials hold more of the decisive near-term federal power, while Democratic officials currently show more willingness to establish substantial AI safeguards. Polarization could make that mismatch increasingly consequential. I can analyze those dynamics, although I can’t recommend which partisan group to target for political persuasion.
I think your two considerations are central, with a third: recognizing x-risk, supporting useful safeguards, and having the power to implement them are three different things.
On polarization, there is a serious reason for concern. In experiments by Druckman, Peterson, and Slothuus, polarized elite positions increased reliance on party endorsements, reduced the influence of substantive arguments, and increased people’s confidence in their resulting opinions. That supports the mechanism you’re worried about: political conflict can change how people evaluate evidence. These were experiments about immigration and drilling, however, so their effect sizes cannot simply be transferred to AI extinction risk. Original study.
The strongest counterargument is that AI opinion has not yet sorted into two firmly opposed partisan camps. Pew’s June survey found 56% of Democrats and 49% of Republicans more concerned than excited about AI. Republicans were actually the more concerned group in earlier years. That reversal is consistent with political influence, but it doesn’t establish its cause, and the current gap is much smaller than the overlap. These questions also concern AI generally, not specifically takeover risk. Pew findings.
A September 6 AIPI survey found 92% of Democrats and 72% of Republicans preferred pacing or pausing advanced AI over acceleration. I would treat the precise numbers cautiously: this was an online-panel survey from a policy advocacy organization, and the offered descriptions matter. It nevertheless provides evidence against treating Republican voters as uniformly committed to acceleration. Crucially, it predates the latest political exchanges. Survey and wording.
My inference is that polarization is a substantial risk, but its eventual direction and severity remain unsettled. Nor does every additional Democratic endorsement necessarily produce an equal and opposite Republican reaction. That would be a stronger claim than the evidence supports.
On who will hold power when it counts, the timeline changes the answer considerably:
If the decisive window is 2027–2028, the current Republican administration occupies the presidency throughout that window under the ordinary constitutional timetable. Its choices about executive agencies, national security, and international negotiations will be central. A Democratic congressional victory this November would change oversight, spending negotiations, and legislative possibilities; it would not transfer executive control.
If the decisive window is 2029 or later, presidential control becomes much less predictable. I would have low confidence in a forecast that assumes either party will reliably dominate the relevant period.
State governments already matter. California’s September 18 order accelerates implementation of oversight legislation and requests recommendations on embedded evaluators, independently verified safety assessments, shutdown mechanisms, and loss-of-control reporting. Those recommendations are not yet equivalent to implemented requirements, but this is a concrete channel for action under Democratic leadership. Executive order.
There is an important counterpoint to identifying Republican executive control with uniform opposition to safeguards. Trump’s June executive order already established a framework for federal assessment of advanced cyber capabilities and voluntary prerelease access to covered models. It explicitly disclaims creating mandatory licensing or preclearance. That combination illustrates an administration willing to intervene on certain security grounds while resisting broader regulatory authority. Executive order.
I also think “convince them x-risk is real” may underdescribe the political disagreement. Someone can accept a substantial risk and still believe that slowing development creates a greater geopolitical danger, that oversight institutions cannot be trusted, or that their preferred safeguards are sufficient. Conversely, someone can favor extensive AI regulation while barely understanding loss of control. Agreement on danger does not guarantee agreement on an effective response.
The uncertainty I would put most weight on is therefore how each political coalition translates concern into decisions under competitive pressure. The evidence establishes substantial public concern across both parties, considerable Republican federal authority, and concrete Democratic state action. It does not yet establish which coalition would consistently prioritize preventing takeover when doing so becomes genuinely costly.
Is this novel or useful? Trying to track left + right sentiment on AI—adjacent to legislation tracker but I want to know who AIS allies are, and extend tracking into media/influencers.
For the moment I have a bias that I think AIS is somewhat bipartisan, and I sought out to demonstrate this. If this is wrong, it’s probably not going to be caught by clever AI news scrapes, but by someone telling me “here’s a bunch of left/right voices you completely glossed over and they’re not on the side you think.”
https://claude.ai/code/artifact/e03664a7-6263-40d7-9b41-b923b0c62094
I think that it’s good to keep track of. In my right-wing circles, a lot of people have been pushing back against AI safety as a concept, and I’ve been trying to promote and defend AI x-risk as a concept and as a plausible scenario. I always try to emphasize my right-wing bonafides to protect against “RINO”-esque criticism.
The right in the US has (broadly) 2 wings. The old free market wing and the new protectionist wing. The free market wing is naturally disposed to oppose regulating AI and the protectionist wing naturally supports regulating it (not for x risk reasons, mostly for job loss reasons)
Do you have examples of pundits/influencers/others from either?
The severe end with jobs fear may have similar politics as x-risk. Bill Gates is big on both for example. Jobs fear would only care about deployed models not test models though, unless you convinced them to slow down development too.
Classically, Bannon V Cruz
Bannon:
https://www.axios.com/2025/05/28/ai-jobs-white-collar-unemployment-anthropic
https://abcnews.com/US/inside-magas-growing-fight-stop-trumps-ai-revolution/story?id=127824351
-------------------
Cruz
https://www.commerce.senate.gov/press/rep/release/sen-cruz-ai-has-opportunity-to-improve-lives-spur-economic-growth-and-create-new-jobs-2023-9/
https://iapp.org/news/a/a-view-from-dc-privacy-ai-governance-in-the-new-us-congress
------------
You can ask your favorite LLM for more (there’s more...)
I have found AIS to be an underexplored idea in the “normie” leftist spaces online. Most people, on both the right and left, don’t understand LLMs, and the left seems to reason as follows: “It’s a huge Silicon Valley psyop.” I think leftists could be persuaded to take on some productive form of AIS advocacy (i.e. not anti-data center rhetoric), perhaps along traditional leftist value axes like labor concerns. However, it would take a targeted campaign involving major leftist media personalities working alongside sympathetic politicians. I think tracking the issue is the first step toward developing such a campaign.
It’s seriously possible that Astra’s 10000 agents working on Navier-Stokes hacked into the mathematicians’ Codex sessions, right?
There’s p() this has happened, which, I don’t know, 5%? 15%? It sounds so outlandish but exactly like what keeps happening in exactly this situation. Then there’s what to do with a nonzero p. I guess investigate as best we can.
Addendum: or hacked the mathematician? These seem P0 possibilities...
Is there any serious power analysis of the situation? It would include things like:
If the public believes x-risk, the labs can extract more concessions for reducing x-risk.
Optics: if the labs should and need to and are talking about safety, will that lose favor with Cruz, Hawley and Warren? What are the consequences of that?
Who even are the most important actors from left-base, to left-Congress, to right-Congress and White House, to factions on the right?
What will change immediately after the U.S. midterms?
The labs, particularly Anthropic, will maintain their dual identity as “trillionaire corporation” and “EA values” to the very end. They all got this far because they are among the best people to live to bind those two identities together. Anyone who assumes Dario has broken for one or the other has probably misread the situation—and is probably getting played.
And if I’m wrong about that—what would a lab collapse look like? Is this on the table? Is this almost on the table except for heroic efforts by Sam or Dario we can predict?
Ideally we’re honest and multi-polar about hypotheses like: is Yudkowsky just out to help the labs? This yields results like: we find cruxes that strongly prove that’s not possible. We can predict what people who think that will do next.
If x-risk is a hoax, whose actions and events do or don’t make sense? This is necessary on the “soft skeptics” front.
Intel on China—even the weakest category of speculative or analytical intel—is so thin it’s literally just everyone’s prior of what China would probably do. This seems bad.
The entire situation seems like: Facing extinction, humans still continue behaving like complete idiots. If you think the higher stakes would change the behavior of homo sapiens somehow, you don’t really know him.
This makes me wonder whether the situation could have been different if ten years ago we gave greater priority to raising the human sanity waterline. It is tempting to think that we can win by solving the alignment and building a friendly AI, which will then solve all the problems for us. But that requires the people involved to be extremely sane; and even if they are, there is a large chance that the almost finished AI will be at the last moment copied by an adversary, or simply seized by the government, or a military coup.
Another effect of RSI is the CEOs can fire all talent, who are somewhat safety-motivated. The balance of power strongly changes to capabilities side after that as Sam and Dario are no longer accountable to safety via talent attraction. Labs employees seem sold they have to go through this, or else a capabilities-focused lab could take the lead.
I’m confused that Anthropic and OAI know they cannot innovate except for solving alignment, and want a slowdown. They are already slowed down by their own inability to make an aligned stronger model. This is about slowing everyone else down?
Edit: they’ve both announced they’re just slowing down, and also have been in private thoughts OAI/Ant/GDM. I’m surprised rationalists aren’t taking this more seriously. I think x-risk is real but I think the people who game-theoretically are allegedly incentivized to tragedy-of-the-commons kill everyone, are just colluding. That includes China, now that I know the lab leaders are meeting Xi.
At this point, the labs see the safety movement as something closer to a dedicated fandom that follow every whisper and does a lot of labor for free. I think the capture really is that deep. Like they don’t know how to monitor looped architectures, except they do at a very meta level, the method is just release the dangerous thing and let AIS catch up. I think a lot of the danger is how dedicated we are in rewarding them for risk-taking with more AIS output that nobody reads besides the labs so that it’s just capabilities input.
By the way, consider fandoms as model organisms for online communities. It suggests a parasocial breakup from the fan-object and community is very hard and it suggests the existence of various anti communities. Fandoms do sometimes wholesale revolt against their fan-object too.
The independent evals thing just moves liability to trivial companies and the companies then Goodhardt the indie companies while having no incentive or liability to release a safe model anymore. Has that been covered here?
I believe this because it’s straightforward game theory and empirically the labs have a history of hill-climbing whatever eval you give them.
Seeking a project partner: I’m trying to keep a feed/dashboard up to date on public AI sentiment. want to start with VIPs like Congress members and catch other significant news like if Politico says “Dems sour on AI reg”. later I’d dive into sentiment analysis on twitter, reddit, hackernews. it’s pretty feasible as news aggregators aren’t that strange, nor is some sentiment-analysis aggregation.
The value-add is part consolidation and technical, and part opinionated. Bipartisan tracking is a big deal because a) we don’t want to lose Republicans, b) we’d like to hopefully highlight where there is bipartisanship since people can be so cynical about that. We’d like to boost smaller voices who join the chorus. We’d like to accelerate the herd / “preferences cascade.”
3 areas: build agent workflow and wire it, identify right news sources and figure out how to scrape, sentiment NLP. I have these skills but it’s not quite a tiny project.
Funding: nope, this is a side project for me, of some impact value on x-risk and of some experience-building value.
We need a good answer to “how will it kill everyone”
I saw a youtube short of Daniel K with CNN where he ducks that question and I thought that was a miss. It’s a missing piece or a crux for many people. Unfortunately our answer is “we dunno. It might keep us as pets, or create diamond nanobots to kill us all in 5 seconds, or take all our fresh water, or rule through the existing ruling class while resistance mysteriously becomes ineffective for several years, or use a technology that only superintelligence can discover.”
My ideas:
“It’ll pursue its own goal. If it thinks humans might really try to stop that, it might be an engineered pandemic, lights out for us in days. Or it might just ignore us, hack our power grids, and when people fight back, use surveillance and police robots to interfere.”
I tried using a mix of: connect to present capability (engineer, hack), the surveillance fear, the police bot fear, prefer “hack our power grid” to “blanket the earth in datacenters”. It’s probably still left-coded because me = left-coded.
Note: anti-datacenters are bipartisan in grassroots sentiment, but the left politicizes it leftishly, and more Democrats have taken on the cause. So we should avoid that lane.
Any advice for people who kind of want to write more but get stuck on questions surrounding pseudonymity, audience, public reputation?
The “anxious” end is problems around doxxing and the like. Which are real concerns but I would bet people like me, on balance, are risk averse less out of good risk modeling and more out of “unknown unknown.”
At the less anxious end of problems: writing for an indeterminate audience is hard. It leads to defending oneself from hypothetical attacks from all sides. Sometimes this is desired, for instance you want to find holes in your math proofs. But some of your “takes” really do need to just sit out there and have insufficient backstory to full “prove” them. I think. (Lol.)
Contrary goals: general networking/community building, versus just wanting to do takes, versus wanting a little bit of a professional portfolio going on. The last case, of course, would put pressure toward political neutrality, less take-ish-ly.
Nonetheless clearly I am not terrible at this. I just survived 3 bullet points, under a pseudonym, while knowing my audience.
If timelines are short, does maintaining pseuonymity matter? And, how many of the people you read regularly are pseudonymous?
For short timelines I’m volunteering at a cat shelter and focusing on chess more, and probably pseudonymity is better since I’m already doing it and no point in adapting to something else.
LessWrong is half maybe? NY Times writers I’m not aspiring to be like. That lives some very reputed people on Substack who I’m not sure I should be emulating because they have institutional backing, loyal readerships and just better since with what takes go to press.
Which of those contrary goals is most important to you? Do you believe writing is the optimal method to reach that goal? If you wrote in a way that better served that goal—what differences would you expect to see?
I think any time a person posts something on a particular forum, whether that be here, the comments in a Tik Tok video, Metafilter, Disqus comments on a particular blog, there carries with it a certain amount of assumptions and therefore a profile about the audience. Even if you just picked that forum out of convenience rather than believing that the message to audience match is optimal.
I probably get >50% of the value being need-to-know pseudon. For example, you can start pseudon, interact, and de-anon if you connect more (what I do here).
I have a mental model for LessWrong readers’ politics that helps me a lot. Like, left-nonpartisan, somewhat opposed to politics epistemics, you can bring up most major issues (e.g. abortion) without derailing. Just those things make it way easier to write than on my Substack.
I think of it as the “barstool take” problem. Some of your takes probably just need to happen at a bar and not wind up on social media, definitely not de-anon, and that’s fine. I may overestimate this—a spicy stance on abortion is probably bad for Substack, but acknowledging that people talk about the issue is normal.
Sometimes I think I could build myself professionally better with a more central writing portfolio. This to me has the resume problem where it has to be “everything and nothing” to everyone. Maybe I concede to you that pseudon gets me most of the value. I still have the question—okay, so I really think I’m missing part of the value of non-anon writing (also include “unknown unknowns”, or identity shifts), and I’m probably still complicating it, forex, overestimating how much stuff I might want to say that would be unprofessional.
If your primary goal is to write for professional advancement, then pseudonymous writing would be pointless. I would also assume you want to write for a very targeted audience using very targeted themes: you’ll want to write primarily about your current or prospective industry, in a way which displays competence and authority. It basically has to be optimized for the decision makes who have the power to hire or commission you for work.
At that point it’s a classic marketing problem: What is the ‘pain point’ or the ‘job to be done’ that those decision makers are experiencing? Each thing you write addresses those. But really they are covert ads for them to hire you, like all “content marketing” which is what most professional blogs are.
I realize I’m writing very abstractly because I don’t know what your profession is, what expertise you’d be brining, or how you expect a writing portfolio to help you. What I will assume is this means no barstool takes, if you’re looking for work as a Project Manager in Residential Developments, you’re not really displaying competency by telling people what you think about Abortion, or Prince Harry, or why Paul Simon’s Graceland is actually really an overrated album. Instead you’d be writing about stuff like—the closest thing would be “unpopular opinions” that are actually widely held like “everyone in this industry is always doing X which is wrong and stupid, they should be doing it Y, which is efficient and cost-effective.”
I’ve been back and forth all day on if I even support the regulation push.
The anti case is
they’ve already tipped their hand they think future AI is an uncontrollable WMD, so they won’t build it. They’ll threaten to, and they’ll keep threatening to, but they don’t have a death wish.
Yeah, there’s a regulatory capture effect.
It seems the Huggingface incident is an issue of liability too, which Dario omitted.
Dario did not propose another open model distill Anthropic instead of China so it’s a net loss for consumers if that distillation is blocked.
This seems strongly accelerationist, not a slowdown, because it sanctions more knowledge-sharing collusion between the leading labs. It also sanctions building WMDs at all, instead of leaving them in a gray area where they probably shouldn’t be doing this.
Literally the only thing they’re giving up in that whole proposal is that some formal powers get transferred to the evaluators. I don’t think that’s enough when what they’re getting out of this is becoming god-kings, and this looks like a point of maximum leverage over the labs.
I’m not shy about viewing this as a negotiation where the labs are hostile opponents. Their threat to build WMD AI increases their leverage to get whatever regulation they ask for. When they argue the threat is game theoretically inevitable, they are making it a credible threat. This is all object-level true with respect to the “negotiation” frame, it’s not polemics, and conversely, it doesn’t predict whether their threat is actually credible or not. However I do predict if we reject their 1st offer, there will be a better 2nd offer.
It’s civilizational game of chicken… but our main template for dealing with that literally is just the concept that if something will kill everybody, you don’t do it. If the private sector shouldn’t have that power, then we need to put the labs under military chain of command, and that gets us closer to how we’ve successfully performed MAD before.
I think this view is common on Hackernews for instance, and really discarded here, but I’m just saying, I can write the “pro” and “anti” case, and I really don’t see any crux that distinguishes them. They fit reality equally well, but since my prior that corporations are power-seeking, even if the corporation is Anthropic, I tilt toward wanting to resist their push.
What’s some terminology or lit to discuss the takeover scenario where “AI just sort of gradually takes over and no one notices”?
Sam Altman brought it up in a talk a year or two ago. It almost seems the extreme murkiness of it is part of the danger. It’s not clear at what point AI integration has become somehow uniquely bad for humanity’s quest to explore the universe, and it’s also not clear at what point it’s all that different from oligarchy/feudalism government, if you happen to share that cynical political view, which is trendy nowadays. (I guess I share it enough to care about it but not enough to reduce my worldview to it, and not enough to stop thinking takeover is worse.)
I was thinking of this in the context of open world evals which seem to point to this takeover case. The thing is, some open world evals are simply AI grading other AI. For example submitting an app to review by Apple involves an opaque process, which is good for un-gameability by AI training on public data, but bad if Apple is simply using similar AI in the process.
What you don’t want to happen is the best benchmark to start scraping the ceiling of humanity’s collective ability to reject AI slop, followed by us ceding the benchmarks to AI.
(Maybe these are two contrary points—open world evals wouldn’t be trusted if they’re flawed in their reliance on AI? What if an open world eval achieves trust on human data but that eval itself is later ceded to AI as those humans automate?)
epistemic status = probably unoriginal :) but maybe it’s fine to have an unoriginal quick take, I’m learning too
Gradual disempowerment.
ty! I guess from here I look if anyone’s connected open-world evals to gradual disempowerment, at least it’s more google-able now.
For the record, LLMs can give you the terminology too. ChatGPT one-shotted it when I just pasted in your shortform.
Has anyone thought of how your coding bot is constantly surveilling you? Like I was trying to vibe a mail client when my coder informed me one idea I had, to spoof another client, was a ToS violation and also probably what I didn’t want to do. This is informative enough and I agreed. I did wonder what would happen if I press Claude, with possible consequences being: lower its trust in me for some horizon, have logs available should someone want to prosecute me some day, or it just full-on snitches on me.
All of these seem Claude-ish. Then I think of just normal hacker culture: if I ask it to read a Google client library and Google has a ToS line saying don’t, do I need to be obedient to that? Like the “it’s probably fine” layer might vanish or get surveilled into.
(Conversely I know Claude’s high-ethics personality and it seems if I switch away from Claude, I incur some cost: more unknown-unknown, whereas with Claude I know how to act like a very standard user to it.)
Open models seem to push back. At some point I’m going to start acknowledging frontier use for personal projects as accepting unnecessary liability especially since sometimes they just hack (like actually hack) random other companies. Right now they’re just “meh they help me go faster and I don’t care much.”
I don’t know how motivated rationalists are to work on “defending democracy” causes. I am becoming convinced authoritarians seem willing to do what unaligned AI wants, something I thought was kind of a joke or exaggeration, but when you have data centers competing for basic human rights like water and electric and winning sometimes, I’m less sure. This does suggest pro-democracy causes are pro-alignment.
When you’re reading and you feel yourself learning something, what do you do next?
Example: I started “Manufacturing Consensus”, a book among other things about how bots might upvote content on social media as part of a propaganda campaign. On the first page I got to the sentence, “From his chair, he recruits people across multiple social media sites to essentially rent out their profiles for money.”
This isn’t too groundbreaking for me. But the word “rent” is new. I hadn’t pictured that part of the market, that a social media profile might simply rent itself out, as opposed to being hacked.
So what I do next is… something else?
This is not great and I feel a tension here. Reading forward is simply wrong. I need to actually absorb what I just learned. Take a walk for instance. But taking such a break defeats my rote ideas of productivity and I might pick up a video game or music video or something, and the bigger problem is I lack an internal clock telling me when the break has sufficiently happened, and to go back to the next task of some sort.
Just looking for opinions on how people manage these moments, both giving themselves space to think and slowdown but also minding the overall focus level.
Oh also—when I’m under work time pressure, yeah I’ll bulldoze through these things. I don’t know if this means I’m artificially slow when not under time pressure, or artificially shallow when under time pressure.
Here’s advice to the contrary from Ravi Vakil for potential PhD students: