More people should write basic AI safety explainers

Linch12 Oct 2026 20:50 UTC
110 points
19 comments3 min readLW link

I think people continue to underrate the value of writing the clearest, most basic, simplest possible case for important parts of AI safety. For example, before I wrote my own explainer for the simplest possible case for AI doom, I noticed that all existing explainers, of which there aren’t many, had multiple of the six problems below:

  1. They have too many extraneous details

  2. They appeal to overly complex analogies

  3. They seem to spend much of their time responding to insider debates and come across as shadow-boxing objections.

  4. They have “meta” stuff prominently in the text.

  5. Often they “sound like science fiction.” I think this plausibly was correct historically but in the year 2026 they don’t need to be.

  6. Often they reference too much insider jargon and language.

I set out to improve on the SOTA. I did not fully succeed[1], but I think it was moderately better. Since then, there have been other, probably better, explainers (Yay!).

But actually this is true not just for the core case but many other parts as well. For example, before I wrote my explainer on recurrent neuralese (which has since been superseded by Scott’s better explainer), the top Google search results for “neuralese” was some combination of a hacker news thread on AI 2027, AI-generated blogs and some tweet that confuses more than it enlightens. Oops.

But maybe neuralese just isn’t that big a deal? Well, we also don’t have a good longform post explaining the details/​mechanics of how AIs can actually kill everyone (this may sound obvious to ppl in the LW crowd, but it’s patently not obvious to the general public[2]!)

We also don’t have great explanations for what RSI even is, never mind why and how to ban it. In my own research area, AI superpersuasion, I see many areas of confusion regularly[3]. And so forth.

A lot of the time I think people just don’t try to explain things clearly and often. In 2019, back when I lived in Silicon Valley, I’d talk to my programmer friends and often see them thinking they understand the case for risk but totally not getting basic concepts. So I’d try to explain what I believe in simple technical concepts that are graspable to programmers. I did it a few times and people quite liked it/​were moderately convinced. I asked a seniorish person in AI safety whether it was worth writing it up and they were unconvinced it’d be worth my time. He was skeptical my article would lead to much since it wouldn’t move people beyond Kelsey’s explainer. I think this is miscalibrated[4]:

  1. Different things convince different people. Kelsey’s a better writer than me, and writing to a general audience. I was targeting programmers/​technical talent.

  2. If something’s sufficiently important, you want “more shots on target”

  3. Mere exposure effect (above some quality bar) is important for something like AI doom. In 2019 (and even today), the amount the public is talking about AI risk is far below the optimal level.

  4. My time in 2019 wasn’t that valuable anyway.

I ended up spending my free time learning more about ML, and writing up why I think climate change is overrated[5]. I enjoyed and learned from those experiences, but overall I think this was a mistake in prioritization. I believe our community is continuing to make this mistake, both in terms of the general case for AI doom and in the particulars of things like what misalignment is, various threat models, what policymakers could do, and so forth.

I briefly helped work on AGI.FYI, and definitely think it’s a net positive project, but imo the biggest bottleneck in comms is the lack of good explainers, rather than the explainers failing to surface to the top.

I understand that many people have very large opportunity costs. Even so, I suspect a) this is worth it to many people who are good communicators and yet believe that their opportunity costs are quite large (for example I think Nate Soares is making the right call in spending a lot of his time on AI doom comms) and b) people with lower opportunity costs should attempt to do this despite on average expecting to produce lower quality content, and bite off different parts of the information pipeline. For example, at AI safety fellowships, people at the 90th percentile of writing/​communications ability or higher should aim to spend around two weeks to write their own version of the “simplest case for AI catastrophe” as they understand it, as well as address confusions about certain concepts that have not been adequately explained. Lurkers/​writers on LessWrong who are unsure about what to do and feeling sad about being unable to contribute directly to technical AI safety research or policymaking should also consider communications as a route to positive impact.

Sometimes people are worried about low-quality comms/​arguments drowning out the good ones, and/​or the bad arguments inoculating you against good ones. I’m skeptical, for three reasons:

  1. In many cases, if you delve[6] into the details, the proper response is “what good ones?” Like there isn’t a better argument to supplant!

  2. I think reality does a decent job of sorting out good arguments on-target from the trash. Both in terms of esteem and in terms of popularity. At least in AI safety, I expect there just aren’t many good arguments that get buried by trash, as evidenced by the top of Google search results, what shows up in algorithmic news feeds, what elites talk about, etc.

  3. To the extent that the bad arguments drowning out the good is a real problem in the future, it probably isn’t that hard to course correct and fix. I think our community has pretty natural systems for doing this.

This also applies to other mediums, like shortform text (Tweets), shortform video, longform video, podcasts, speeches, and so forth[7]. More dakka is good.

  1. ^

    AIs flag my article as having too many inappropriate jokes and gesturing at the last step “This can all end badly” rather than going through the details. I agree, but I believe both problems are structural to my article rather than things I can easily fix with the article as-is. For example I think removing the jokes will harm both readability and theme, and wasting screen time on covering the details of how AI will kill everyone will be net less convincing (for my article specifically) than “this can all end badly.”

  2. ^

    Tbc your SOTA to beat imo is “Terminator but worse.” I think this is a fine baseline, enough to get most reasonable people to be at least a bit concerned. Most explanations are in fact worse than this.

  3. ^

    Obviously it’s partially my fault for not making a “simplest case” argument somewhere for superhuman persuasion risks. I’m part of the problem!

  4. ^

    Though the advice might accidentally have been good in that writing an explainer in 2019 targeted at programmers might’ve had a negative net effect on the world.

  5. ^

    Stefan Schubert mentioned this at the time, saying something like “this is very clear and simple communication on climate change, wish more people did this for AI safety.” I should’ve taken his points more to heart then. Oops.

  6. ^

    What a great and underused word! I genuinely love it.

  7. ^

    Though imo I think within AI safety comms we’re slightly overinvested in being twitter reply-guys[8] and maybe podcasts vs advancing our own position clearly and forcefully. Otoh I think video and speeches are probably underrated relative to longform text. But this is micro-optimizing. The broader message of doing everything more is more critical.

  8. ^

    My general rule of thumb in comms is that it’s usually a mistake to devote time to objections, replies, etc, unless you absolutely have to. Though maybe this is because I spend too much time on replies, and the law of equal and opposite advice can apply to other people.

Linch12 Oct 2026 20:50 UTC
110 points
19 comments3 min readLW link