I’m Michael “Valentine” Smith. Cofounder & senior instructor at CFAR back in the day. I’ve been in the rationalist scene since 2011 but mostly left in late 2018. To the extent that “post-rationalist” means anything, the term should probably apply to me.
You can find my non-LW writing on my Substack. You can also find my social media profiles via my Linktree.
This was a great steelmanning, and is exactly the kind of thing I hope people will do in contact with what I offer. Even though I don’t agree with every detail, I feel received and like the thing I care about is being well enough held. Thank you.
Good call. I haven’t been reading Less Wrong in enough detail for a while to pull this up usefully. My impression comes from in-person conversations plus Twitter interactions. The thickest use of my encountering these terms in rationality circles was admittedly about a decade ago. But I’m not sure how much of that is due to my not spending as much time in rationality circles versus discourse norms moving on. I still encounter it almost solely from folk tied to LW-style rationality.
I don’t recall hearing you use the terms in ways that bothered me this way, FWIW.
it still seems bad to advocate for the exactly wrong policy, especially one that doesn’t make sense even if you turn out to be correct (as habryka points out in the original comment, many think 2028 is not really when most people expect agi to have happened).
I’m super sensitive to framing effects. I notice one here. I could be wrong, and I’m guessing that even if I’m right you didn’t intend it. But I want to push back against it here anyway. Framing effects don’t have to be intentional!
It’s not that I started with what I thought was a wrong or bad policy and tried to advocate for it. It’s that given all the constraints, I thought that preregistering a possibility as a “pause and reconsider” moment might be the most effective and respectful. It’s not what I’d have preferred if things were different. But things aren’t different from how they are, so I made a guess about the best compromise.
I then learned that I’d made some assumptions that weren’t right, and that determining such a pause point that would have collective weight is much more tricky. Alas.
But it was Oliver’s comment that brought this problem to my awareness. At no point did I advocate for what I thought at the time was the wrong policy. I had hope because I thought folk were laying down some timeline predictions that could be falsified soon. Turns out, approximately nope.
i think you would have much more luck advocating for chilling today and citing past evidence to make your case..
Empirically I disagree. That demonstrably has not been within the reach of my skill to do effectively. But it’s a sensible thing to consider trying again sometime.
Although I don’t like comments starting with “your logic slipped” because it sounds passive-aggressive “you are stupid” vibes I will reply.
Sorry, that’s not how I meant it. I meant it more like “Oh, I think your foot slipped there, so if you take another step I think it won’t have the effect you’re looking for.” We can all slip up. It’s intended as a friendly note.
I agree that on rereading it it didn’t come across that way.
So what you are saying is that yes this time is different just not today. It will definately happen and all the doomerism is correct but not on a short timeline because ____ insert reasoning that is different than what the top AI minds are saying today.
Uh, no. That’s not what I’m saying.
I’m saying something more like: if it turns out that doomerism is once again exaggerated, perhaps we should take a step back and ask what’s creating the exaggeration instead of plowing ahead as we have been.
I love this direction of inquiry. It’s tricky to get right because of lots of confounds. But I think something like it should be doable.
I love that this is a space where people care about this stuff.
I’m poorly suited to actually carry out these examinations on my own. But if someone does, or if someone wants to lead the charge and would like me to help design the study, I’d love to hear about it!
I think personally I’m more grim and more traumatized than most people around here, and other people are too happy and not neurotic enough to understand how fucked alignment research is; and yet I have much longer timelines than most people around here.
Two notes:
I think this is a little evidence against the trauma model, but not much. Most forms of trauma don’t cause people to become AI doomers, just like most forms of trauma don’t cause most people to become alcoholics. I think the form of the trauma, and the set of coping mechanisms, and the set of life opportunities, all have to converge to result in this specific flavor of doom focus. (And I hypothesize that LW has long been an attractor for people who’ve been hit with that set!)
I don’t care that much about the trauma model in particular. I should have been clearer in the OP. What I meant was more like, “Gosh it sure seems to me that fixating on doom seems to be a drive independent of truth. Keeps happening all over the place. Sure looks like that’s maybe happening here too. That seems important.” The trauma thing was meant to both (a) highlight the type of phenomenon I’m talking about and (b) give an example of a loosely gearsy mechanism for producing it. I think you’re offering a slightly different model — which is great! I think they could be empirically distinguished (and aren’t mutually exclusive, and the whole scene is probably multifaceted).
Overall I like your comment.
I agree with you.
My best guess is that the degree of doom is exaggerated but not fabricated. The exaggeration matters because if it’s there, it’s warping perception about what to do about the real problem. So if it’s there, it would be ideal to address the cause of the exaggeration, even though on the inside that’s probably always going to feel like the wrong thing to focus on.
In the end, I think a healthy attitude looks more like facing the darkness hand-in-hand with joy in our hearts and music in our throats.
I agree. To be honest I didn’t think chilling out now was a real option. I hoped to encourage it in a few years with the aid of preregistration.
May I suggest a reading of “When Prophecy Fails”?
Oh yeah! My father would talk a lot about that book. And yeah, I think something like it could be very pertinent here. It does keep coming to mind for me.
Assume, for a moment, that scaring people into action is the correct response to imminent doom.
[…]
Note that this is all conditioned on scaring people into action being the the best action conditioned on imminent doom, which I am unsure about.
Noted. FWIW I’m seriously skeptical of whether it’s true. For two main reasons:
The kind of “imminent doom” we’re talking about is years away. People can’t stay actively scared and useful on timescales that long. I think they go numb and adjust instead.
Fear distorts thinking. Saying “But it’s dire!” as an argument for scaring people creates a memetic evolutionary incentive to make that claim. You can severely warp the epistemic commons if the norms around pulling the fire alarm let it happen too easily.
With that said: I think if we were pretty confident that AI doom were coming in the next year, then freaking people out about it might make a lot of sense.
But I think the bar for making that call should be very high. Otherwise bad memes eat us all.
we could possibly use other events to get quicker results.
Yep, sounds like that’d be the wholesome and good-faith thing to do at this point, best as I can tell.
you don’t give much evidence that this particular emotional mechanism is involved as opposed to others.
True. I was trying to suggest a gearsy model of how impressions of doom might get exaggerated and warp perceptions. I’ve seen stuff very much like the trauma model play out best as I can tell. So it seemed like a plausible model here. It’s my go-to default.
If you or someone else would care to spell out a different gearsy model, and if we can distinguish between them, I’d be quite happy to hear it and look at the truth.
In particular you gesture at:
I think that the desire to be part of a secret, specially enlightened in-group is likely a big factor as well.
Intuitively that seems plausible to me. At a glance I think it’d put a different set of incentives on memetic evolution, but that they’d still warp shared vision away from truth.
I’m running out of reply steam, so I’m not going to go much deeper than I just have here. But I want to note that I think your idea makes some sense, and I bet we could tell empirically to what extent it vs. the trauma thing is happening.
The main thing I keep seeing missed when coming up with experiments is, we’re talking about subjective structures. Which means that we’re inside the things we’re trying to analyze. Which means that the scissors we construct to distinguish between worlds have to account for that self-referential element.
I’m reminded of how a bunch of people jumped in on an experiment Scott Alexander spelled out in “The Apologist and the Revolutionary” where dropping ice cold water in one ear had been reported to sort of disable the internal rationalizer for a little while, causing patients to rescind some absurd beliefs for a few minutes — and then later deny that they had rescinded them once the rationalizer came back online. Eliezer in particular went to do the experiment and came back saying that he hadn’t changed his mind about anything notable. Folk had to point out that that’s precisely what the experimental results would have predicted he’d say.
People make this kind of logic error with subjective science all the time. You really do have to account for the fact that you are inside the structure you’re trying to explore, and think through the implications, in order for your experiments to matter.
So, yeah, if we can account for that, then I bet we can tell to what extent this secret in-group drive is playing a role vs. something like the trauma model I currently use as my default go-to.
You describe a pattern in the community, and then you ask if the dire predictions of AI 2027, specifically, are distorted by that pattern.
Er, I don’t think that’s quite what I meant to say. I think it’s close, but this frame feels like it’s requiring me to take a stand I don’t care that much about. Let me see if I can clarify a bit.
My sense is more like, I see a pattern in the community. And I think the inclination to spell out AI 2027 and spread it came from that place. I thought that since AI 2027 was spelling out a pretty specific family of scenarios and in January 2028 we’d be able to tell to what extent those specific scenarios had in fact played out, we could use January 2028 to seriously re-evaluate what’s generating these predictions.
I’m now learning that the situation is more messy than that, and it’s much harder to trigger a “halt, melt, and catch fire” event with regard to producing dire predictions. Alas. I’m hoping we can still do something like it, but it’s not as clean and clear as I’d originally hoped.
Hopefully that clears up what I was saying and trying to do.
Now, you ask:
Imagine if the AI Futures team had nothing to do with the rationality community, but they’d made the same dire predictions. Then 2028 rolls around, and we don’t seem to be any closer to doom. People ask why AI 2027 was wrong. Would you still point to trauma-leading-to-doom-fixation as a hypothesis? (Or, rather, would you point to it just as readily?)
I’m afraid I don’t know what “the AI Futures team” is. So I can’t really answer this question in full honesty.
But with that caveat… I think my answer is “Yes.”
I’m not that picky about which subset of the community is making dire predictions and ringing the doom bell. I’m more attending to the phenomenon that doomy predictions seem to get a memetic fitness boost purely by being doomy. There are other constraints, because this community really does care about truth-tracking. But I still see a pretty non-truth-tracking incentive on memetic evolution here, and I think it’s along an axis that rationalists in general have tended to underestimate the impact of IME.
The trauma thing is a specific gearsy model of how doominess could be a memetic fitness boost independent of truth-tracking. But I’m not that picky about specifically the trauma thing being right. I just don’t know of another good gearsy model, and the trauma thing sure looks plausible to me.
Is that a good-faith response to your question?
Feels like that’s a Motte and Bailey?
I really don’t think so, but I’m not sure why you’re saying so, so maybe I’m missing something. If I keep doing something that looks to you like a motte-and-bailey, could you point out specifically the structure? Like, what looks my motte and what looks like my bailey?
So it’s not just that these are strong arguments. It’s that these are arguments that you might expect to provide some signal even if you thought the claimed effect was strong, but not overwhelmingly so.
Sure, but arguing that doom is real does nothing to say what proportion of the doom models’ spread is due to something akin to the trauma model. And that matters because… well, there’s a different motte-and-bailey structure that can go “OMG it’s SO BAD… but look, here’s the very reasonable argument for how it’s going to be this particular shape of challenging for us… SO IT’S SUPER BAD!!!” It’ll tend to warp how we see those otherwise reasonable arguments, and which ones get emphasized, and how.
Listing more such arguments, or illustrating how compelling they are, doesn’t do anything to suggest how much this warping phenomenon is or isn’t happening.
I mean, by analogy: someone can have a panic attack due to being trapped in a burning building. They’re going to have a hard time doing anything about their situation while having a panic attack. Arguments that they really truly are in a burning building don’t affect the truth of that point, regardless of how compelling those arguments that the building really is on fire are.
Analyzing from the outside, I agree.
The pesky thing is that we can’t fully analyze it from the outside. The analysis itself can be colored by a terror generator that has nothing to do with the objective situation.
So if there’s reason to think there’s a subconscious distortion happening to our collective reasoning, distinguishing between doom and nodoom might functionally have sorting out terror as a prerequisite. Which sucks in terms of “If there’s doom then we don’t want to waste time working on things that aren’t related.” But if we literally cannot tell what’s real due to distortions in perception, then sorting out those perception errors becomes the top priority.
(I’m describing it in black-and-white framing to make the logic clear, not to assert that we literally cannot tell at all what’s going on.)
I’m not sure whether that’s supposed to be compatible with your model of preverbal trauma...?
Sure! It sounds like maybe it’s not what’s going on for you personally. And I still wonder whether it’s what’s going on for a large enough subset (maybe a majority?) of folk who are reacting to AI risk. Which someone without a trauma slot for it might experience as something like:
“Gosh, they really seem kind of freaked out. I mean, sure, it’s bad, but what’s all this extra for?”
“Why are they fleshing out these scary details over here so much? How does that help anything?”
“Well sure, I’m having kids. Yeah, I don’t know what’s going to happen. That’s okay.”
“I’m taking up a new hobby. Why? Oh, because it’s interesting. What? Uh, no, I’m not concerned about how it relates to AI.”
I’m making that up off the top of my head based on some loose models. I’m more trying to convey a feel than give a ton of logical details. Not that they can’t be fleshed out; I just haven’t done so yet.
I think your logic slipped. I’m not saying creating AI wouldn’t be different. I’m saying that the generator of doom predictions keeps predicting doom, and then hedging, and then re-predicting, while insisting that its new prediction is “really actually different this time”. Just like the last twenty times. But no, really, this time is really actually truly different!
If you’re talking about a real doom, such as building a superior species, and if it’s actually going to happen at some point, then yes at some point those doom predictions will in fact pan out. But if the generator of doom predictions is optimizing for feeling and expressing doom more than making accurate predictions, then it’s close to accidental that it ends up right at some point.
I don’t think it’s quite that extreme here. But saying “No, creating AI really would be different!” doesn’t affect the reasoning whatsoever. That just makes it a potent source of viral doom memes.
I think I agree. To answer your question:
So… why advocate waiting until 2028?
I’d prefer we didn’t. But my impression at this point is that Less Wrong as a community needs to dialogue with models carefully to be willing to adopt them, and “Let’s talk about it right now to sort out whether to adopt it right now” requires quite a lot of social buy-in. The most norm-respecting way I know to get there is to agree on a prediction that folk agree distinguishes between key possibilities, and then revisit it when the prediction’s outcome is determined.
But yes, I agree with what I think you’re saying. I’d love to see this space take on a more “We’re all in this together” attitude toward the rest of humanity, and focus more on the futures we do want. And I think those are things that could happen without waiting until 2028.
I also wish folk would take more seriously that their feelings and subtle stuff they encountered preverbally plays a way bigger role than it’s going to ever logically appear to. What I see in lots of rationalist spaces, and especially in AI risk spaces, is not subtle in terms of the degree of disembodiment and visceral fear. But I haven’t found a way to convey what I’m seeing there without it sounding condescending or disrespectful or something.
So, I’m trying for something more like collectively preregistering a prediction and revisiting later. I’m now getting the impression that’s not going to work either, but I’m observing what I think is something good coming out of the discussion anyway.
there exists a single, clever insight which would close at least half the remaining distance to AGI
By my recollection, this specific possibility (and neighboring ones, like “two key insights” or whatever) has been one of the major drivers of existential fear in this community for at least as long as I’ve been part of it. I think Eliezer expressed something similar. Something like “For all we know, we’re just one clever idea away from AGI, and some guy in his basement will think of it and build it. That could happen at any time.”
I don’t know your reasons for thinking we’re just one insight away, and you explicitly say you don’t want to present the arguments here. Which makes sense to me!
I just want to note that from where I’m standing, this kind of thinking and communicating sure looks like a possible example of the type of communication pattern I’m talking about in the OP. I’m actually not picky about the trauma model specifically. But it totally fits the bill of “Models spread based significantly on how much doom they seem to plausibly forecast.” Which makes some sense if there really is a severe doom you’re trying to forecast! But it also puts a weird evolutionary incentive on the memes: if they can, they’ll develop mutations designed to seem very plausible and amplify the feeling of doom, decoupling from that pesky reality that slows down how effectively the memes can mutate content designed to encourage their spread.
I can’t know whether or not that’s what’s going on with your “one clever insight away” model, or with why you’re sharing it the way you are. I’d have to see the reasoning. I don’t mean to dismiss what you’re saying as “just trauma”; that’d be a doubly unkind way of oversimplifying what I’m saying.
But at the same time, I find myself skeptical of any naked AI doom model that sounds like “Look, we’re basically guaranteed to be screwed, but this margin is too small for me to explain. But the conclusion is very very bad.”
I cannot distinguish between that being an honest report of a good model that’s too big or gnarly to explain, versus a virulent meme riding on folks’ subconscious fixation (for whatever reason) on doom.
And thus I put it in my “Well, maybe.” box, and mostly ignore it.
Isn’t it more like “I think there’s a 10% chance of transformative AI by 2027, and that is like 100x higher than what it looks like most people think, so people really need to think thru that timeline”?
That might be. It sounds really plausible. I don’t know why they wrote it!
But all the same: I don’t think most people know what 10% likelihood of a severe outcome is like or how to think about it sensibly. My read is that the vast majority of people need to treat 10% likelihood of doom as either “It’s not going to happen” (because 10% is small) or “It’s guaranteed to happen” (because it’s a serious outcome if it does happen, and it’s plausible). So, amplifying the public awareness of this possibility seems more to me like moving awareness of the scenario from “Nothing existential is going to happen” to “This specific thing is the default thing to expect.”
So I expect that unless something is done to… I don’t know, magically educate the population on statistical thinking, or propagate a public message that it’s roughly right but its timeline is wrong? then the net effect will be that either (a) AI 2027 will have been collectively forgotten by 2028 in roughly the same way that, say, Trudeau’s use of the Emergencies Act has been forgotten; or (b) the predictions failing to pan out will be used as reason to dismiss other AI doom predictions that are apparently considered more likely.
The main benefit I see is if some key folk are made to think about AI doom scenarios in general as a result of AI 2027, and start to work out how to deal with other scenarios.
But I don’t know. That’s been part of this community’s strategy for over two decades. Get key people thinking about AI risk. And I’m not too keen on the results I’ve seen from that strategy so far.
There’s a logic error I keep seeing here.
I agree, there’s more reason to think LLMs might not just magically align.
That fact does not distinguish between the worlds I’m pointing out though.
Here are two worlds:
We’re hurdling toward AI doom.
We’re subconsciously terrified and are projecting our terror onto the world, causing us to experience our situation as hurdling toward AI doom, and creating a lot of agreement between one another since we’re converging on that strategy for projecting our subconscious terror.
There are precious few, if any, arguments of the form “But here’s a logical reason why we’re doomed!” that can distinguish between these two worlds. It doesn’t matter how sound the argument is. Its type signature isn’t cruxy.
I’m not claiming there’s no way to distinguish between these worlds, to be clear. I’m pointing out that the approach of “But this doomy reasoning is really, really compelling!” is simply not relevant as a distinguisher here.
The whole point of the OP is to suggest that we get clear on the distinction so that we can really tell in 2.5 years, and possibly pivot accordingly. But not to argue that we pivot that way now. (That might make sense, and it might turn out to be something we’ll later wish we’d figured out to do sooner. But I’m not trying to argue that here.)
Not entirely. It’s a bit of a misreading. In this case I think the bit matters though.
(And it’s an understandable bit! It’s a subtle point I find I have a hard time communicating clearly.)
I’m trying to say two things:
There sure do seem to be some bad psychological influences going on.
It’s harder to tell what’s real when you have sufficiently bad psychological influences going on.
I think some people, such as you, are reacting really strongly to that second point. Like I’m taking a stand for AI risk being a non-issue and saying it’s all psychological projection.
I’m saying that nonzero, but close to zero. It’s a more plausible hypothesis to me than I think it is to this community. But that’s not because I’m going through the arguments that AI risk is real and finding refutations. It’s because I’ve seen some shockingly basic things turn out to be psychological projection, and I don’t think Less Wrong collectively understands that projection really can be that deep. I just don’t see it accounted for in the arguments for doom.
But that’s not the central point I’m trying to make. My point is more that I think the probability of doom is significantly elevated as a result of how memetic evolution works — and, stupidly, I think that makes doom more likely as a result of the “Don’t hit the tree” phenomenon.
And maybe even more centrally, you cannot know how elevated the probability is until you seriously check for memetic probability boosters. And even then, how you check needs to account for those memetic influences.
I’m not trying to say that AI safety shouldn’t exist as a field though.
Wow, you and I sure must be seeing different parts of the debate! I approximately only hear people talking about the object level. That’s part of my concern.
I mean, I see some folk doing hot takes on Twitter about psychological angles. But most of those strike me as more like pot shots and less like attempts to engage in a dialogue.