LessWrong team member / moderator. I’ve been a LessWrong organizer since 2011, with roughly equal focus on the cultural, practical and intellectual aspects of the community. My first project was creating the Secular Solstice and helping groups across the world run their own version of it. More recently I’ve been interested in improving my own epistemic standards and helping others to do so as well.
Raemon
Karma: 66,617
Having mulled this slightly more… I don’t actually have a very clear idea on what would work here (in the sense of “be memetically successful enough to reach a meaningful number of people.”)
I think there’s a moderately effortful job here of “figure out how to communicate about this in terms the math community will be motivated by and engage with” which can only really be done by someone in the math community.
I’m curious, when you imagine doing “the obvious thing”, what ends up being hard or not working? (presumably there’s a reason you wrote this post targeted at LW users than some other post targeted at math people?)
A specific thing I’m curious about: there was a strategy of “turn some open alignment problems into nice little math problem to nerd-snipe people with.” Did that strategy accomplish anything? (i.e did it successfully get someone who would have bounced off the general conceptual arguments to engage enough to seem to eventually get the conceptual argument? And/or just produce obviously useful work)
I guess I’m now confused “what additional work does the bucket do, if you cleared out all stagnant water?”
The way I’d frame this is “it would be pretty good for some people to start making bet about what actually will play out in the nearer term.” I think there’s enough information that you should be able to collect some bayes points about it.
Yeah fair.
Partly, I am kinda assuming people have a harder time wrapping their brain around “what not to do” vs “what to do”, and I think you’ll be more successful with that goal if you present a positive vision of how to orient to x-risk that speaks their language.
But, yeah that is a pretty different framing that would be suspicious if it didn’t look different at all from where I was pointing.
During the 2012-2014-ish era, I recall a fairly common statement from MIRI being “you might think AI alignment is a computer programming problem but it’s more of a math problem”, and there was some attempt to carve it into fun little chunk for Math people.
Then MIRI mostly gave up on solving it’s technical agenda in time. I’m not sure what the state of things are in terms of “what problems are still open and seem to matter?”.
Seems probably worth someone putting in legwork to organize the state of the problem into something comprehensible to the math world. I vaguely am thinking of @Max Harms maybe being a good fit for this?
Curious for braindumps from @abramdemski @Scott Garrabrant, @TsviBT, @johnswentworth and @Alex_Altair on the state of thing heres.
Ah, that’s fair. I didn’t mean it was close to the bar.
I actually hadn’t thought that much about exactly how close it was bar (it was sufficiently obviously over-the-bar upon reflection). The motivation for the notice was
a) the considerations were something I hadn’t quite thought through and articulated before
b) if you hadn’t paid much attention to the content and were just going off initial vibes, it’d be less obvious
c) future people who do similar writeups that are some mix of “somewhat less timeless, somewhat more social-move-y” might get a different decision.
I didn’t want this to turn into a sprawling meta-discussion isn’t really about this post, so I replied over on the open thread: https://www.lesswrong.com/posts/MHi9yrQgXupYJni6n/open-thread-summer-2026?commentId=6oYk7rt4tf9GyYoXv
Thomas Kwa responded to my notes on frontpaging Why I Left Google DeepMind. I didn’t want to make the comment section about this and it seemed like it might turn into a sprawling metadiscussion, so I put it over here.
It would have been an outrage IMO to not frontpage this, so I’m glad it’s frontpaged. It’s super timeless and valuable, and the opinions were tactfully conveyed and necessary for the post.
To not express strong opinions about Google Alex would have to self-censor, which is counter to the entire spirit of the post and would make it substantially harder to follow his strategic and moral thinking.
I agree “the post does a good job being frontpage” (like obviously we agree on that). But, you seem worried about some class of problem that’s orthogonal to or ignoring why we have a frontpage distinction.
Plenty of good posts are not frontpage.
Sometimes, a good post has hypothetical frontpage version of itself but the “real post” won’t be frontpage. In those cases, the standard recommendation is “write the frontpage version, and a separate version that covers the less-frontpagey parts.”
In this case that wasn’t necessary (In addition to generally being a good, timeless piece, this also does the “social move” part in a minimal, evenhanded, factual way that I liked).
(Alex also made a separate post that was “even more frontpagey”, just focusing on the internal governance details).
But, it sounds like you don’t like that I was even counting “is a social move” in the “evidence for not frontpaging” column, and, dunno what to tell you other than “yep, there are some costs to having the frontpage policy that might even come up.” (Or, maybe seems like think I’m saying it’s bad that this was nonzero a social move? I don’t think that! Social moves are correct sometimes! “counts somewhat against frontpage” !== “shouldn’t have been made”)
Seems plausible but my baseline expectation of most academics of any sort would be similarly dismissive.
Are there handy links for what they think would happen if mass skilled immigration happens?
Who are some economists who are “serious” about it? I don’t really know what filter to run here.
Mod note: This is an edge-case of “frontpage or no?”. We have a somewhat tricky stance on posts that are giving money to straightforward intellectual labor of a type we generally encourage on LessWrong.
The difference between these and “Job announcement” type posts is something like “how much does it feel like the post is Out To Get You, or slips LW to a world that feels more Out To Get You?”. When we imagine people showing up to see a bunch of job ads, it feels like LinkedIn, whereas if a new user sees a frontpage with like 1 bounty for an intellectual problem, it still feels more like an intellectual space.
Frontpage mod note: This felt slightly edge-casey re: “is it Frontpage?”. I think it’s pretty timeless and will still be relevant in a few years and am frontpaging it, but wanted to leave some notes about it for people modeling the frontpage rules.
The post seems to be doing two things:
Be a detailed recounting / postmortem of a process of internal governmance (i.e. information)
Being kind of social move of “hey, FYI I think Google Deepmind fucked up here and should be socially punished”.
The latter is a totally fine kind of post to write but makes it somewhat less the sort of thing I want people’s “training data for what LW is primarily about” to be dominantly seeded by, which is part of what the frontpage/personal distinction is for.
There isn’t a hard rule here, but I think “is a significant point of the post to be a social move?” is something that counts at least a bit against frontpageness, on-the-margin (it’s not the primary determinant, which is still “timelessness”).
Other moderators might have somewhat different takes here, I didn’t run this by everyone.
(We were behind on moderating new posts and the “frontpage or no?” call here is mostly about whether it’ll show up in the recommendations feel in the future. Although it does have enough karma that’ll still have some impact on views for the next few days)
I didn’t read article in depth, but, quick comment on AI 2040: A lot of the point of AI 2040 is to change the realpolitik incentives that underly the law, so you aren’t very reliant on law. (it’s not very specific about the actual laws, and I think this is large because they aren’t as much the point).
Roughly:
stage 1: major world powers get good intelligence (in the CIA or spy-satellite sense) tracking where all the compute is going. This enables pauses that are verified by seeing that datacenters are turned off. You could defect from the pause, but then the other guy would too.
stage 2: verification devices installed in all datacenters, that let you not just verify total pauses but also “are we using datacenters to do anything other than run whitelisted AI models.” Again, someone could defect, but then the other guys would too and your short term advantage from defecting would be pretty short livedstage 3: all research information is transmitted instantly/publicly. Individual datacenter + company cartels could try to defect, but would quickly be pressured by various military power apparata to knock it off, or, a chain of defections would make everyone worried about what everyone else was doing and then everyone would be racing again (and meanwhile, the information sharing means people’s starting positions in the race are fairly neck-in-neck)
stage 4: mutually assured compute destruction. Not only can you no longer just return to racing without the other guy returning to racing. Now, if you go back to racing, the other guy will destroy your compute and you’ll destroy their compute and we’re all back to square 1.
The whole thing is trying to be backed by the military power incentives, not “law” per se.
I agree things about Citizen’s dividend are dependent on there being some kind of rule-of-law about taxation and distribution, though not IMO on any particular implementation of rule-of-law.
I liked the term “AI mania” as replacing most instances of “AI psychosis” and “Claude Code Mania” as a variant that made me go “oh, yeah I’ve totally had Claude Code mania.”
A few distinctions I think are worth tracking, in rough clusters:
Overuse
AI Addiction
AI Mania
Epistemic Capture
AI Mania
AI Reality Bubbling (Sycophancy)
AI Abdication
AI Atrophy
Relational Capture
AI Mis-anthropomorphism
AI Seduction
I think each of these has mild forms that probably most people have, and more extreme forms.
I like this quote from Buck Shlegeris: “Five years ago I thought of misalignment risk from AIs as a really hard problem that you’d need some really galaxy-brained fundamental insights to resolve. Whereas now, to me the situation feels a lot more like we just really know a list of 40 things where, if you did them — none of which seem that hard — you’d probably be able to not have very much of your problem. But I’ve just also updated drastically downward on how many things AI companies have the time/appetite to do.”
I vaguely recall @Buck having clarified this is (something along the lines of) handling near-human-level-ish AI, not scaling to superintelligence (maybe thing #41 is just “so don’t do that”?), and/or having some kind of update that made this not straightforwardly true. But, don’t remember the details, so, pinging him to check on the state of his take here.
as a meta point, i get the impression that you think i’m being obtuse for rhetorical reasons. but i promise you: i’m just obtuse!
Okay lol, fairnuff.
The thing I vaguely remember from the book’s early chapter was, a fishing town where they rotated slots you get in the lake each season. I can’t remember if that was a thing the town elders/government decided sort of from on-high, or that the fishermen negotiated together and organically turned into a norm, or what.
But, the point was “rotate slots in the lake” is a mechanism that is neither exactly private property, nor setting a limit on number-of-fish. This doesn’t have much to do with labor unions.
I believe there were 1-2 other mechanical details in that chapter but I don’t remember, that were similarly “oh, huh, I guess you could do it that way.”
(The implication here not being that AI safety unions should use rotating slots, which is nonsensical in this case – I was mostly replying narrowly to the “how do the proverbial fisherman avoid overfishing without limiting supply?”. With a bit of a broader implication for “mechanism-space is deep and wide and local-details-space is deep and wide”, which might or might not be relevant to the OP idk)
Nope, I did not actually read it that thoroughly.
But, I think you’re sort of prematurely/overly lumping things together (see: lumper/splitter), which I think is erasing detail that is worth actually thinking through.
The point is the people in all the different situations did not start by asking “do we want to handle this with labor unions or government centralization or privatization”, they started by working through the details of the individual cases, and maybe the stuff ends up sharing some structures or could be ontologized a particular way, but the details of what the people-on-the-ground want/need and the realpolitik matter a bunch.
I think there’s something especially bad about AI, where people disagree a lot or are confused about what exactly the problems are, and have a lot of motivated cognition towards thinking “if they’re working on it, it’ll go better instead of worse.”
(i.e. if people were right that the main problem is misuse, going to work at a lab is more reasonable)
There’s something additionally hard about the existential stakes, where it’s hard to admit the problem is real if you don’t feel like you have some way of engaging with it.