Mathematicians are Feeling the Doom
I think there’s a critical opportunity for someone here.
Mathematicians are feeling the doom (mostly in the “lose our jobs” sense).
Academics are freaking out about the daily news that amateurs are asking GPT “prove career-defining theorem, make no mistakes” and it’s just working.
Senior researchers are leaving for frontier AI labs. Many are mentally spiralling or flailing about their life’s work not mattering anymore.
Last year, when I tried explaining IABIED to colleagues, I would be met with incredulous stares. This year, I’m met with incredulous stares and “So what should I do now?” (I don’t have a good answer for them, which is part of why I’m posting this.)
I’ve quarantined AI discussions on my research discord because otherwise it would overwhelm everything else.
Top mathematicians, people on par in mathematical ability with Critch, Christiano, and Steinhardt, people who have been running leading-edge research groups for decades, people with enormous soft power in academic circles, are going to OpenAI without even having considered x-risk for five minutes.
Many more will be leaving soon.
If you want these folks to hear something at all, to consider some other option in the rest of their life, now is the time.
Seems like the fundamental problem is that humanity’s strategic competence is extremely inadequate relative to its technical capabilities, and this (limited) strategic competence was front-loaded into AI risk (the most strategically competent people foresaw the problem and went into the field early, before AI was even a thing for everyone else). As AI risk awareness grows, the average strategic competence goes down as more and more people enter the field. (Consider the gap between someone who “are going to OpenAI without even having considered x-risk for five minutes”, and what’s actually needed to make a positive contribution or avoid making a negative contribution.)
This seems clearly true and like an interesting frame, but I’d have to think more about it to agree with the full conclusion. It seems to me not clear that average strategic competence is a thing to care about. I think there’s a good case that average people with very low agency still have an instinctual respect for/deference to people who have higher agency, and that our natural/institutional hierarchical structures support this stratification. Most problems seem to be solved like this, if imperfectly? Organize people into systems where small numbers of outliers with high strategic competence are deferred to by larger numbers of people with only technical competence.
To me it seems like we have a crazy problem where the so-called highly strategically competent people are not able to capture the highly technically competent nerds from academia.
Note that I never used “highly strategically competent”! If we did have a group of highly strategically competent people, then what you say becomes a lot more plausible, but even the early AI risk people are far from that. Consider for example that a strategic consideration as obvious (in retrospect) as Legible vs. Illegible AI Safety Problems was written down only in late 2025.
During the 2012-2014-ish era, I recall a fairly common statement from MIRI being “you might think AI alignment is a computer programming problem but it’s more of a math problem”, and there was some attempt to carve it into fun little chunk for Math people.
Then MIRI mostly gave up on solving it’s technical agenda in time. I’m not sure what the state of things are in terms of “what problems are still open and seem to matter?”.
Seems probably worth someone putting in legwork to organize the state of the problem into something comprehensible to the math world. I vaguely am thinking of @Max Harms maybe being a good fit for this?
Curious for braindumps from @abramdemski @Scott Garrabrant, @TsviBT, @johnswentworth and @Alex_Altair on the state of thing heres.
(I’ve been “out of the game” for a couple years.) I would append to
a follow-on:
(I hesitate to use the word “phenomenology” because it will definitely be misunderstood by ~everyone, but there just isn’t a better word; an internal phrase that was floated was “core theory of mind”, and I mean to gesture at “mathematico-introspective investigation of core theory of mind”.) This is elaborated on (cryptically / elliptically, but you could read slowly & think) here: https://www.lesswrong.com/posts/TNQKFoWhAkLCB4Kt7/a-hermeneutic-net-for-agency
In this context, the long and short of it is, I don’t think you can feasibly point people at the parts of the problem that matter.
I’m just such a special genius that only I could possibly understand the real alignment problemFor some reason people don’t seem to be inclined to plant difficult questions and let them grow over time, sit with unresolved questions, look at big things and keep their bigness firmly in mind while also somehow making cumulative progress, overhaul key elements or at least keep them firmly provisional, keep staring outside of the streelight’s glow for years, and so forth. (Something something Grothendieck something something Peter Scholze something something staring at conceptual foundations.) Or maybe it’s the thing about security mindset (Yudkowsky), or overconfidence (Dai), etc. Or maybe we’re just not high g enough. Or all of the above or something else, IDK. But again, it just doesn’t seem to work to point people at the problem. Or maybe I’m deluded, who could know.If you take all the alignment research, insofar as I’m aware of it, in the past 2 decades, and multiply that by 3, it still doesn’t even come close to solving alignment. This is very rough and grim, but it’s what I think.
I’m not sure I buy that most mathematicians will be out of a job. Maybe some of the number theorists, analysts, and combinatorialists? Or something, IDK, I’m making that up. (Based on what fields vaguely seem to have some niches for people doing mostly high-algebraicness work.) Or maybe they will be.
What should they do instead? IDK. Human intelligence amplification is still insanely underresearched, and can definitely use more brainpower. I’d be very happy to be connected with smart motivated scientists / mathematicians who may want to work on empowering humans!! My gmail for that would be: tsvibtcontact
To be clear, when writing this post, recruiting any mathematicians to alignment work was much more of a stretch goal.
The low-hanging fruit I’m trying to point to is that there are droves of academics currently considering going to work at frontier AI labs without having even heard the basic arguments for x-risk. I think it would be much much easier to convince a substantial fraction of them to do NOT THAT, than to direct them to do any active technical alignment work.
Having mulled this slightly more… I don’t actually have a very clear idea on what would work here (in the sense of “be memetically successful enough to reach a meaningful number of people.”)
I think there’s a moderately effortful job here of “figure out how to communicate about this in terms the math community will be motivated by and engage with” which can only really be done by someone in the math community.
I’m curious, when you imagine doing “the obvious thing”, what ends up being hard or not working? (presumably there’s a reason you wrote this post targeted at LW users than some other post targeted at math people?)
The “obvious thing” that I have done is state my views on doom to people when it comes up and point them to IABIED, and I think it has largely been successful in causing ~20 people I have personal relationships with to arrive at p(doom) > 5%.
The next “obvious thing” is to scale this up by a factor of 10 to people I’m only vaguely acquainted with, and to write more publicly in a mathematician-facing way as you suggested. These feel both costly, risky, and less effective in ways that the previous thing was not. I am still considering doing it, but need to sit down with my aversions for a while. Some examples: these conversations sometimes require me to be forceful/activist to keep people in my frame instead of sliding off into more comfortable topics, and they plausibly only work because I have significant personal rapport already with a colleague or student.
I would certainly prefer to live in a world where 1 in 20 mathematicians are already awake and I can just create common knowledge that the private conversation thing works, and then not do anything more effortful. As far as I know the only other academic mathematician who writes about Doom is Jacob Tsimerman, who just won the Fields Medal and left to do safety at OpenAI. He would certainly be a person who could do the broadcasting thing (and is doing it already, but in a less forceful way than I’m imagining).
(This is probably not helpful, but just want to note that I’m interested, in a “hobbyist” sense, in the problem of the general kind of confrontation you’re mentioning, under the moniker “confrontation-worthy empathy”; if you happen to want to chat about it I’m interested, e.g. to consider different ideas for making the process more wholesome and similar.)
I would be interested in having this conversation! It seems to me that the kind of confrontation I’m mentioning is a much easier version of the thing you’re pointing to? I tend to just point amateur levels of unconditional positive regard at people and under pre-conditions of existing trust conversations mostly go well. I imagine scaling up the operation requires more finesse, but even so it’s hard to imagine needing the amount of skill you describe there. I’ll message you.
Yeah, I think it should be much less hard for people not already embedded in working on AI; just seems kinda related.
Yeah fair.
Partly, I am kinda assuming people have a harder time wrapping their brain around “what not to do” vs “what to do”, and I think you’ll be more successful with that goal if you present a positive vision of how to orient to x-risk that speaks their language.
But, yeah that is a pretty different framing that would be suspicious if it didn’t look different at all from where I was pointing.
IDK, there being problems that are hard to fix but easy to make worse seems like a sort of common sense situation? Though I’m struggling to think of any good examples off the top of my head. Also just “we aren’t ready for the machine god, don’t help them build it faster” is super simple?
I guess math very much isn’t like this. You can generally make at least some progress on a problem, but you can’t really make a problem worse except by giving bad advice to others working on it?
I think there’s something especially bad about AI, where people disagree a lot or are confused about what exactly the problems are, and have a lot of motivated cognition towards thinking “if they’re working on it, it’ll go better instead of worse.”
(i.e. if people were right that the main problem is misuse, going to work at a lab is more reasonable)
There’s something additionally hard about the existential stakes, where it’s hard to admit the problem is real if you don’t feel like you have some way of engaging with it.
E.g. Musk seems like mega-biased toward taking action, with bad results (responsible for like a quarter of frontier AI lab stuff, in some sense). The politician’s syllogism bites real hard when accelerating bad stuff is 1. convergent and 2. much easier than accelerating good stuff.
“Please read IABIED” is my first thought, but sounds like you’ve probably tried that? Maybe there should be a TLDR aimed at mathematicians? We could ask Yudkowsky to write one?
I think recommending a book just doesn’t work without a long preamble or some unusual social leverage. I’ve had better success recommending online articles, my current go-to is Scott’s book review of IABIED, which is still tl;dr and has a bunch of missing context.
The main thing a good first-intro has to do, honestly, is to be as short as possible while explaining instrumental convergence, orthogonality, and countering some representative examples of crazy moon logic that people come up with initially.
Have you tried AISafety.info’s intro sequence? There’s also a shorter stand-alone article, but that one is written for the average person.
Thanks for the tip! On first pass I think it is more measured and comprehensive but less fun-to-read than Scott’s review so I would likely not try that instead.
TBH, I thought mathematicians would prefer that.
I’d be curious if you (or anybody else) want to test to see if my intro is superior[1]. I weakly think it would be, but I’m pretty unconfident and there are good reasons for me to share my guide with people I know over other people’s guides regardless of objective merit.
I deliberately wrote the intro to avoid many mistakes I saw in other intros (eg too many extraneous details, appealing to overly complex analogies, shadowboxing insider objections, or “going meta” in an intro post).
eg by randomizing my intro vs Scott’s review to send to different people.
My sense is many of these problems are either ill-defined or too hard to be tractable.
In certain fields like computability theory most problems are intractable, just because programs are very complicated and diverse objects that are hard to prove things about. Progress in such fields is made by working in the areas of the field where there is enough structure. Unfortunately proofs over programs have featured in agent foundations since the beginning (eg tiling agents).
As for ill-defined problems, ontology identification, embedded agency, and decision theory are full of them. Eg finding something that behaves like counterlogicals, which are nonexistent objects. Because they are nonexistent objects it requires philosophical progress to make a list of properties they need to satisfy. This doesn’t mean it’s impossible to make progress, but the problems need to be formalized in a way that are tractable and don’t lose all their relevance to AI.
As a working mathematician who occasionally tries to think about some of this stuff, that’s also my feeling.
I’ll also point to the people and agenda at Resolution: https://resolution.org/launch
A specific thing I’m curious about: there was a strategy of “turn some open alignment problems into nice little math problem to nerd-snipe people with.” Did that strategy accomplish anything? (i.e did it successfully get someone who would have bounced off the general conceptual arguments to engage enough to seem to eventually get the conceptual argument? And/or just produce obviously useful work)
The MAIS (Math for AI Safety) repo is my new effort to draw mathematicians into AI safety.
It started as a survey paper Math for AI Safety: An Invitation for Mathematicians. During the writing of that paper, Claude and GPT generated 100+ pages of open problems organized in eight research agendas. I’m releasing that material as a public hub for mathematicians to collaborate on AI safety problems.
The repo just went live today, and it’s very much a work in progress! I’d appreciate any feedback on how to make it better.
Hi Lionel! I remember your name from doing some research on chip-firing and rotor-routers in high school, many lifetimes ago. I didn’t know this existed and am glad it’s a thing!
Pitching working at ARC seems pretty good to me.
We’ve been shouting about approaches that could use mathematical research talent for several years, which don’t have huge downsides such as those pointed out by Wei Dai. If mathematicians are interested, they should look at, e.g., the work ARIA has been doing on mathematical foundations, at work and the research agenda by Vanessa Kosoy, and at the new work being launched by Resolution.
So far, the big breakthroughs are coming from strong professionals asking LLMs to prove truly important results.
While the amateurs do get empowered quite a bit, right now the situation rewards high competence.
I will say that Fable + Codex & on-demand VMs honestly feels superhuman, lacking only the “spark of genius” heuristic (heuristic set?). The problem search spaces are nevertheless very large, so brute-force simply isn’t an option without some clever perspective.
I think also that people underestimate the utility of partial results and we need a way to verify and document these to avoid wasted and duplicated work.
I get that change is scary, but this is honestly the most exciting time to be alive as a mathematician. even if all LLM use were banned tomorrow, each of these discoveries would still be talked about in 100 years. And, capabilities-wise, this is the worst it is ever going to be.
given that AI-associated x-risk largely depends on our level of mathematical competence, this is also arguably the most important time in history for the social role of mathematicians.
So far, mathematics have been used mainly to make the global situation worse, i.e., by enabling AI development to proceed in a way that maximizes capabilities with almost no regard for how easy it will prove to maintain control or alignment of the capabilities.
It would take a lot to persuade me that that trend will reverse any time soon.
I will concede that doing mathematics is probably good training for alignment research because it teaches mental moves that are useful outside mathematics, but I wouldn’t recommend mathematics as the majority of the training time.
I say this as someone who really enjoys doing mathematics.
Why do you think this?
I think controlling AI x-risk requires insight into the behavior of complex mathematical objects. To be a little less fuzzy, we need a working theory of mind. It is our mathematical competence which will determine how much control we have here.
I think RSI (a.k.a. self-improving AI autoresearchers) is inevitable for a wide variety of commercial, military, and/or geopolitical reasons, no matter what treaties are signed, no matter what promises are made. At least the NSA will do it. This is a curse in that it speeds the timeline. This is a blessing in that it gives us a chance. Again, though, the key to bounding the alignment delta between versions is going to be a theory of mind and its associated complex mathematics. If the delta is unbounded or even just too large, we lose and die. If the delta is small, we have a chance.
I think just waiting, or rather just researching, allows successive model versions to drift apart in mind space and makes it more difficult to bound the alignment delta. An all round catastrophe. We need to cognitively enhance each mathematician we have and put them to work developing (the mathematics behind) a theory of mind and/or on their favorite alignment subproblem.
Strongly doubt that.
My view is that we probably aren’t anywhere remotely close to knowing how to align ASI, and also that the current people don’t seem to be capable of making progress, so we should be working on augmenting human intelligence instead while halting AGI research. So the two things I’d want to suggest they work on are:
Figuring out how to to prevent anyone from building ASI anytime soon
Augmenting human intelligence
(I believe this is essentially the position of IABIED)
Both of these endeavors could absorb a lot of brain power, maybe especially the augmentation. I’d expect mathematicians to have an appreciation for how much humans vary in cognitive ability, and have benefited from high cognitive ability themselves; maybe this helps with pitching them on how valuable it would be to augment human intelligence.
Obvious followup to this: would you be interesting/willing to work on human cognitive enhancement? If not, why?
On Copenhagen Interpretation of Ethics grounds I precommitted to not responding to queries like this, or else it would look like anyone who tried to solve any problem had an obligation to consider every related problem.
Fair enough. I guess I want to get a sense of how hopeless an endeavor what I’ve proposed above is.
My intuition is that “the smart shot“ (whatever that is) plus a neural lace in an exponentially expanding number of people looks very similar to an RSI, except that a datacenter is unneeded, the nodes of the ASI have human rights, and it will never actually happen because it requires human experimentation, which would be banned.
I’ve been thinking about this for a long time. Doesn’t mean my thoughts are right, just well worn. Hopefully deep in a truth-tracking direction.
My best guess is that mathematicians will need to make their job be about enriching subjective experience rather than generating proofs.
If humans’ value is defined by their automatable output, then the economy absolutely will replace humans. It seems to have maybe found a way to do that with AI.
If we think there’s something wrong with that, then we’d better start defining humans’ value in terms of whatever it is that gives us that sense of wrongness.
I don’t know exactly how we do that. But I I think I see at least part of what to do.
If we get that part right, I think maybe the economy stops actively maligning AI against human existence.
I feel like mathematicians are being damned by faint praise here. The mathematical project is about precision and certainty, but it is a common experience for students to realize that for the first time they are seeing (for want of better terms) mathematical objects and relationships to which they were formerly blind. and much of the joy of mathematics comes from the ability to now discuss these new entities with precision. and further, part of being a professor of mathematics is exactly inventing and codifying new languages to carry out such discussion. so that a mathematician’s job can be understood as the invention and learning of new languages to discuss previously unseen entities and associated phenomena. this strikes me as an extraordinarily valuable social role, far beyond what mathematicians are usually granted.
I think the path here is to decouple human value from economic output. mere human existence will be, for all practical purposes, of negligible cost. with luxuries not so very much more.
Totally agree. I think mathematics is dramatically and tragically misunderstood.
I don’t understand how. Mind clarifying?
I see two interpretations here. One I might agree with, the other I fairly strongly disagree with.
The one I might agree with: Economic output is often going to be better done by something nonhuman, so if human value is tied to output, we’re setting humanity up for replacement.
I say I might agree because it really depends on what we mean by “output”. In some sense, all signals are a kind of output. Even if we find some way of valuing human existence inherently, signs that something is human will be an output that gets valued in whatever we might want to call an economy.
The one I fairly strongly disagree with: We have to decouple human value from the economy. Humans have to be seen as valuable for their mere existence.
I think that can’t work. Human existence is a cost, even if negligible. An economy that does anything like optimization will have pressure in such a situation to wipe out humans to recover the cost. If the thing stopping the economy from doing that is some kind of rigid rule around “value”, then the economy will try to find some way to hack around that rule.
It seems to be that for humanity to survive this, human existence needs to be valued in a way that’s intrinsic to the economy.
Said differently, I think it needs to be on the table that what humanity most values and cherishes is best accomplished without humanity’s survival. If it can’t be, then the reason it can’t be needs to be what’s baked into the Great Optimizer (i.e. the thing that showed up as evolution, then the economy, now AI) in order for humanity to survive this juncture.
> > I feel like mathematicians are being damned by faint praise here.
> I don’t understand how. Mind clarifying?
Just that you said …
> mathematicians will need to make their job be about enriching subjective experience rather than generating proofs
This seems to imply that generating proofs (if of value at all) does not enrich or cannot compete with enriching “subjective experience.” Even being generous, it damns the supposed role of the mathematician (“generating proofs”) with faint praise (they’ll be forced to do something useful, like enrich subjective experience). Of course as discussed, proof generation is already a most human of activities entailing language invention and novel expression.
Perhaps further, while proof generation is an important act in mathematics, able to give confidence when taking and refraining from project actions and time allocation, it is universally acknowledged among mathematicians that particular points in proof space are less interesting than the charts produced on the journey to those points. That unlocking new reasoning techniques is more important. Again, referencing generating proofs without the larger context of enhanced human cognition is “damning with faint praise.”
Sorry I’m ranting a bit, but I’m on something of a mission to see mathematics held in at least as high esteem as literature. No one admits to illiteracy (or at least pre-TikTok this was a shameful thing), but will casually assert innumeracy even though it is just as intellectually impoverishing.
> > I think the path here is to decouple human value from economic output.
> I see two interpretations here. One I might agree with, the other I fairly strongly disagree with.
I confess I am unable to distinguish between “economic output” and “the economy,” so I will refrain from sowing further confusion on this topic.
Would you say that LLMs are already good at the ‘invention’ part? My mostly-outsider impression is that they have a decent grasp of already existing languages and can apply them in appropriate circumstances that humans have overlooked, which is what’s driving the current action.
it is hard to measure how “good” they are without a truly formidable experimentation program. however, models have been observed inventing languages to solve a hard problem (and others in its class), which is already astounding to me, and is literally the definition of intelligence used by some computer scientists.
for example see section 6.4.5 in the Claude Sonnet 5 System Card …
https://www.anthropic.com/claude-sonnet-5-system-card
the behavior is somewhat forced by limited prompt context and hours long runs, but is still super interesting.
> they have a decent grasp of already existing languages and can apply them in appropriate circumstances that humans have overlooked
”overlooked” makes it sound like everything was right before the expert and they still failed to notice it. I think that there is a little more going on in mathematics in that “proof space” is so vast that an expert could search it forever and still never encounter the particular region with the key that unlocks the solution. while the patterns of mathematics and the LLM’s brute force speed makes up for some of this, it is still only a few of orders of magnitude, so that’s where the mysterious “research taste” comes in, which somehow predicts productive areas to look with only a preliminary survey of the problem (both in humans and LLM-based autoresearchers!).
As a mathematician currently in the middle of this transition (leaving an academic math-phys postdoc for AI safety work), two data points that might be useful here:
Resolution, one of the orgs named above, is a good illustration of the capacity problem on the other side of this: their Learning Theory team isn’t taking new applications right now because they’re overwhelmed by applicant volume and are putting hiring-manager time toward their less developed tracks instead. So the constraint isn’t always “not enough mathematicians are interested”, sometimes it’s “not enough hiring bandwidth to process them for a given subfield right now.”
I’d be curious if anyone here has a view on how ARIA’s mathematical-foundations programme and Vanessa Kosoy’s agenda compare in terms of what kind of mathematical background they actually draw on (learning theory vs. more classical formal-methods territory, say). I’m putting together a short survey of orgs/agendas a theory-leaning mathematician might fit into as I explore my options, and first-hand comparisons would help.
Starting a series that goes into math and safety case of theory-first AI safety orgs. From a mathematician pivoting into AI safety. First up, ARC. https://kubuondr.substack.com/p/arcs-research-agenda-solid-mathematics
I would love to see the output of this short survey!
easy thing to do: commit to voting against ai datacenters and convince at least one other person to also do so—on the argument of extinction
extra: commit to voting against ai datacenters and convince at least one other person to also do so and convince them such that they also want to convince at least one other person—on the argument of extinction
Can you give examples? I don’t follow any sources that would give me this impression. What sources shape your perspective here?
Almost exclusively conversations with other mathematicians.