Statement 9.Math and coding are two key ingredients of AI research.
Argument. What does AI research actually consist of? Operationally, two activities: writing code and doing mathematics. A researcher’s day at a frontier lab is designing an architecture or an optimizer (mathematics), implementing it as a training run (code), reading the loss curves (mathematics again), and fixing the distributed-systems bug that silently poisoned the gradients (code again).
it’s good to see this in writing. so many people seem to think the barrier between math autoresearcher and AI autoresearcher is some sort of chasm. nothing could be further from the truth. even the genius of “research taste” appears to show cracks under the relentless pressure of speed—proposals per hour, experiments per day
My impression is that cleaning data, implementing RL environments, just trying stuff to see what works and other less theoretically rich activities are a pretty core part of capabilities research. (Not that these activities are necessarily less amenable to automation).
in my mind a lot of that work per experiment can get bundled under coding, and the more challenging task is constructing a research portfolio that makes (predictions about) optimal use of your resources, particularly in the RSI regime. however from time to time there will be a phase shift as the AI architecture changes, resulting in a large re-coding effort
This is great work. To zero in on the part that repulsed me rather than gave me clear new ways of putting things:
“My colleague Harriet, a senior researcher in her subfield, left her tenured professorship to work at a frontier AI lab not too long ago. This summer, we got on the phone and chatted about AI risk.
I showed her a seed of this essay and explained that I wanted to create common knowledge about x-risk in the mathematical community. She was dismissive.
She said, “On the current trajectory, humanity is 99% doomed, and this is a whole lot of effort for something that won’t obviously help.””
Harriet ought to be screaming her head off! Rather than making the problem worse (as I strongly suspect she is in her position) and belittling you. Outrageous.
I think we should be more considerate of people as they grapple with the heft of the problem. Many will continue to keep their heads in the sand, some will swing wildly in the opposite direction because [we’re definitely doomed, there’s nothing to be done] can sometimes be the next psychologically safest place to be after [we’re definitely ok, there’s nothing to worry about].
Being empathetic and being nice aren’t always the same thing. Loving actions are sometimes adversarial. When I imagine myself in the same position, I would want a friend to confront me and demand to know what the hell I’m thinking.
“We are doomed and there is nothing to be done, so I might as well doom us harder” can be made psychologically unsafe. It’s plenty of reason to end a friendship. They should feel pressure and shame for doing such a taboo thing, not to punish them, but because the lack of hard pushback acts as tacit approval, as if we can all be friends at the end of the day, as if friendship is more important than the death of all things.
Is it? I have often felt my relationship with a friend strengthen when I frankly tell them that they ought to quit their jobs on ethical and/or mission grounds. (Prior to this whole AI labs business.)
Sorry, I should have expanded a bit more. The quoted part is only the first five minutes of a two-hour conversation; I definitely believe in having frank conversations about x-risk with people who are on the ground.
What I don’t believe in is jumping directly to [here’s what you ought to do], which feels deeply condescending, as opposed to staying at [let’s find our cruxes, why do you believe what you believe?].
It is also not obvious to me that having them leave is the right decision, as it might have been ~5 years ago. There seems to be delicate internal opinion politics right now in the labs where the x-risk-pilled are a non-negligible minority and their presence could significantly improve our chances of (for example) future pauses/bans, or deployment of alignment ideas, or alignment problems being solved by powerful internal models.
Context: I’m a mathematics professor in the Netherlands, and one of the coauthors of the Leiden Declaration on Artificial Intelligence and Mathematics. I think this article is was a nice summary of the current situation. My impression from talking to my colleagues is not that there is a lack of concern; people have not necessarily thought through the details, but the idea of existential risk from AI is taken seriously. What is missing is any clear idea of what to do about it; people feel powerless. Near the end of your essay you wrote:
“Many people say that alignment is primarily a mathematics problem, so I think we have a lot to offer.”
I’m curious if you can make that more concrete? I expect that if someone stated a reasonably concrete mathematical conjecture and said “solving this would really help with existential risk” there would be plenty of people happy to devote effort to it. But my own (perhaps very wrong) impression of the field it that the questions are more like “come up with a solution to inner alignment” which is too vague for most mathematicians to make a start on. There are some very mathematical alignment programmes out there (e.g. that of Vanessa Kosoy); do you think that having lots of people pile on a programme like that would help?
Somehow it would be great to have something like the Langland’s Programme for AI alignment; it can be hard and incompletely specified, but it gives a convincing direction, is concrete enough to get started on, and it’s clear that if you succeed you have really made progress on the problem. At the moment that seems to me to be missing, but I’d love to be corrected.
Hi David! Thanks for your input and for the work on the Leiden Declaration. My essay was intentionally vague on what to do, as I don’t strongly believe in a particular approach that is likely to help, and also have not worked deeply enough in technical alignment to have strong opinions.
To be honest, one of the main goals of this essay was an implicit call for inaction: “Don’t go work for an AI company without sitting down and thinking about x-risk (but maybe do if you have a concrete theory of change after thinking about it, like Tsimerman seems to).”
Some obvious places for mathematicians to start looking for concrete directions are ARC and Lionel Levine’s work. Many major universities already have extant AI Safety groups, so senior mathematicians can probably pick up some low-hanging fruit just by getting affiliated to lend credibility, and asking to be looped in on the mathematical parts of their research.
My current conjecture (lightly held) is that senior mathematicians concerned about x-risk can have the most immediate impact not by trying to solve technical problems, as we are conditioned to do, but by throwing our weight around in universities and public discourse until we reach a critical mass. I understand that many individual mathematicians are becoming aware of the case for x-risk, but there is almost 0 public discourse or common knowledge about it. I think we could very easily completely shift the discourse if ~20 mathematicians wrote their own version of my essay, or there was a version of the Leiden Declaration centered around alignment concerns.
The alignment problem factors into several problems, all of which are individually open. We don’t know how to make an AI robustly avoid acting like a utility-maximizer. We do not know a safe utility function for an AI to maximize in the limit (S12). We do not know how to exactly specify the utility function of an LLM (S13). We do not know how to make alignment properties invariant under the dynamics of recursive self-improvement (S11). Solving alignment seems to require solving all of these open problems simultaneously.
The framing gestures at obviously wrong decision theories. Fixing the decision theory plausibly makes “utility” the wrong concept to focus on. Worse, “fixing the decision theory” is a framing that fits some possible solutions to the metaproblem of being confused about normativity, but it doesn’t fit other possible solutions to that problem. Without sufficient clarity, a process that makes progress in resolving confusion about normativity is a more robust bet than either fixing the decision theory or specifying utility functions (which is obviously doomed without the preceding steps working out in its direction).
(This is why talking about “values” or “preferences” is more accurate than talking about expected utility, even as it’s less precise, when a particular toy setting isn’t being assumed.)
it’s good to see this in writing. so many people seem to think the barrier between math autoresearcher and AI autoresearcher is some sort of chasm. nothing could be further from the truth. even the genius of “research taste” appears to show cracks under the relentless pressure of speed—proposals per hour, experiments per day
My impression is that cleaning data, implementing RL environments, just trying stuff to see what works and other less theoretically rich activities are a pretty core part of capabilities research. (Not that these activities are necessarily less amenable to automation).
in my mind a lot of that work per experiment can get bundled under coding, and the more challenging task is constructing a research portfolio that makes (predictions about) optimal use of your resources, particularly in the RSI regime. however from time to time there will be a phase shift as the AI architecture changes, resulting in a large re-coding effort
This is great work. To zero in on the part that repulsed me rather than gave me clear new ways of putting things:
“My colleague Harriet, a senior researcher in her subfield, left her tenured professorship to work at a frontier AI lab not too long ago. This summer, we got on the phone and chatted about AI risk.
I showed her a seed of this essay and explained that I wanted to create common knowledge about x-risk in the mathematical community. She was dismissive.
She said, “On the current trajectory, humanity is 99% doomed, and this is a whole lot of effort for something that won’t obviously help.””
Harriet ought to be screaming her head off! Rather than making the problem worse (as I strongly suspect she is in her position) and belittling you. Outrageous.
I think we should be more considerate of people as they grapple with the heft of the problem. Many will continue to keep their heads in the sand, some will swing wildly in the opposite direction because [we’re definitely doomed, there’s nothing to be done] can sometimes be the next psychologically safest place to be after [we’re definitely ok, there’s nothing to worry about].
Being empathetic and being nice aren’t always the same thing. Loving actions are sometimes adversarial. When I imagine myself in the same position, I would want a friend to confront me and demand to know what the hell I’m thinking.
“We are doomed and there is nothing to be done, so I might as well doom us harder” can be made psychologically unsafe. It’s plenty of reason to end a friendship. They should feel pressure and shame for doing such a taboo thing, not to punish them, but because the lack of hard pushback acts as tacit approval, as if we can all be friends at the end of the day, as if friendship is more important than the death of all things.
Pressuring the people at frontier ai labs who believe in x-risk to leave is an extremely naive way of relating to doom.
Is it? I have often felt my relationship with a friend strengthen when I frankly tell them that they ought to quit their jobs on ethical and/or mission grounds. (Prior to this whole AI labs business.)
Sorry, I should have expanded a bit more. The quoted part is only the first five minutes of a two-hour conversation; I definitely believe in having frank conversations about x-risk with people who are on the ground.
What I don’t believe in is jumping directly to [here’s what you ought to do], which feels deeply condescending, as opposed to staying at [let’s find our cruxes, why do you believe what you believe?].
It is also not obvious to me that having them leave is the right decision, as it might have been ~5 years ago. There seems to be delicate internal opinion politics right now in the labs where the x-risk-pilled are a non-negligible minority and their presence could significantly improve our chances of (for example) future pauses/bans, or deployment of alignment ideas, or alignment problems being solved by powerful internal models.
I’d be curious about reception to this piece from the intended audience.
How would you feel about me sharing it on r/math by the way?
Do it!
Thank you for this, I shared it in my advisor’s lab WhatsApp group.
Context: I’m a mathematics professor in the Netherlands, and one of the coauthors of the Leiden Declaration on Artificial Intelligence and Mathematics. I think this article is was a nice summary of the current situation. My impression from talking to my colleagues is not that there is a lack of concern; people have not necessarily thought through the details, but the idea of existential risk from AI is taken seriously. What is missing is any clear idea of what to do about it; people feel powerless. Near the end of your essay you wrote:
“Many people say that alignment is primarily a mathematics problem, so I think we have a lot to offer.”
I’m curious if you can make that more concrete? I expect that if someone stated a reasonably concrete mathematical conjecture and said “solving this would really help with existential risk” there would be plenty of people happy to devote effort to it. But my own (perhaps very wrong) impression of the field it that the questions are more like “come up with a solution to inner alignment” which is too vague for most mathematicians to make a start on. There are some very mathematical alignment programmes out there (e.g. that of Vanessa Kosoy); do you think that having lots of people pile on a programme like that would help?
Somehow it would be great to have something like the Langland’s Programme for AI alignment; it can be hard and incompletely specified, but it gives a convincing direction, is concrete enough to get started on, and it’s clear that if you succeed you have really made progress on the problem. At the moment that seems to me to be missing, but I’d love to be corrected.
Hi David! Thanks for your input and for the work on the Leiden Declaration. My essay was intentionally vague on what to do, as I don’t strongly believe in a particular approach that is likely to help, and also have not worked deeply enough in technical alignment to have strong opinions.
To be honest, one of the main goals of this essay was an implicit call for inaction: “Don’t go work for an AI company without sitting down and thinking about x-risk (but maybe do if you have a concrete theory of change after thinking about it, like Tsimerman seems to).”
Some obvious places for mathematicians to start looking for concrete directions are ARC and Lionel Levine’s work. Many major universities already have extant AI Safety groups, so senior mathematicians can probably pick up some low-hanging fruit just by getting affiliated to lend credibility, and asking to be looped in on the mathematical parts of their research.
My current conjecture (lightly held) is that senior mathematicians concerned about x-risk can have the most immediate impact not by trying to solve technical problems, as we are conditioned to do, but by throwing our weight around in universities and public discourse until we reach a critical mass. I understand that many individual mathematicians are becoming aware of the case for x-risk, but there is almost 0 public discourse or common knowledge about it. I think we could very easily completely shift the discourse if ~20 mathematicians wrote their own version of my essay, or there was a version of the Leiden Declaration centered around alignment concerns.
I am personally a fan of Resolution’s program …
https://www.lesswrong.com/posts/AP7YDke5jjY4v3X9Z/resolution-fka-sequent-scale-and-automation-for-higher
much of which is still listed here …
https://timaeus.co/projects
there is also …
https://www.lesswrong.com/posts/KfkpgXdgRheSRWDy8/a-list-of-45-mech-interp-project-ideas-from-apollo-research
(a little over 2 years old now) and …
https://www.lesswrong.com/posts/mtGpdtDdmkRC3ZBuz/list-of-lists-of-project-ideas-in-ai-safety
which is not quite a year old and a little more diverse.
oh my gosh yes. The woods are lovely, dark and deep, But I have promises to keep, And miles to go before I sleep, etc.
Thanks, this motivated me to search around a bit more, the situation is better than last time I looked; I’ve tarted reading Lionel Levine’s recent paper [https://github.com/lionellevine/MAIS/blob/main/papers/P1/MAIS-P1.pdf].
The framing gestures at obviously wrong decision theories. Fixing the decision theory plausibly makes “utility” the wrong concept to focus on. Worse, “fixing the decision theory” is a framing that fits some possible solutions to the metaproblem of being confused about normativity, but it doesn’t fit other possible solutions to that problem. Without sufficient clarity, a process that makes progress in resolving confusion about normativity is a more robust bet than either fixing the decision theory or specifying utility functions (which is obviously doomed without the preceding steps working out in its direction).
(This is why talking about “values” or “preferences” is more accurate than talking about expected utility, even as it’s less precise, when a particular toy setting isn’t being assumed.)