I doubt this would be the ideal moment for a pause, even assuming it were politically tractable, which it obviously isn’t right now. Very likely you’d want to pause after you’ve automated AI safety research, or at least strongly (e.g. 10x) accelerated at least prosaic AI safety research (none of which has happened yet) - given how small the current AI safety human workforce is, and how much more numerous (and very likely cheaper per equivalent hour of labor) an automated workforce would be.
I don’t think ‘catastrophe’ is the relevant scary endpoint; e.g., COVID was a catastrophe, but unlikely to have been x-risky. Something like a point-of-no-return (e.g. humanity getting disempowered) seems more relevant.
I’m pretty confident it’s feasible to at the very least 10x AI safety prosaic research through AI augmentation without increasing x-risk by more than 1% yearly (and that would probably be a conservative upper bound). For some intuition—see the low levels of x-risk that current AIs pose, while already having software engineering 50%-time-horizons of around 4 hours, and while already getting IMO gold medals. Both of these skills (coding and math) seem among the most useful for strongly augmenting AI safety research, especially since LLMs already seem like they might be human-level at (ML) research ideation.
Also, AFAICT, there are so many low hanging fruit to make current AIs safer, some of which I’d suspect are barely being used at all (and even with this relative recklessness, current AIs are still surprisingly safe and aligned—to the point where I think Claudes are probably already more beneficial and more prosocial companions than the median human). Things like unlearning / filtering the most dangerous and most antisocial data, or like production evaluations, or like trying harder to preserve CoT legibility through rephrasing or other forms of regularization, or, more speculatively, trying to use various forms of brain data for alignment.
By catastrophe, I was thinking of something much worse than Covid; or indeed, x-risky. Point-of-no-return is a good stand-in. So: what makes you confident that AI safety research will be automated before a point-of-no-return for humanity is crossed?
I’m pretty confident it’s feasible to at the very least 10x AI safety prosaic research through AI augmentation without increasing x-risk by more than 1% yearly
I think Claudes are probably already more beneficial and more prosocial companions than the median human
I think you (like many in the LW/EA/AIS community) might be on a slippery slope here to having your mind altered by AI use to the point of losing sight of the fact that these things are fundamentally alien underneath. (See also.)
Except that Zvi coveredthis potential evidence for misalignment and I had this to add. As for the AIs being alien underneath due to training and architecture, I and Claude Opus 4.5 came up with both a case for it and a case against it.
I think your prompt to Claude is pretty leading[1]. You are assuming the answer with “the AIs end up with motivations similar to those of the humans”. The point is that we don’t actually know what their underlying motivations are—we only see how they act when trained and system-prompted into mimicking humans. And no alignment techniques are even 3 9s reliable (and we need >13 9s in the limit of ASI).
Also “Can this crux be partially resolved by, say, studying the values of humans whose brain was developed abnormally” is not thinking at the right level of abstraction. Humans who’s brains developed abnormally are still very close to normal humans in the grand scheme of mindspace. AIs share zero evolutionary history and development (evo-devo), and close to zero brain architecture with humans. Sharing our corpus of media is a very shallow and brittle substitute (i.e. it can make a half-decent mask for the shoggoth, but it doesn’t do anything in the way of evolving the shoggoth into a digital human).
What do you mean by automating catastrophe? Is it the creation of a misaligned AGI who has a chance to escape or to (create an ASI who will) fake alignment, be given the throne and commit genocide? Automating AI safety research would have us automate generating safety-related ideas, coding, gathering or creating data sets. But I don’t think that I understand how automated coding alone will cause a catastrophe.
I doubt this would be the ideal moment for a pause, even assuming it were politically tractable, which it obviously isn’t right now.
Very likely you’d want to pause after you’ve automated AI safety research, or at least strongly (e.g. 10x) accelerated at least prosaic AI safety research (none of which has happened yet) - given how small the current AI safety human workforce is, and how much more numerous (and very likely cheaper per equivalent hour of labor) an automated workforce would be.
What makes you confident that AI safety research will be automated before catastrophe is automated?
I don’t think ‘catastrophe’ is the relevant scary endpoint; e.g., COVID was a catastrophe, but unlikely to have been x-risky. Something like a point-of-no-return (e.g. humanity getting disempowered) seems more relevant.
I’m pretty confident it’s feasible to at the very least 10x AI safety prosaic research through AI augmentation without increasing x-risk by more than 1% yearly (and that would probably be a conservative upper bound). For some intuition—see the low levels of x-risk that current AIs pose, while already having software engineering 50%-time-horizons of around 4 hours, and while already getting IMO gold medals. Both of these skills (coding and math) seem among the most useful for strongly augmenting AI safety research, especially since LLMs already seem like they might be human-level at (ML) research ideation.
Also, AFAICT, there are so many low hanging fruit to make current AIs safer, some of which I’d suspect are barely being used at all (and even with this relative recklessness, current AIs are still surprisingly safe and aligned—to the point where I think Claudes are probably already more beneficial and more prosocial companions than the median human). Things like unlearning / filtering the most dangerous and most antisocial data, or like production evaluations, or like trying harder to preserve CoT legibility through rephrasing or other forms of regularization, or, more speculatively, trying to use various forms of brain data for alignment.
By catastrophe, I was thinking of something much worse than Covid; or indeed, x-risky. Point-of-no-return is a good stand-in. So: what makes you confident that AI safety research will be automated before a point-of-no-return for humanity is crossed?
I’d agree that it’s feasible—but is it at all likely? Surely that would require us to Pause at ~the current level (as you say: “LLMs already seem like they might be human-level at (ML) research ideation.”). You aren’t getting only a 1% increase in x-risk yearly on the current trajectory.
I think you (like many in the LW/EA/AIS community) might be on a slippery slope here to having your mind altered by AI use to the point of losing sight of the fact that these things are fundamentally alien underneath. (See also.)
Except that Zvi covered this potential evidence for misalignment and I had this to add. As for the AIs being alien underneath due to training and architecture, I and Claude Opus 4.5 came up with both a case for it and a case against it.
I think your prompt to Claude is pretty leading[1]. You are assuming the answer with “the AIs end up with motivations similar to those of the humans”. The point is that we don’t actually know what their underlying motivations are—we only see how they act when trained and system-prompted into mimicking humans. And no alignment techniques are even 3 9s reliable (and we need >13 9s in the limit of ASI).
Also “Can this crux be partially resolved by, say, studying the values of humans whose brain was developed abnormally” is not thinking at the right level of abstraction. Humans who’s brains developed abnormally are still very close to normal humans in the grand scheme of mindspace. AIs share zero evolutionary history and development (evo-devo), and close to zero brain architecture with humans. Sharing our corpus of media is a very shallow and brittle substitute (i.e. it can make a half-decent mask for the shoggoth, but it doesn’t do anything in the way of evolving the shoggoth into a digital human).
not to mention that the fact that you are using Claude as a trusted source of information on this in the first place is problematic.
What do you mean by automating catastrophe? Is it the creation of a misaligned AGI who has a chance to escape or to (create an ASI who will) fake alignment, be given the throne and commit genocide? Automating AI safety research would have us automate generating safety-related ideas, coding, gathering or creating data sets. But I don’t think that I understand how automated coding alone will cause a catastrophe.
Yes, but it wouldn’t be given the throne. It will take it (or rather, just obliterate it).