1. The 80⁄20 isn’t a claim about sufficiency for superintelligence. I already grant we don’t know how to align a superintelligence, and Plan A might not be enough. The 80⁄20 claim says that for the risk at current and near-term capability levels, there’s already a long list of cheap measures we’re simply not doing. Buck’s “~40 things, none of which seem that hard” is the crux.
2. Even if you think best practices are insufficient, implementing them is how we’d find that out. This is the part I’d stress most. Transparency, evals, red-teaming, incident reporting, safety cases, these aren’t just mitigations: in the long run those are the instruments that generate the data on whether we’re on track or not. Right now we’re flying blind: we don’t even have the feedback loop that would tell us “the 80⁄20 isn’t enough, we need to escalate.”
Debating the ceiling while we’re this far from the floor is a bit of a luxury problem.
On the researcher-pivot point: agreed, and I’d only add that the proficiency bar for advocacy is lower than people assume (similar to Asya’s post that I reference), and the opportunities are abundant; this can be scaled.
How do you know that our current methods will scale as well as you say to problems with future models? I for example have been imagining continual learning systems as the potential problem for the last two years or so which means that a lot of the existent ways of measuring dangers change?
I understand that we need to add evidence for misalignment over time to establish a precedent but isn’t it better to model policy change as a sort of Milton Friedman model where plans are mostly useful after shifts?
I guess this is what you’re saying with the “why not just wait for a warning shot” section? But why is the feedback that we need to escalate dependent on government buy in?
I think you might be generally correct in your argument, I just notice myself being a bit confused with some of these things.
Thanks Jonas, happy to see you in the comment section.
Scaling: I’m not claiming current methods scale to superintelligence. I grant they may not, and continual learning is the kind of thing that would probably break a good part of today’s evals. But a good part of risk management is precisely detecting when our present methods start to fail (to be concrete, you can run control evaluations, like this one).
Why feedback needs government: the instruments that generate the evidence needed to escalate (third-party evals, incident reporting, training transparency) are not really enforced without a mandate (the METR risk report is a heroic effort in this direction, but that’s mostly it). Labs really won’t volunteer the data that would justify constraining them, and they are even reticent to create data that could later be used against them. Which is why I think it would be a huge win to enforce those best practices like third-party evals, etc.
Friedman: partly agree, having ideas lying around matters. But when the crisis hits, the plans that get picked up are the ones with champions already in place and decision-makers who already trust the source . If you do no will-building at first, worse ideas win by default.
Labs really won’t volunteer the data that would justify constraining them, and they are even reticent to create data that could later be used against them.
Yeah, for some reason this wasn’t part of my model which shows my experience in this area I guess. That’s a good point.
I agree with the final point as well, I guess there’s some sort of directionality and overclaiming in terms of the exact risks that I worry about? Like, if we say AI 2027 is our danger scenario and AI 2027 turns out not to be true then you lose credibility but I assume this is not how you go about building up these things.
I guess my thought would be that some of the evals advocacy might be negative EV if it isn’t pointed at things that is likely to scale to a continual learning regime? Now that doesn’t mean that the point of focusing more on advocacy is a bad one, it just is more like a prioritisation question?
2 things:
1. The 80⁄20 isn’t a claim about sufficiency for superintelligence. I already grant we don’t know how to align a superintelligence, and Plan A might not be enough. The 80⁄20 claim says that for the risk at current and near-term capability levels, there’s already a long list of cheap measures we’re simply not doing. Buck’s “~40 things, none of which seem that hard” is the crux.
2. Even if you think best practices are insufficient, implementing them is how we’d find that out. This is the part I’d stress most. Transparency, evals, red-teaming, incident reporting, safety cases, these aren’t just mitigations: in the long run those are the instruments that generate the data on whether we’re on track or not. Right now we’re flying blind: we don’t even have the feedback loop that would tell us “the 80⁄20 isn’t enough, we need to escalate.”
Debating the ceiling while we’re this far from the floor is a bit of a luxury problem.
On the researcher-pivot point: agreed, and I’d only add that the proficiency bar for advocacy is lower than people assume (similar to Asya’s post that I reference), and the opportunities are abundant; this can be scaled.
Okay, why is this?
How do you know that our current methods will scale as well as you say to problems with future models? I for example have been imagining continual learning systems as the potential problem for the last two years or so which means that a lot of the existent ways of measuring dangers change?
I understand that we need to add evidence for misalignment over time to establish a precedent but isn’t it better to model policy change as a sort of Milton Friedman model where plans are mostly useful after shifts?
I guess this is what you’re saying with the “why not just wait for a warning shot” section? But why is the feedback that we need to escalate dependent on government buy in?
I think you might be generally correct in your argument, I just notice myself being a bit confused with some of these things.
Thanks Jonas, happy to see you in the comment section.
Scaling: I’m not claiming current methods scale to superintelligence. I grant they may not, and continual learning is the kind of thing that would probably break a good part of today’s evals. But a good part of risk management is precisely detecting when our present methods start to fail (to be concrete, you can run control evaluations, like this one).
Why feedback needs government: the instruments that generate the evidence needed to escalate (third-party evals, incident reporting, training transparency) are not really enforced without a mandate (the METR risk report is a heroic effort in this direction, but that’s mostly it). Labs really won’t volunteer the data that would justify constraining them, and they are even reticent to create data that could later be used against them. Which is why I think it would be a huge win to enforce those best practices like third-party evals, etc.
Friedman: partly agree, having ideas lying around matters. But when the crisis hits, the plans that get picked up are the ones with champions already in place and decision-makers who already trust the source . If you do no will-building at first, worse ideas win by default.
Yeah, for some reason this wasn’t part of my model which shows my experience in this area I guess. That’s a good point.
I agree with the final point as well, I guess there’s some sort of directionality and overclaiming in terms of the exact risks that I worry about? Like, if we say AI 2027 is our danger scenario and AI 2027 turns out not to be true then you lose credibility but I assume this is not how you go about building up these things.
I guess my thought would be that some of the evals advocacy might be negative EV if it isn’t pointed at things that is likely to scale to a continual learning regime? Now that doesn’t mean that the point of focusing more on advocacy is a bad one, it just is more like a prioritisation question?
Thanks for answering my question!