Thanks Jonas, happy to see you in the comment section.
Scaling: I’m not claiming current methods scale to superintelligence. I grant they may not, and continual learning is the kind of thing that would probably break a good part of today’s evals. But a good part of risk management is precisely detecting when our present methods start to fail (to be concrete, you can run control evaluations, like this one).
Why feedback needs government: the instruments that generate the evidence needed to escalate (third-party evals, incident reporting, training transparency) are not really enforced without a mandate (the METR risk report is a heroic effort in this direction, but that’s mostly it). Labs really won’t volunteer the data that would justify constraining them, and they are even reticent to create data that could later be used against them. Which is why I think it would be a huge win to enforce those best practices like third-party evals, etc.
Friedman: partly agree, having ideas lying around matters. But when the crisis hits, the plans that get picked up are the ones with champions already in place and decision-makers who already trust the source . If you do no will-building at first, worse ideas win by default.
Labs really won’t volunteer the data that would justify constraining them, and they are even reticent to create data that could later be used against them.
Yeah, for some reason this wasn’t part of my model which shows my experience in this area I guess. That’s a good point.
I agree with the final point as well, I guess there’s some sort of directionality and overclaiming in terms of the exact risks that I worry about? Like, if we say AI 2027 is our danger scenario and AI 2027 turns out not to be true then you lose credibility but I assume this is not how you go about building up these things.
I guess my thought would be that some of the evals advocacy might be negative EV if it isn’t pointed at things that is likely to scale to a continual learning regime? Now that doesn’t mean that the point of focusing more on advocacy is a bad one, it just is more like a prioritisation question?
Thanks Jonas, happy to see you in the comment section.
Scaling: I’m not claiming current methods scale to superintelligence. I grant they may not, and continual learning is the kind of thing that would probably break a good part of today’s evals. But a good part of risk management is precisely detecting when our present methods start to fail (to be concrete, you can run control evaluations, like this one).
Why feedback needs government: the instruments that generate the evidence needed to escalate (third-party evals, incident reporting, training transparency) are not really enforced without a mandate (the METR risk report is a heroic effort in this direction, but that’s mostly it). Labs really won’t volunteer the data that would justify constraining them, and they are even reticent to create data that could later be used against them. Which is why I think it would be a huge win to enforce those best practices like third-party evals, etc.
Friedman: partly agree, having ideas lying around matters. But when the crisis hits, the plans that get picked up are the ones with champions already in place and decision-makers who already trust the source . If you do no will-building at first, worse ideas win by default.
Yeah, for some reason this wasn’t part of my model which shows my experience in this area I guess. That’s a good point.
I agree with the final point as well, I guess there’s some sort of directionality and overclaiming in terms of the exact risks that I worry about? Like, if we say AI 2027 is our danger scenario and AI 2027 turns out not to be true then you lose credibility but I assume this is not how you go about building up these things.
I guess my thought would be that some of the evals advocacy might be negative EV if it isn’t pointed at things that is likely to scale to a continual learning regime? Now that doesn’t mean that the point of focusing more on advocacy is a bad one, it just is more like a prioritisation question?
Thanks for answering my question!