“Evals don’t represent the real world, and are increasingly recognized as evals by AIs. And, capabilities we don’t know about, and can’t predict or account for from first principles, can arise in the cracks left behind by that fact. So unless we can fix this, there’s a real chance our attempts to coordinate and steer policy go astray because we are blind to what’s going on.”
I think it makes sense, but I also think that focusing on evals isn’t going to help without first actually having a governance regime that cares about them, rather than it being eval outfits completely dependent on frontier lab funding, or government orgs that aren’t willing to say “extinction risk” out loud. To get to that point, we need the force of the democratic process on our side, and a real pause. Else, there will be no time or willpower to do careful evals like you describe; eval outfits only get a few days to a few weeks to do their work before a model is released, and there are too many incentives for decision makers to downplay or ignore bad news such as high eval awareness. I think if you were to get what you wish for, we’d have to already be in a competent slowdown regime.
I also think that a noticeable number of bad futures are locked in before evals even happen, during internal deployment or even development. Careful evals do nothing there; you would need to basically solve alignment for those, and no one has a credible plan for that that scales to arbitrary levels of intelligence.
Yeah, i think that that makes a lot of sense. I’m personally thinking of trying to do evals that INFORM policy, although advocacy is also not off the table.
To summarize,
“Evals don’t represent the real world, and are increasingly recognized as evals by AIs. And, capabilities we don’t know about, and can’t predict or account for from first principles, can arise in the cracks left behind by that fact. So unless we can fix this, there’s a real chance our attempts to coordinate and steer policy go astray because we are blind to what’s going on.”
Does that sound about right?
Yeah i think so! Do you think this is mostly true?
I think it makes sense, but I also think that focusing on evals isn’t going to help without first actually having a governance regime that cares about them, rather than it being eval outfits completely dependent on frontier lab funding, or government orgs that aren’t willing to say “extinction risk” out loud. To get to that point, we need the force of the democratic process on our side, and a real pause. Else, there will be no time or willpower to do careful evals like you describe; eval outfits only get a few days to a few weeks to do their work before a model is released, and there are too many incentives for decision makers to downplay or ignore bad news such as high eval awareness. I think if you were to get what you wish for, we’d have to already be in a competent slowdown regime.
I also think that a noticeable number of bad futures are locked in before evals even happen, during internal deployment or even development. Careful evals do nothing there; you would need to basically solve alignment for those, and no one has a credible plan for that that scales to arbitrary levels of intelligence.
Yeah, i think that that makes a lot of sense. I’m personally thinking of trying to do evals that INFORM policy, although advocacy is also not off the table.