@StanislavKrym:
My impression is that pre-deployment evals currently take orgs like METR a few weeks. If this is still about the amount of time that pre-deployment evals take, then optimistically (this might be pretty expensive in practice) you’d be deploying models that haven’t been trained online for the past several weeks but have been audited. It’s unclear how big an issue this would be in practice.
@StanislavKrym: My impression is that pre-deployment evals currently take orgs like METR a few weeks. If this is still about the amount of time that pre-deployment evals take, then optimistically (this might be pretty expensive in practice) you’d be deploying models that haven’t been trained online for the past several weeks but have been audited. It’s unclear how big an issue this would be in practice.