Reminds me of the predictability (or not) of Black Swans, aka Tetlock v. Taleb.
Also Tetlock’s point that nothing is truly unique: you can usually learn at least something from similar cases/reference classes. (I know @Eliezer Yudkowsky isn’t saying you can’t learn anything at all).
So the question is how much can you learn beforehand
If frontier-lab safety people think they can learn a great deal from model to model, that would be important evidence for me (do they?).
Conversely, if the claim is that transfer from previous systems to ASI is necessarily too weak, or that one future step is crucially different from all previous steps, that needs an argument beyond just asserting it and is very suspect from the forecasting experience. The prior should be against the qualitative/unpredictable/uncontrollable sudden jump.
Tbf, I may very well missing a lot of context and maybe that argument has been made elsewhere.
Reminds me of the predictability (or not) of Black Swans, aka Tetlock v. Taleb. Also Tetlock’s point that nothing is truly unique: you can usually learn at least something from similar cases/reference classes. (I know @Eliezer Yudkowsky isn’t saying you can’t learn anything at all). So the question is how much can you learn beforehand If frontier-lab safety people think they can learn a great deal from model to model, that would be important evidence for me (do they?).
Conversely, if the claim is that transfer from previous systems to ASI is necessarily too weak, or that one future step is crucially different from all previous steps, that needs an argument beyond just asserting it and is very suspect from the forecasting experience. The prior should be against the qualitative/unpredictable/uncontrollable sudden jump.
Tbf, I may very well missing a lot of context and maybe that argument has been made elsewhere.