Good point, and I think I somewhat agree. If you think we just reach an intelligence explosion at some level (seems pretty plausible), you wouldn’t update to previous pre-training levels because we’d be closer and what really matters is hitting that point (and post-training can possibly take you to that point). While it means that you shouldn’t update towards before pre-training, I still think the general point of being a large update back still stands (perhaps this point—the degree—depends on some other priors, though, which I didn’t want to get into).
Good point, and I think I somewhat agree. If you think we just reach an intelligence explosion at some level (seems pretty plausible), you wouldn’t update to previous pre-training levels because we’d be closer and what really matters is hitting that point (and post-training can possibly take you to that point). While it means that you shouldn’t update towards before pre-training, I still think the general point of being a large update back still stands (perhaps this point—the degree—depends on some other priors, though, which I didn’t want to get into).