I heard some rumors that gpt 4.5 got good pretraining loss but bad downstream performance. If that’s true the loss scaling laws may have worked correctly. If not, yeah a lot of things can go wrong and something did, whether that’s hardware issues, software bugs, or machine learning problems or problems with their earlier experiments
I heard some rumors that gpt 4.5 got good pretraining loss but bad downstream performance. If that’s true the loss scaling laws may have worked correctly. If not, yeah a lot of things can go wrong and something did, whether that’s hardware issues, software bugs, or machine learning problems or problems with their earlier experiments