This is evidence that fixing such issues even to a first approximation still takes at least many months and can’t be done faster, as o3 was already trained in some form by December[1], it’s been 4 months, and who knows how long it’ll take to actually fix. Since o3 is not larger than o1 and so releasing it doesn’t depend on securing additional hardware, plausibly the time to release was primarily determined by the difficulty of getting post-training in shape and fixing the lying (which is systematically beyond “hallucinations” on some types of queries).
If o3 is based on GPT-4.1′s base model, and the latter used pretraining knowledge distillation from GPT-4.5-base, it’s not obviously possible to do all that by the time of Dec 2024 announcement. Assuming GPT-4.5 was pretrained for 3-4 months since May 2024, the base model was done in Aug-Sep 2024, logits for the pretraining dataset for GPT-4.1 were collected by Sep-Oct 2024, and GPT-4.1 itself got pretrained by Nov-Dec 2024, with almost no margin for post-training.
The reasoning training would need to either be very fast or mostly SFT from traces of a GPT-4.5′s reasoning variant (which could start training in Sep 2024 and be done to some extent by Nov 2024). Both might be possible to do quickly R1-Zero style, so maybe this is not impossible given that o3-preview only needed to pass benchmarks and not be shown directly to anyone yet.
This is evidence that fixing such issues even to a first approximation still takes at least many months and can’t be done faster, as o3 was already trained in some form by December[1], it’s been 4 months, and who knows how long it’ll take to actually fix. Since o3 is not larger than o1 and so releasing it doesn’t depend on securing additional hardware, plausibly the time to release was primarily determined by the difficulty of getting post-training in shape and fixing the lying (which is systematically beyond “hallucinations” on some types of queries).
If o3 is based on GPT-4.1′s base model, and the latter used pretraining knowledge distillation from GPT-4.5-base, it’s not obviously possible to do all that by the time of Dec 2024 announcement. Assuming GPT-4.5 was pretrained for 3-4 months since May 2024, the base model was done in Aug-Sep 2024, logits for the pretraining dataset for GPT-4.1 were collected by Sep-Oct 2024, and GPT-4.1 itself got pretrained by Nov-Dec 2024, with almost no margin for post-training.
The reasoning training would need to either be very fast or mostly SFT from traces of a GPT-4.5′s reasoning variant (which could start training in Sep 2024 and be done to some extent by Nov 2024). Both might be possible to do quickly R1-Zero style, so maybe this is not impossible given that o3-preview only needed to pass benchmarks and not be shown directly to anyone yet.