I’ve heard from a credible source that OpenAI substantially overestimated where other AI companies were at with respect to RL and reasoning when they released o1. Employees at OpenAI believed that other top AI companies had already figured out similar things when they actually hadn’t and were substantially behind. OpenAI had been sitting the improvements driving o1 for a while prior to releasing it. Correspondingly, releasing o1 resulted in much larger capabilities externalities than OpenAI expected. I think there was one more case like this either from OpenAI or GDM where employees had a large misimpression about capabilities progress at other companies causing a release they wouldn’t do otherwise.
One key takeaway from this is that employees at AI companies might be very bad at predicting the situation at other AI companies (likely making coordination more difficult by default). This includes potentially thinking they are in a close race when they actually aren’t. Another update is that keeping secrets about something like reasoning models worked surprisingly well to prevent other companies from copying OpenAI’s work even though there was a bunch of public reporting (and presumably many rumors) about this.
One more update is that OpenAI employees might unintentionally accelerate capabilities progress at other actors via overestimating how close they are. My vague understanding was that they haven’t updated much, but I’m unsure. (Consider updating more if you’re an OpenAI employee!)
Interesting. What confuses me a bit: What made other companies be able to copy OpenAI’s work after it was released, conditional on your story being true? As far as I know, OpenAI didn’t actually explain their methods developing o1, so what exactly did other companies learn from the release which they didn’t learn from the rumors that OpenAI is developing something like this?
Is the conclusion basically that Jesse Hoogland has been right that just the few bits that OpenAI did leak already constrained the space of possibilities enough for others to copy the work? Quote from his post:
For all its secrecy, OpenAI has leaked enough bits to tightly constrain the space of possibilities.
The few bits they leaked in the release helped a bunch. Note that these bits were substantially leaked via people being able to use the model rather than necessarily via the blog post.
Other companies weren’t that motivated to try to copy OpenAI’s work until it was released as they we’re sure how important it was or how good the results were.
“Employees at OpenAI believed…” — do you mean Sam Altman and the board?
If this information is accurate, it speaks volumes about how flawed their alignment predictions might also be. If a company with vast resources and insider access like OpenAI can’t predict the capabilities of competing firms (a relatively simple problem with objectively knowable answers), how can we expect them to predict the behavior of advanced AI models, where the unknowns are far greater and often unknowable?
I’ve heard from a credible source that OpenAI substantially overestimated where other AI companies were at with respect to RL and reasoning when they released o1. Employees at OpenAI believed that other top AI companies had already figured out similar things when they actually hadn’t and were substantially behind. OpenAI had been sitting the improvements driving o1 for a while prior to releasing it. Correspondingly, releasing o1 resulted in much larger capabilities externalities than OpenAI expected. I think there was one more case like this either from OpenAI or GDM where employees had a large misimpression about capabilities progress at other companies causing a release they wouldn’t do otherwise.
One key takeaway from this is that employees at AI companies might be very bad at predicting the situation at other AI companies (likely making coordination more difficult by default). This includes potentially thinking they are in a close race when they actually aren’t. Another update is that keeping secrets about something like reasoning models worked surprisingly well to prevent other companies from copying OpenAI’s work even though there was a bunch of public reporting (and presumably many rumors) about this.
One more update is that OpenAI employees might unintentionally accelerate capabilities progress at other actors via overestimating how close they are. My vague understanding was that they haven’t updated much, but I’m unsure. (Consider updating more if you’re an OpenAI employee!)
Alex Mallen also noted a connection with people generally thinking they are in race when they actually aren’t: https://forum.effectivealtruism.org/posts/cXBznkfoPJAjacFoT/are-you-really-in-a-race-the-cautionary-tales-of-szilard-and
Interesting.
What confuses me a bit: What made other companies be able to copy OpenAI’s work after it was released, conditional on your story being true? As far as I know, OpenAI didn’t actually explain their methods developing o1, so what exactly did other companies learn from the release which they didn’t learn from the rumors that OpenAI is developing something like this?
Is the conclusion basically that Jesse Hoogland has been right that just the few bits that OpenAI did leak already constrained the space of possibilities enough for others to copy the work? Quote from his post:
I think:
The few bits they leaked in the release helped a bunch. Note that these bits were substantially leaked via people being able to use the model rather than necessarily via the blog post.
Other companies weren’t that motivated to try to copy OpenAI’s work until it was released as they we’re sure how important it was or how good the results were.
“Employees at OpenAI believed…” — do you mean Sam Altman and the board?
If this information is accurate, it speaks volumes about how flawed their alignment predictions might also be. If a company with vast resources and insider access like OpenAI can’t predict the capabilities of competing firms (a relatively simple problem with objectively knowable answers), how can we expect them to predict the behavior of advanced AI models, where the unknowns are far greater and often unknowable?