Seeing this new internal model solve open after open math problem shortly after training commenced was the wildest thing I have ever witnessed at my time at OpenAI
Possibly. But tweets like this one from someone on the RL team appear to indicate that this is something weird. Would certainly be nice if OpenAI gave some more details!
Since August 28 we have been training a new internal model that has exhibited unprecedented performance in our benchmarks, including mathematics.
**On Tuesday, September 1**, [...] we launched an effort to evaluate it on all open Millennium Prize problems.
These quotes from OpenAI’s blogpost on Navier-Stokes appear to suggest that a model with under a week of “training” (presumably RL/post—training) solved it.
Is this as completely insane as it appears to be?
PSA: I find you can totally just use Opus 4.6 instead of Opus 4.7/4.8/5 for ~most things that are sub-Fable level. The ease of understanding outweighs the capability gain for me nearly always.
(interested to hear if anyone has tried this and found that the trade-off was not good for them)
Did earlier models ever try to break out of their sandbox and fail? Or are the current crop of the models the first to even try? This seems relevant for whether this is new misalignment emerging at a particular capability level, or already-existing misalignment hitting a capability level where it can have obvious repercussions
If you thought that there was likely to be a serious cybersecurity crisis in the next 1-2 years, how could you trade on this? It is somewhat unclear to me what you would expect to do well vs. badly.
Seems plausible! Perhaps the Astra results are the first outputs of this pipeline? I imagine it wouldn’t be super hard to put together a functional pipeline here (maybe this is naive, but apparently Fable solved 5⁄10 of these with a bare prompt)
Does anyone know how systematically OpenAI/Anthropic are throwing the models at open conjectures? Are they selecting conjectures that seem tractable? Or are they just throwing thousands at the wall and seeing which small subset end up getting solved?
This seems relevant! If the models are solving a high proportion of the conjectures given to them, that means something quite different than if they are solving a tiny proportion (in particular, it might update you towards these conjectures being outliers in “easiness”).
My expectation is that the labs are probably (?) doing the firehose of conjectures thing, based on the fact that the recent 10 Astra results apparently only cost 2k in total.
in the third, it stopped the attack when it believed that it was real.
this seems good and worth studying
it would be interesting to get some rate of how often models engage in this behaviour when it is available to them, and how often they continue to completion
Very quick take inspired by seeing repeated cycles of “wow this model is insane”/qualitative/vibey analysis on ~every new frontier model release.
LLMs are, at least in part, trained to look good (/impressive?) to humans.
A huge amount of optimization pressure is going towards this. It is definitely the case that LLMs can do lots of impressive, verifiable things. But it seems plausible to me that their impressiveness is inflated in our minds to some extent because they are optimized to seem impressive to us.
I am very unsure how much correction this effect requires (possibly very little?). I am unsure how to check whether this is really happening or not. But it does seem worth noting that when something is optimized hard to look good to us, it’s not that weird that it looks good to us.
If this is a big effect, then this might point towards the “mind-blown” vibey qualitative responses to new models actually not being a great indicator. And benchmarks/METR-graph-style stuff become more important. This is concerning, because benchmarks also don’t feel like great indicators! And I feel like many people (including me) are relying more and more on vibes-based analysis from people we trust or personal qualitative experience over benchmarks.
(This also may be happening less now due to increased RLVR.)
Hm, I wonder if sprinkling in some pairs of examples of reward hacking with explicit verbalization/no verbalization being reinforced/punished respectively might help the model learn to verbalize when it reward hacks?
I’m finding it hard to set aside the fact that I really want access to Fable when assessing the export control action. I am still pretty confident that I think it’s bad, but I expect this might be a problem for objectively assessing more coherent actions at some point in the future
update: i lost access at exactly 21:59
As of 21:54 ET (~1 hour after the statement on twitter), I still have access to Fable. Seems unclear when this goes into effect.
Is there some central source for tracking major cybersecurity incidents?
Seems like it would be useful to check for inflection points on the inevitable eventual release of a Mythos-level model without the Fable guardrails.
Maybe! It feels weird that the personality and general vibe of Mythos wouldn’t have transferred over though (if that’s what it was distilled from). My weakly held guess is that 4.7/4.8 were beaten with RL too much, or something
Opus 4.7/4.8 feel the least Claude-like of any Claude model I’ve interacted with, but Mythos/Fable is very much back in the Claude basin. I am very curious as to what happened with 4.7/4.8 to make them so weird and different.
Nice example! My procrastination often takes the form of mentally hugely inflating the stakes for no reason and I have definitely noticed this specific fake pre-requisite thing happening to me pretty frequently
This is a big part of the appeal of travel for me. I feel like time speeds up and and presence drops a lot when my brain has a sufficiently good representation of the location I’m in.
This resignation seems to have gone way more viral than I would have expected. (almost 300k likes??)
Anyone have any idea why this is