Thanks for the writeup. One confusion I think many have that I didn’t see addressed here is the distinction between the version of Mythos Preview that was launched on April 7th (henceforth ‘Launch’) and the early pre-release checkpoint that some where given access to (‘Early’).
Some of the less impressive results for Mythos Preview were using the early checkpoint (AISI’s original results, CTI-REALM, also METR Time Horizon results) which seems to be notably less capable than the April 7th release. AISI include some benchmarks with both ‘early’ and ‘launch’ (that they call ‘new’) in their most recent piece on AI progress trends:
CyberGym is interested as an example of a benchmark that afaik uses the ‘launch’ version of Mythos but still only has it a small amount ahead of 5.5
Thanks for the writeup. One confusion I think many have that I didn’t see addressed here is the distinction between the version of Mythos Preview that was launched on April 7th (henceforth ‘Launch’) and the early pre-release checkpoint that some where given access to (‘Early’).
Some of the less impressive results for Mythos Preview were using the early checkpoint (AISI’s original results, CTI-REALM, also METR Time Horizon results) which seems to be notably less capable than the April 7th release. AISI include some benchmarks with both ‘early’ and ‘launch’ (that they call ‘new’) in their most recent piece on AI progress trends:
CyberGym is interested as an example of a benchmark that afaik uses the ‘launch’ version of Mythos but still only has it a small amount ahead of 5.5