While I have deep uncertainty on timelines, takeoff speeds, and other factors. I believe that it is highly unlikely we end up with broad superhuman AI in the next 2 years. A few of the reasons I believe this:
Current LLM architectures still make incredibly basic mistakes across a wide variety of disciplines. I would expect to see AI that can perform across all major human tasks at an adequate level prior to seeing AI that is capable of sustained self improvement. Right now using Opus4.6/4.7 half the time is like magic, and half the time is head scratching.
I think you may be misunderstanding vulnerability/0 Day exploitation a little bit. I work in cybersecurity. The vulnerabilities that Mythos is discovering all could have been easily discovered by human researchers. Vulnerabilities exist everywhere and time/effort is the primary bottleneck to find them. In a lot of ways vuln discoveries play very heavily to LLM strengths, namely broad knowledge with an expansive reference class applied to narrow verifiable problems and the ability to iterate many times to find the correct answer.
Labs have significant compute limitations that make it difficult to continue scaling at the current pace. Particularly because increasing amounts of compute are needing to go to paying customers to sustain rates of growth and investment.
Internal bottlenecks at labs are likely to begin playing a greater role. LLM capabilities are becoming truly high-risk in some areas (Biological, Cyber, Chemical) and my best guess is that AI providers will have to increasingly invest in safeguards and mundane alignment to prevent a minor catastrophe that could bring down regulation.
With lower confidence I also am somewhat skeptical of the Transformer/LLM architecture scaling to a feature complete AGI. Lack of continuous learning, sample efficiency, and generalization may be insurmountable with this particular architecture.
Sorry, answering quickly with mostly cached thoughts without engaging deeply:
Current LLM architectures can probably do everything. There are ways of compiling code into transformer’s weights. they’re not good at everything, yet, and generally end up kinda spiky; but it’d be surprising if you couldn’t just scale them up and do a lot more RL and get something pretty general.
While the 0days discovered by Mythos are all of the kind that could be discovered by humans, humans have in fact looked at some of the code, a lot, manually and using instruments, and failed to find this stuff. I don’t dispute these are not on a level above best human cybersecurity researchers (and have a market on whether there will be any such vulnerabilities discovered by AI that couldn’t have been discovered by humans at all, in the next few years). Being just as good as best humans but much faster is sufficient to take over the world.
SSI has zero customers, they still get billions of dollars. Google has a lot of money and compute and use AI to develop better chips/compute infra. There are lots of efficiency gains that stack, and some of those are getting automated. I can imagine three main AI companies running out of money, but find that unlikely. The promise of higher intelligence is worth a lot: even if it’s expensive to get to it and to use it, it’s cheaper than paying humans to perform this same tasks, and due to that the demand is enormous.
I’d not want to bet on AI companies forever being unable to solve jailbreaks sir caring sufficiently about those to not release models at all/ever. In the limit, you do a huge ton of classifiers of texts and activations and report users who try to do bad things to authorities, monitor wetlabs/have honeypot wetlabs that LLMs recommend, make LLMs unaware of some knowledge, etc.; if this really prevents AI companies from releasing and earning tens+ of billions of dollars, they will throw billions of dollars at solving this and will solve this successfully. (E.g., imagine anyone who submits a working jailbreak gets $10k, up to a million submissions each of which gets patches together with similar things; do you think it even gets to a million submissions?)
”Alignment“ of current models to not teach people to kill a lot of people mostly has nothing to do with the difficulties we’d face at the superintelligent level.
What is a thing a transformer architecture cannot do in principle? Like, are you imagining that if we figure out how to make literal superintelligences, something would prevent us from compiling them into transformers? Given in-context learning and scratchpad maintenance are allowed etc.
While I have deep uncertainty on timelines, takeoff speeds, and other factors. I believe that it is highly unlikely we end up with broad superhuman AI in the next 2 years. A few of the reasons I believe this:
Current LLM architectures still make incredibly basic mistakes across a wide variety of disciplines. I would expect to see AI that can perform across all major human tasks at an adequate level prior to seeing AI that is capable of sustained self improvement. Right now using Opus4.6/4.7 half the time is like magic, and half the time is head scratching.
I think you may be misunderstanding vulnerability/0 Day exploitation a little bit. I work in cybersecurity. The vulnerabilities that Mythos is discovering all could have been
easilydiscovered by human researchers. Vulnerabilities exist everywhere and time/effort is the primary bottleneck to find them. In a lot of ways vuln discoveries play very heavily to LLM strengths, namely broad knowledge with an expansive reference class applied to narrow verifiable problems and the ability to iterate many times to find the correct answer.Labs have significant compute limitations that make it difficult to continue scaling at the current pace. Particularly because increasing amounts of compute are needing to go to paying customers to sustain rates of growth and investment.
Internal bottlenecks at labs are likely to begin playing a greater role. LLM capabilities are becoming truly high-risk in some areas (Biological, Cyber, Chemical) and my best guess is that AI providers will have to increasingly invest in safeguards and mundane alignment to prevent a minor catastrophe that could bring down regulation.
With lower confidence I also am somewhat skeptical of the Transformer/LLM architecture scaling to a feature complete AGI. Lack of continuous learning, sample efficiency, and generalization may be insurmountable with this particular architecture.
Sorry, answering quickly with mostly cached thoughts without engaging deeply:
Current LLM architectures can probably do everything. There are ways of compiling code into transformer’s weights. they’re not good at everything, yet, and generally end up kinda spiky; but it’d be surprising if you couldn’t just scale them up and do a lot more RL and get something pretty general.
While the 0days discovered by Mythos are all of the kind that could be discovered by humans, humans have in fact looked at some of the code, a lot, manually and using instruments, and failed to find this stuff. I don’t dispute these are not on a level above best human cybersecurity researchers (and have a market on whether there will be any such vulnerabilities discovered by AI that couldn’t have been discovered by humans at all, in the next few years). Being just as good as best humans but much faster is sufficient to take over the world.
SSI has zero customers, they still get billions of dollars. Google has a lot of money and compute and use AI to develop better chips/compute infra. There are lots of efficiency gains that stack, and some of those are getting automated. I can imagine three main AI companies running out of money, but find that unlikely. The promise of higher intelligence is worth a lot: even if it’s expensive to get to it and to use it, it’s cheaper than paying humans to perform this same tasks, and due to that the demand is enormous.
I’d not want to bet on AI companies forever being unable to solve jailbreaks sir caring sufficiently about those to not release models at all/ever. In the limit, you do a huge ton of classifiers of texts and activations and report users who try to do bad things to authorities, monitor wetlabs/have honeypot wetlabs that LLMs recommend, make LLMs unaware of some knowledge, etc.; if this really prevents AI companies from releasing and earning tens+ of billions of dollars, they will throw billions of dollars at solving this and will solve this successfully. (E.g., imagine anyone who submits a working jailbreak gets $10k, up to a million submissions each of which gets patches together with similar things; do you think it even gets to a million submissions?)
”Alignment“ of current models to not teach people to kill a lot of people mostly has nothing to do with the difficulties we’d face at the superintelligent level.
What is a thing a transformer architecture cannot do in principle? Like, are you imagining that if we figure out how to make literal superintelligences, something would prevent us from compiling them into transformers? Given in-context learning and scratchpad maintenance are allowed etc.