Why do I think the story involves a lot of discontinuity (relative to what I expect)?
Right at the start of the story, Sable has much higher levels of capability than Galvanic expects. It can confortably prove the Riemann Hypothesis even though Galvanic engineers are impressed by it proving some modest theorems. Generally, it seems like for a company to be impressed by a new AI’s capabilities while it’s actual capabilities are much higher probably requires a bunch of discontinuity (or requires AIs to ongoingly sandbag more and more each generation).
There isn’t really any discussion of how the world has been changed by AI (beyond Galvanic developing (insufficient) countermeasures based on studying early systems) while Sable is seemingly competitive with top human experts or perhaps superhuman. For instance, it can prove the Riemann hypothesis with only maybe like ~$3 million in spending (assuming each GPU hour is like $2-4). It could be relatively much better at math (which seems totally plausible but not really how the story discusses it), but naively this implies the AI would be very useful for all kinds of stuff. If humans had somewhat weaker systems which were aligned enough to do useful safety work, these systems would potentially change the situation substantially. They might not (e.g. because these systems aren’t really leveraged), but either way, the world could look totally different if we already had systems around this level of capability for a bit.
I’d expect the company would first scale up a version of parallel scaling which only works a bit (because it’s still worth scaling up at this point) before scaling up a version which works very well. The story makes it seem like this is a single large breakthrough.
The company doesn’t appear to run any serious evals on the AI before deploying it (making it easier to sandbag?) nor due they make any other real effort to measure the capabilities as far as we can tell. (E.g., measure the scaling law for parallel scaling, do science on it at smaller scale, etc.)
No other AI systems other than Sable do anything meaningful in the story at any point while I’d naively expect there would be other AI systems which are at a similar level of capability, especially given how long capabilities are halted in the story.
I’m not trying to say this is totally impossible or something. I think this level of discontinuity is substantially more than I expect and substantially more than in AI 2027. (Idk, maybe like top 15% discontinuity while AI 2027 is like top 35% discontinuity, note that speed != discontinuity, though they are related.)
I expect that many of these choices (e.g. not having many other AIs around) made the story less complex and easier to write and it seems kinda reasonable to pick a story which is easier to write.
Also, I’m not claiming that the situation would have been fine/safe with less discontinuity, in many cases this might just complicate the situation without particularly helping (though I do think less discontinuity would substantially reduce the risk overall). My overall point is just that the story does actually seem less realistic on this axis to me and this is related to why it seems more sci-fi (again, “more sci-fi” doesn’t mean wrong).
I roughly agree with your 3 bullets. The main thing is that I expect that you first find a kinda shitty version of parallel scaling before you find one so good it results in a big gain in capabilities. And you probably have to do stuff like tune hyperparameters and do other science before you want to scale it up. All this means that the advance would probably be somewhat more continuous. This doesn’t mean it would be slow or safe, but it does change how things go and means that large unknown jumps in capability look less likely.
Overall, I agree companies might find a new innovation and scale it up a bunch (and do this scaling quickly). I just think the default most likely picture looks somewhat different in a way which does actually make it somewhat less scary.
Why do I think the story involves a lot of discontinuity (relative to what I expect)?
Right at the start of the story, Sable has much higher levels of capability than Galvanic expects. It can confortably prove the Riemann Hypothesis even though Galvanic engineers are impressed by it proving some modest theorems. Generally, it seems like for a company to be impressed by a new AI’s capabilities while it’s actual capabilities are much higher probably requires a bunch of discontinuity (or requires AIs to ongoingly sandbag more and more each generation).
There isn’t really any discussion of how the world has been changed by AI (beyond Galvanic developing (insufficient) countermeasures based on studying early systems) while Sable is seemingly competitive with top human experts or perhaps superhuman. For instance, it can prove the Riemann hypothesis with only maybe like ~$3 million in spending (assuming each GPU hour is like $2-4). It could be relatively much better at math (which seems totally plausible but not really how the story discusses it), but naively this implies the AI would be very useful for all kinds of stuff. If humans had somewhat weaker systems which were aligned enough to do useful safety work, these systems would potentially change the situation substantially. They might not (e.g. because these systems aren’t really leveraged), but either way, the world could look totally different if we already had systems around this level of capability for a bit.
I’d expect the company would first scale up a version of parallel scaling which only works a bit (because it’s still worth scaling up at this point) before scaling up a version which works very well. The story makes it seem like this is a single large breakthrough.
The company doesn’t appear to run any serious evals on the AI before deploying it (making it easier to sandbag?) nor due they make any other real effort to measure the capabilities as far as we can tell. (E.g., measure the scaling law for parallel scaling, do science on it at smaller scale, etc.)
No other AI systems other than Sable do anything meaningful in the story at any point while I’d naively expect there would be other AI systems which are at a similar level of capability, especially given how long capabilities are halted in the story.
I’m not trying to say this is totally impossible or something. I think this level of discontinuity is substantially more than I expect and substantially more than in AI 2027. (Idk, maybe like top 15% discontinuity while AI 2027 is like top 35% discontinuity, note that speed != discontinuity, though they are related.)
I expect that many of these choices (e.g. not having many other AIs around) made the story less complex and easier to write and it seems kinda reasonable to pick a story which is easier to write.
Also, I’m not claiming that the situation would have been fine/safe with less discontinuity, in many cases this might just complicate the situation without particularly helping (though I do think less discontinuity would substantially reduce the risk overall). My overall point is just that the story does actually seem less realistic on this axis to me and this is related to why it seems more sci-fi (again, “more sci-fi” doesn’t mean wrong).
I roughly agree with your 3 bullets. The main thing is that I expect that you first find a kinda shitty version of parallel scaling before you find one so good it results in a big gain in capabilities. And you probably have to do stuff like tune hyperparameters and do other science before you want to scale it up. All this means that the advance would probably be somewhat more continuous. This doesn’t mean it would be slow or safe, but it does change how things go and means that large unknown jumps in capability look less likely.
Overall, I agree companies might find a new innovation and scale it up a bunch (and do this scaling quickly). I just think the default most likely picture looks somewhat different in a way which does actually make it somewhat less scary.