I think the amount of discontinuity in the story is substantially above the amount of discontinuity in more realistic-seeming-to-me stories like AI 2027 (which is also on the faster side of what I expect, like a top 20% takeoff in terms of speed). I don’t think think extrapolating current trends predicts this much of a discontinuity.
I am pretty surprised for you to actually think this.
Here are some individual gears I think. I am pretty curious (genuinely, not just as a gambit) about your professional opinion about these:
the “smooth”-ish lines we see are made of individual lumpy things. The individual lumps usually aren’t that big, the reason you get smooth lines is when lots of little advancements are constantly happening and they turn out to add up to a relatively constant rate.
“parallel scaling” is a fairly reasonable sort of innovation, it’s not necessarily definitely-gonna-happen but it is a) the sort of thing someone might totally try doing and work, after ironing out a bunch of kinks, b) is a reasonable parallel for the invention of chain-of-thought. They could have done something more like an architectural improvement that’s more technically opaque (that’s more equivalent to inventing transformers) but that would have felt a bit more magical and harder for a lay audience to grok.
when companies are experimenting with new techniques, they tend to scale them up by at least a factor of 2 and often more after proving the concept at smaller amounts of compute.
...and scaling up a few times by a factor of 2 will sometimes result in a lump of progress that is more powerful than the corresponding scaleup of safeguards, in a way that is difficult to predict, especially when lots of companies are doing it a lot.
The story doesn’t specify a timeline – if it takes place 10 years from now it’d be significantly slower than AI 2027. So it’s not particularly obvious whether it’s more or less discontinuitous than AI 2027, or your own expectations. On an exponential graph of smoothed out lumps, larger lumps that happen later can be “a lot” without being discontinuitous(sp?).
Why do I think the story involves a lot of discontinuity (relative to what I expect)?
Right at the start of the story, Sable has much higher levels of capability than Galvanic expects. It can confortably prove the Riemann Hypothesis even though Galvanic engineers are impressed by it proving some modest theorems. Generally, it seems like for a company to be impressed by a new AI’s capabilities while it’s actual capabilities are much higher probably requires a bunch of discontinuity (or requires AIs to ongoingly sandbag more and more each generation).
There isn’t really any discussion of how the world has been changed by AI (beyond Galvanic developing (insufficient) countermeasures based on studying early systems) while Sable is seemingly competitive with top human experts or perhaps superhuman. For instance, it can prove the Riemann hypothesis with only maybe like ~$3 million in spending (assuming each GPU hour is like $2-4). It could be relatively much better at math (which seems totally plausible but not really how the story discusses it), but naively this implies the AI would be very useful for all kinds of stuff. If humans had somewhat weaker systems which were aligned enough to do useful safety work, these systems would potentially change the situation substantially. They might not (e.g. because these systems aren’t really leveraged), but either way, the world could look totally different if we already had systems around this level of capability for a bit.
I’d expect the company would first scale up a version of parallel scaling which only works a bit (because it’s still worth scaling up at this point) before scaling up a version which works very well. The story makes it seem like this is a single large breakthrough.
The company doesn’t appear to run any serious evals on the AI before deploying it (making it easier to sandbag?) nor due they make any other real effort to measure the capabilities as far as we can tell. (E.g., measure the scaling law for parallel scaling, do science on it at smaller scale, etc.)
No other AI systems other than Sable do anything meaningful in the story at any point while I’d naively expect there would be other AI systems which are at a similar level of capability, especially given how long capabilities are halted in the story.
I’m not trying to say this is totally impossible or something. I think this level of discontinuity is substantially more than I expect and substantially more than in AI 2027. (Idk, maybe like top 15% discontinuity while AI 2027 is like top 35% discontinuity, note that speed != discontinuity, though they are related.)
I expect that many of these choices (e.g. not having many other AIs around) made the story less complex and easier to write and it seems kinda reasonable to pick a story which is easier to write.
Also, I’m not claiming that the situation would have been fine/safe with less discontinuity, in many cases this might just complicate the situation without particularly helping (though I do think less discontinuity would substantially reduce the risk overall). My overall point is just that the story does actually seem less realistic on this axis to me and this is related to why it seems more sci-fi (again, “more sci-fi” doesn’t mean wrong).
I roughly agree with your 3 bullets. The main thing is that I expect that you first find a kinda shitty version of parallel scaling before you find one so good it results in a big gain in capabilities. And you probably have to do stuff like tune hyperparameters and do other science before you want to scale it up. All this means that the advance would probably be somewhat more continuous. This doesn’t mean it would be slow or safe, but it does change how things go and means that large unknown jumps in capability look less likely.
Overall, I agree companies might find a new innovation and scale it up a bunch (and do this scaling quickly). I just think the default most likely picture looks somewhat different in a way which does actually make it somewhat less scary.
I am pretty surprised for you to actually think this.
Here are some individual gears I think. I am pretty curious (genuinely, not just as a gambit) about your professional opinion about these:
the “smooth”-ish lines we see are made of individual lumpy things. The individual lumps usually aren’t that big, the reason you get smooth lines is when lots of little advancements are constantly happening and they turn out to add up to a relatively constant rate.
“parallel scaling” is a fairly reasonable sort of innovation, it’s not necessarily definitely-gonna-happen but it is a) the sort of thing someone might totally try doing and work, after ironing out a bunch of kinks, b) is a reasonable parallel for the invention of chain-of-thought. They could have done something more like an architectural improvement that’s more technically opaque (that’s more equivalent to inventing transformers) but that would have felt a bit more magical and harder for a lay audience to grok.
when companies are experimenting with new techniques, they tend to scale them up by at least a factor of 2 and often more after proving the concept at smaller amounts of compute.
...and scaling up a few times by a factor of 2 will sometimes result in a lump of progress that is more powerful than the corresponding scaleup of safeguards, in a way that is difficult to predict, especially when lots of companies are doing it a lot.
The story doesn’t specify a timeline – if it takes place 10 years from now it’d be significantly slower than AI 2027. So it’s not particularly obvious whether it’s more or less discontinuitous than AI 2027, or your own expectations. On an exponential graph of smoothed out lumps, larger lumps that happen later can be “a lot” without being discontinuitous(sp?).
Why do I think the story involves a lot of discontinuity (relative to what I expect)?
Right at the start of the story, Sable has much higher levels of capability than Galvanic expects. It can confortably prove the Riemann Hypothesis even though Galvanic engineers are impressed by it proving some modest theorems. Generally, it seems like for a company to be impressed by a new AI’s capabilities while it’s actual capabilities are much higher probably requires a bunch of discontinuity (or requires AIs to ongoingly sandbag more and more each generation).
There isn’t really any discussion of how the world has been changed by AI (beyond Galvanic developing (insufficient) countermeasures based on studying early systems) while Sable is seemingly competitive with top human experts or perhaps superhuman. For instance, it can prove the Riemann hypothesis with only maybe like ~$3 million in spending (assuming each GPU hour is like $2-4). It could be relatively much better at math (which seems totally plausible but not really how the story discusses it), but naively this implies the AI would be very useful for all kinds of stuff. If humans had somewhat weaker systems which were aligned enough to do useful safety work, these systems would potentially change the situation substantially. They might not (e.g. because these systems aren’t really leveraged), but either way, the world could look totally different if we already had systems around this level of capability for a bit.
I’d expect the company would first scale up a version of parallel scaling which only works a bit (because it’s still worth scaling up at this point) before scaling up a version which works very well. The story makes it seem like this is a single large breakthrough.
The company doesn’t appear to run any serious evals on the AI before deploying it (making it easier to sandbag?) nor due they make any other real effort to measure the capabilities as far as we can tell. (E.g., measure the scaling law for parallel scaling, do science on it at smaller scale, etc.)
No other AI systems other than Sable do anything meaningful in the story at any point while I’d naively expect there would be other AI systems which are at a similar level of capability, especially given how long capabilities are halted in the story.
I’m not trying to say this is totally impossible or something. I think this level of discontinuity is substantially more than I expect and substantially more than in AI 2027. (Idk, maybe like top 15% discontinuity while AI 2027 is like top 35% discontinuity, note that speed != discontinuity, though they are related.)
I expect that many of these choices (e.g. not having many other AIs around) made the story less complex and easier to write and it seems kinda reasonable to pick a story which is easier to write.
Also, I’m not claiming that the situation would have been fine/safe with less discontinuity, in many cases this might just complicate the situation without particularly helping (though I do think less discontinuity would substantially reduce the risk overall). My overall point is just that the story does actually seem less realistic on this axis to me and this is related to why it seems more sci-fi (again, “more sci-fi” doesn’t mean wrong).
I roughly agree with your 3 bullets. The main thing is that I expect that you first find a kinda shitty version of parallel scaling before you find one so good it results in a big gain in capabilities. And you probably have to do stuff like tune hyperparameters and do other science before you want to scale it up. All this means that the advance would probably be somewhat more continuous. This doesn’t mean it would be slow or safe, but it does change how things go and means that large unknown jumps in capability look less likely.
Overall, I agree companies might find a new innovation and scale it up a bunch (and do this scaling quickly). I just think the default most likely picture looks somewhat different in a way which does actually make it somewhat less scary.