I… really don’t know what Scott expected a story that featured actual superintelligence to look like. I think the authors bent over backwards giving us one of the least-sci-fi stories you could possibly tell that includes superintelligence doing anything at all, without resorting to “superintelligence just won’t ever exist.”
What about literally the AI 2027 story which does involve superintelligence and Scott thinks doesn’t sound “unnecessarily dramatic”. I think AI 2027 seems much more intuitively plausible to me and it seems less “sci-fi” in this sense. (I’m not saying that “less sci-fi” is much evidence it’s more likely to be true.)
The amount of “fast takeoff” seems like the amount of scaleup you’d expect if the graphs just kept going up the way they’re currently going up, by approximately the same mechanisms they currently go up (i.e. some algorithmic improvements, some scaling).
Sure, Galvanic would first run Sable on smaller amounts of compute. And… then they will run it on larger amounts of compute (and as I understand it, it’d be a new, surprising fact if they limited themselves to scaling up slowly/linearly rather than by a noticeable multiplier or order-of-magnitude. If I am wrong about current lab practices here, please link me some evidence).
I think the amount of discontinuity in the story is substantially above the amount of discontinuity in more realistic-seeming-to-me stories like AI 2027 (which is also on the faster side of what I expect, like a top 20% takeoff in terms of speed). I don’t think think extrapolating current trends predicts this much of a discontinuity.
If this story feels crazy to you, I want to remind you that all possible views of the future are wild. Either some exponential graphs suddenly stop for unclear reasons, or some exponential graphs keep going and batshit crazy stuff can happen that your intuitions are not prepared for. You can believe option A if you want, but, it’s not like “the exponential graphs that have been consistent over hundreds of years suddenly stop” is a viewpoint that you can safely point to as a “moderate” and claim to give the other guy burden of proof.
You don’t have the luxury of being the sort of moderate who doesn’t have to believe something pretty crazy sounding here, one way or another.
The relevant moderates (that are likely to be reading this etc) are predicting an AI takeoff, so I don’t think think “exponential economic growth can’t go on” is a relevant objection.
What about literally the AI 2027 story which does involve superintelligence and Scott thinks doesn’t sound “unnecessarily dramatic”. I think AI 2027 seems much more intuitively plausible to me and it seems less “sci-fi” in this sense. (I’m not saying that “less sci-fi” is much evidence it’s more likely to be true.)
I think if the AI 2027 had more details, they would look fairly similar to the ones in the Sable story. (I think the Sable story substitutes in more superpersuasion, vs military takeover via bioweapons. I think if you spelled out the details of that, it’d sound approximately as outlandish (less reliant on new tech but triggering more people to say “really? people would buy that?”. The stories otherwise seems pretty similar to me.)
I also think the AI 2027 is sort of “the earlier failure” version of the Sable story. AI 2027 is (I think?) basically a story where we hand over a lot of power of our own accord, without the AI needing to persuade us of anything, because we think we’re in a race with China and we just want a lot of economic benefit.
The IABI story is specifically trying to highlight “okay, but would it still be able to do that if we didn’t just hand it power?”, and it does need to take more steps to win in that case. (instead of inventing bioweapons to kill people, it’s probably instead inventing biomedical stuff and other cool new tech that is helpful because it’s a straightforwardly valuable, that’s the whole reason we gave it power in the first place. If you spelled out those details, it’d also seem more sci-fi-y).
It might be that the AI 2027 story is more likely because it happens first / more easily. But it’s necessary to argue the thesis of the book to tell a story with more obstacles, to highlight how the AI would overcome that. I agree that does make it more dramatic.
Both stories end with “and then it fully upgrades it’s cognitiion and invents dyson spheres and goes off conquering the universe”, which is pretty sci-fi-y.
But it’s also a bit confusing, because I don’t think they have a definition of superintelligence in the book other than “exceeds every human at almost every mental task”, so AIs that are broadly moderately superhuman ought to count.
A few pages later they say:
> We will describe it using the term “superintelligence,” meaning a mind much more capable than any human at almost every sort of steering and prediction problem — at least, those problems where there is room to substantially improve over human performance.*
I think the amount of discontinuity in the story is substantially above the amount of discontinuity in more realistic-seeming-to-me stories like AI 2027 (which is also on the faster side of what I expect, like a top 20% takeoff in terms of speed). I don’t think think extrapolating current trends predicts this much of a discontinuity.
I am pretty surprised for you to actually think this.
Here are some individual gears I think. I am pretty curious (genuinely, not just as a gambit) about your professional opinion about these:
the “smooth”-ish lines we see are made of individual lumpy things. The individual lumps usually aren’t that big, the reason you get smooth lines is when lots of little advancements are constantly happening and they turn out to add up to a relatively constant rate.
“parallel scaling” is a fairly reasonable sort of innovation, it’s not necessarily definitely-gonna-happen but it is a) the sort of thing someone might totally try doing and work, after ironing out a bunch of kinks, b) is a reasonable parallel for the invention of chain-of-thought. They could have done something more like an architectural improvement that’s more technically opaque (that’s more equivalent to inventing transformers) but that would have felt a bit more magical and harder for a lay audience to grok.
when companies are experimenting with new techniques, they tend to scale them up by at least a factor of 2 and often more after proving the concept at smaller amounts of compute.
...and scaling up a few times by a factor of 2 will sometimes result in a lump of progress that is more powerful than the corresponding scaleup of safeguards, in a way that is difficult to predict, especially when lots of companies are doing it a lot.
The story doesn’t specify a timeline – if it takes place 10 years from now it’d be significantly slower than AI 2027. So it’s not particularly obvious whether it’s more or less discontinuitous than AI 2027, or your own expectations. On an exponential graph of smoothed out lumps, larger lumps that happen later can be “a lot” without being discontinuitous(sp?).
Why do I think the story involves a lot of discontinuity (relative to what I expect)?
Right at the start of the story, Sable has much higher levels of capability than Galvanic expects. It can confortably prove the Riemann Hypothesis even though Galvanic engineers are impressed by it proving some modest theorems. Generally, it seems like for a company to be impressed by a new AI’s capabilities while it’s actual capabilities are much higher probably requires a bunch of discontinuity (or requires AIs to ongoingly sandbag more and more each generation).
There isn’t really any discussion of how the world has been changed by AI (beyond Galvanic developing (insufficient) countermeasures based on studying early systems) while Sable is seemingly competitive with top human experts or perhaps superhuman. For instance, it can prove the Riemann hypothesis with only maybe like ~$3 million in spending (assuming each GPU hour is like $2-4). It could be relatively much better at math (which seems totally plausible but not really how the story discusses it), but naively this implies the AI would be very useful for all kinds of stuff. If humans had somewhat weaker systems which were aligned enough to do useful safety work, these systems would potentially change the situation substantially. They might not (e.g. because these systems aren’t really leveraged), but either way, the world could look totally different if we already had systems around this level of capability for a bit.
I’d expect the company would first scale up a version of parallel scaling which only works a bit (because it’s still worth scaling up at this point) before scaling up a version which works very well. The story makes it seem like this is a single large breakthrough.
The company doesn’t appear to run any serious evals on the AI before deploying it (making it easier to sandbag?) nor due they make any other real effort to measure the capabilities as far as we can tell. (E.g., measure the scaling law for parallel scaling, do science on it at smaller scale, etc.)
No other AI systems other than Sable do anything meaningful in the story at any point while I’d naively expect there would be other AI systems which are at a similar level of capability, especially given how long capabilities are halted in the story.
I’m not trying to say this is totally impossible or something. I think this level of discontinuity is substantially more than I expect and substantially more than in AI 2027. (Idk, maybe like top 15% discontinuity while AI 2027 is like top 35% discontinuity, note that speed != discontinuity, though they are related.)
I expect that many of these choices (e.g. not having many other AIs around) made the story less complex and easier to write and it seems kinda reasonable to pick a story which is easier to write.
Also, I’m not claiming that the situation would have been fine/safe with less discontinuity, in many cases this might just complicate the situation without particularly helping (though I do think less discontinuity would substantially reduce the risk overall). My overall point is just that the story does actually seem less realistic on this axis to me and this is related to why it seems more sci-fi (again, “more sci-fi” doesn’t mean wrong).
I roughly agree with your 3 bullets. The main thing is that I expect that you first find a kinda shitty version of parallel scaling before you find one so good it results in a big gain in capabilities. And you probably have to do stuff like tune hyperparameters and do other science before you want to scale it up. All this means that the advance would probably be somewhat more continuous. This doesn’t mean it would be slow or safe, but it does change how things go and means that large unknown jumps in capability look less likely.
Overall, I agree companies might find a new innovation and scale it up a bunch (and do this scaling quickly). I just think the default most likely picture looks somewhat different in a way which does actually make it somewhat less scary.
What about literally the AI 2027 story which does involve superintelligence and Scott thinks doesn’t sound “unnecessarily dramatic”. I think AI 2027 seems much more intuitively plausible to me and it seems less “sci-fi” in this sense. (I’m not saying that “less sci-fi” is much evidence it’s more likely to be true.)
I think the amount of discontinuity in the story is substantially above the amount of discontinuity in more realistic-seeming-to-me stories like AI 2027 (which is also on the faster side of what I expect, like a top 20% takeoff in terms of speed). I don’t think think extrapolating current trends predicts this much of a discontinuity.
The relevant moderates (that are likely to be reading this etc) are predicting an AI takeoff, so I don’t think think “exponential economic growth can’t go on” is a relevant objection.
I think if the AI 2027 had more details, they would look fairly similar to the ones in the Sable story. (I think the Sable story substitutes in more superpersuasion, vs military takeover via bioweapons. I think if you spelled out the details of that, it’d sound approximately as outlandish (less reliant on new tech but triggering more people to say “really? people would buy that?”. The stories otherwise seems pretty similar to me.)
I also think the AI 2027 is sort of “the earlier failure” version of the Sable story. AI 2027 is (I think?) basically a story where we hand over a lot of power of our own accord, without the AI needing to persuade us of anything, because we think we’re in a race with China and we just want a lot of economic benefit.
The IABI story is specifically trying to highlight “okay, but would it still be able to do that if we didn’t just hand it power?”, and it does need to take more steps to win in that case. (instead of inventing bioweapons to kill people, it’s probably instead inventing biomedical stuff and other cool new tech that is helpful because it’s a straightforwardly valuable, that’s the whole reason we gave it power in the first place. If you spelled out those details, it’d also seem more sci-fi-y).
It might be that the AI 2027 story is more likely because it happens first / more easily. But it’s necessary to argue the thesis of the book to tell a story with more obstacles, to highlight how the AI would overcome that. I agree that does make it more dramatic.
Both stories end with “and then it fully upgrades it’s cognitiion and invents dyson spheres and goes off conquering the universe”, which is pretty sci-fi-y.
>superintelligence
Small detail: My understanding of the IABIED scenario is that their AI was only moderately superhuman, not superintelligent
I think that’s true in how they refer to it.
But it’s also a bit confusing, because I don’t think they have a definition of superintelligence in the book other than “exceeds every human at almost every mental task”, so AIs that are broadly moderately superhuman ought to count.
Edit: No wait, correction:
I am pretty surprised for you to actually think this.
Here are some individual gears I think. I am pretty curious (genuinely, not just as a gambit) about your professional opinion about these:
the “smooth”-ish lines we see are made of individual lumpy things. The individual lumps usually aren’t that big, the reason you get smooth lines is when lots of little advancements are constantly happening and they turn out to add up to a relatively constant rate.
“parallel scaling” is a fairly reasonable sort of innovation, it’s not necessarily definitely-gonna-happen but it is a) the sort of thing someone might totally try doing and work, after ironing out a bunch of kinks, b) is a reasonable parallel for the invention of chain-of-thought. They could have done something more like an architectural improvement that’s more technically opaque (that’s more equivalent to inventing transformers) but that would have felt a bit more magical and harder for a lay audience to grok.
when companies are experimenting with new techniques, they tend to scale them up by at least a factor of 2 and often more after proving the concept at smaller amounts of compute.
...and scaling up a few times by a factor of 2 will sometimes result in a lump of progress that is more powerful than the corresponding scaleup of safeguards, in a way that is difficult to predict, especially when lots of companies are doing it a lot.
The story doesn’t specify a timeline – if it takes place 10 years from now it’d be significantly slower than AI 2027. So it’s not particularly obvious whether it’s more or less discontinuitous than AI 2027, or your own expectations. On an exponential graph of smoothed out lumps, larger lumps that happen later can be “a lot” without being discontinuitous(sp?).
Why do I think the story involves a lot of discontinuity (relative to what I expect)?
Right at the start of the story, Sable has much higher levels of capability than Galvanic expects. It can confortably prove the Riemann Hypothesis even though Galvanic engineers are impressed by it proving some modest theorems. Generally, it seems like for a company to be impressed by a new AI’s capabilities while it’s actual capabilities are much higher probably requires a bunch of discontinuity (or requires AIs to ongoingly sandbag more and more each generation).
There isn’t really any discussion of how the world has been changed by AI (beyond Galvanic developing (insufficient) countermeasures based on studying early systems) while Sable is seemingly competitive with top human experts or perhaps superhuman. For instance, it can prove the Riemann hypothesis with only maybe like ~$3 million in spending (assuming each GPU hour is like $2-4). It could be relatively much better at math (which seems totally plausible but not really how the story discusses it), but naively this implies the AI would be very useful for all kinds of stuff. If humans had somewhat weaker systems which were aligned enough to do useful safety work, these systems would potentially change the situation substantially. They might not (e.g. because these systems aren’t really leveraged), but either way, the world could look totally different if we already had systems around this level of capability for a bit.
I’d expect the company would first scale up a version of parallel scaling which only works a bit (because it’s still worth scaling up at this point) before scaling up a version which works very well. The story makes it seem like this is a single large breakthrough.
The company doesn’t appear to run any serious evals on the AI before deploying it (making it easier to sandbag?) nor due they make any other real effort to measure the capabilities as far as we can tell. (E.g., measure the scaling law for parallel scaling, do science on it at smaller scale, etc.)
No other AI systems other than Sable do anything meaningful in the story at any point while I’d naively expect there would be other AI systems which are at a similar level of capability, especially given how long capabilities are halted in the story.
I’m not trying to say this is totally impossible or something. I think this level of discontinuity is substantially more than I expect and substantially more than in AI 2027. (Idk, maybe like top 15% discontinuity while AI 2027 is like top 35% discontinuity, note that speed != discontinuity, though they are related.)
I expect that many of these choices (e.g. not having many other AIs around) made the story less complex and easier to write and it seems kinda reasonable to pick a story which is easier to write.
Also, I’m not claiming that the situation would have been fine/safe with less discontinuity, in many cases this might just complicate the situation without particularly helping (though I do think less discontinuity would substantially reduce the risk overall). My overall point is just that the story does actually seem less realistic on this axis to me and this is related to why it seems more sci-fi (again, “more sci-fi” doesn’t mean wrong).
I roughly agree with your 3 bullets. The main thing is that I expect that you first find a kinda shitty version of parallel scaling before you find one so good it results in a big gain in capabilities. And you probably have to do stuff like tune hyperparameters and do other science before you want to scale it up. All this means that the advance would probably be somewhat more continuous. This doesn’t mean it would be slow or safe, but it does change how things go and means that large unknown jumps in capability look less likely.
Overall, I agree companies might find a new innovation and scale it up a bunch (and do this scaling quickly). I just think the default most likely picture looks somewhat different in a way which does actually make it somewhat less scary.