you’re assuming sublinear scaling. why wouldn’t it be superlinear post training? it certainly seems like it is now. it need not be sharply superlinear like yud expected to still be superlinear.
Exactly! I’d expect compute to scale way better than humans—not necessarily because the intelligence of compute scales so well, but because the intelligence of human groups scales so poorly...
So I assumed a specific relationship between “one unit of human-brain power”, and “super intelligence capable of killing humanity”, where I use human-brain power as a unit but that doesn’t actually have to be linear scaling—imagine this is a graph with two labeled data points, with a point at (human, X:1) and then another point at (SI, X:10B), you can draw many different curves that connect those two labeled points and the Y axis is sort of arbitrary.
Now maybe 10B HBP to kill humanity seems too high, but I assume humanity as a civilization which includes a ton of other compute, AI, and AGI, and I don’t really put much credence in strong nanotech.
you’re assuming sublinear scaling. why wouldn’t it be superlinear post training? it certainly seems like it is now. it need not be sharply superlinear like yud expected to still be superlinear.
Exactly! I’d expect compute to scale way better than humans—not necessarily because the intelligence of compute scales so well, but because the intelligence of human groups scales so poorly...
So I assumed a specific relationship between “one unit of human-brain power”, and “super intelligence capable of killing humanity”, where I use human-brain power as a unit but that doesn’t actually have to be linear scaling—imagine this is a graph with two labeled data points, with a point at (human, X:1) and then another point at (SI, X:10B), you can draw many different curves that connect those two labeled points and the Y axis is sort of arbitrary.
Now maybe 10B HBP to kill humanity seems too high, but I assume humanity as a civilization which includes a ton of other compute, AI, and AGI, and I don’t really put much credence in strong nanotech.