I’m sure this has been discussed elsewhere ad nauseam, but this view always struck me as extremely overconfident: We have no idea what a superintelligence could discover using today’s data or experiment throughput capacity, and “close to human level” seems a priori unlikely.
Yeah. Given the level of variance among humans we know significant variance is possible even within the strict bounds of “minds that run on 3 pounds of meat and 20 watts of sugar, most of which is spent on things other than thought.” I find this to be a pretty strong argument that the limits at which we hit diminishing returns should be rather far away.
They other side of my mental model is, if (our best model of) the laws of physics fit on a postcard, what exactly does it mean to need to do an experiment, in principle? You need experiments to nail down the laws. Beyond that, they’re convenient for reducing computational requirements, often vastly so, but it’s not something that prevents you from getting things right on the first try way more often than humans do.
My charitable musing is that maybe Dario genuinely hasn’t met anyone whom he judged as smarter than himself by a wide enough margin to develop this intuition via humans, which I think is the somewhat easier path to really feeling the possibility internally than approaching it abstractly/intellectually provides.
if (our best model of) the laws of physics fit on a postcard, what exactly does it mean to need to do an experiment, in principle? You need experiments to nail down the laws. Beyond that, they’re convenient for reducing computational requirements
The general point is solid, but you also need experiments to learn contingent things within physics e.g. how biology works.
Yes. I doubt it would be practical for ASI to solve biology by simulating fundamental physics of various 50-100kg lumps of atoms and seeing what matches available data on humans. I also doubt itbwould need anywhere remotely close to the number of experiments we need to draw the lessons it needs to solve any particular biology problem.
I think Dario assumes wrongly the scope of intelligence itself; let’s say an agent can improve itself roughly from the level of AlexNet to your average coding agent nowadays (eg Opus 4.6). The gap between these is staggering; even if there is some upper limit who’s to say it’s close?
The human mind itself is much more efficient than 8 H100s; eventually a self-improving agent would top out at that (or become more computationally efficient than us) and by that point I’d argue you couldn’t tell the difference between “very superintelligent” and “wildly superintelligent”.
The laws of physics tell you how things progress, but don’t tell you the starting point. So even if you had the enormous computational power to deduce biology from physics, you would still need to know what a dog is to say something useful about it. So maybe an important question is: would a superintelligence have enough information of earth and its inhabitants so it would not need any more info because it can just reason about it, or would it need to gather more?
I’m sure there is still data it would need to collect. I think it’s a mistake to use the amount and type of data humans require as a guide to what that might mean.
I’m sure this has been discussed elsewhere ad nauseam, but this view always struck me as extremely overconfident: We have no idea what a superintelligence could discover using today’s data or experiment throughput capacity, and “close to human level” seems a priori unlikely.
Yeah. Given the level of variance among humans we know significant variance is possible even within the strict bounds of “minds that run on 3 pounds of meat and 20 watts of sugar, most of which is spent on things other than thought.” I find this to be a pretty strong argument that the limits at which we hit diminishing returns should be rather far away.
They other side of my mental model is, if (our best model of) the laws of physics fit on a postcard, what exactly does it mean to need to do an experiment, in principle? You need experiments to nail down the laws. Beyond that, they’re convenient for reducing computational requirements, often vastly so, but it’s not something that prevents you from getting things right on the first try way more often than humans do.
My charitable musing is that maybe Dario genuinely hasn’t met anyone whom he judged as smarter than himself by a wide enough margin to develop this intuition via humans, which I think is the somewhat easier path to really feeling the possibility internally than approaching it abstractly/intellectually provides.
The general point is solid, but you also need experiments to learn contingent things within physics e.g. how biology works.
Yes. I doubt it would be practical for ASI to solve biology by simulating fundamental physics of various 50-100kg lumps of atoms and seeing what matches available data on humans. I also doubt itbwould need anywhere remotely close to the number of experiments we need to draw the lessons it needs to solve any particular biology problem.
I think Dario assumes wrongly the scope of intelligence itself; let’s say an agent can improve itself roughly from the level of AlexNet to your average coding agent nowadays (eg Opus 4.6). The gap between these is staggering; even if there is some upper limit who’s to say it’s close?
The human mind itself is much more efficient than 8 H100s; eventually a self-improving agent would top out at that (or become more computationally efficient than us) and by that point I’d argue you couldn’t tell the difference between “very superintelligent” and “wildly superintelligent”.
The laws of physics tell you how things progress, but don’t tell you the starting point. So even if you had the enormous computational power to deduce biology from physics, you would still need to know what a dog is to say something useful about it. So maybe an important question is: would a superintelligence have enough information of earth and its inhabitants so it would not need any more info because it can just reason about it, or would it need to gather more?
I’m sure there is still data it would need to collect. I think it’s a mistake to use the amount and type of data humans require as a guide to what that might mean.