Well yes, nobody thinks that existing techniques suffice to build de-novo self-replicating nano machines, but that means it’s not very informative to comment on the fallibility of this or that package or the time complexity of some currently known best approach without grounding in the necessity of that approach.
One has to argue instead based on the fundamental underlying shape of the problem, and saying accurate simulation is O(n⁷) is not particularly more informative to that than saying accurate protein folding is NP. I think if the claim is that you can’t make directionally informative predictions via simulation for things meaningfully larger than helium then one is taking the argument beyond where it can be validly applied. If the claim is not that, it would be good to hear it clearly stated.
And what reason do you have for thinking it can’t be usefully approximated in some sufficiently productive domain, that wouldn’t also invalidly apply to protein folding? I think it’s not useful to just restate that there exist reasons you know of, I’m aiming to actually elicit those arguments here.
Thanks, I appreciate the attempt to clarify. I do though think there’s some fundamental disagreement about what we’re arguing over here that’s making it less productive than it could be. For example,
The fact that this has been an extremely active area of research for over 80 years with massive real-world implications, and we’re no closer to finding such a simplified heuristic.
I think both:
Lack of human progress doesn’t necessarily mean the problem is intrinsically unsolvable by advanced AI. Humans often take a bunch of time before proving things.
It seems not at all the case that algorithmic progress isn’t happening, so it’s hardly a given that we’re no closer to a solution unless you first circularly assume that there’s no solution to arrive at.
If you’re starting out with an argument that we’re not there yet, this makes me think more that there’s some fundamental disagreement about how we should reason about ASI, more than your belief being backed by a justification that would be convincing to me had only I succeeded at eliciting it. Claiming that a thing is hard is at most a reason not to rule out that it’s impossible. It’s not a reason on its own to believe that it is impossible.
With regard to complexity,
I failed to understand the specific difference with protein folding. Protein folding is NP-hard, which is significantly harder than O(n³).
I failed to find the source for the claim that O(n³) or O(n⁴) are optimal. Actually I’m pretty confused how this is even a likely concept; surely if the O(n³) algorithm is widely useful then the O(n⁴) proof can’t be that strong of a bound on practical usefulness? So why is this not true of the O(n³) proof as well?
It’s maybe true that protein folding is easier to computationally verify solutions to, but first, can you prove this, and second, on what basis are you claiming that existing knowledge is necessarily insufficient to develop better heuristics than the ones we already have? The claim doesn’t seem to complete to me.
It’s magical thinking to assume that an AI will just one-shot this into existence.
Please note that I’ve not been making the claim that ASI could necessarily solve this problem. I have been making the claim that the arguments in this post don’t usefully support the claim that it can’t. It is true that largely on priors I expect it should be able to, but my priors also aren’t particularly useful ones to this debate and I have tried to avoid making comments that are dependent on them.
If you say “Indeed it’s provable that you can’t have a faster algorithm than those O(n^3) and O(n^4) approximations which cover all relevant edge cases accurately” I am quite likely to go on a digression where I try to figure out what proof you’re pointing at and why you think it’s a fundamental barrier, and it seems now that per a couple of your comments you don’t believe it’s a fundamental barrier, but at the same time it doesn’t feel like any position has been moved, so I’m left rather foggy about where progress has been made.
I think it’s very useful that you say
I’m not saying that AI can’t develop useful heuristic approximations for the simulation of gemstone-based nano-mechanical machinery operating in ultra-high vacuum. I’m saying that it can’t do so as a one-shot inference without any new experimental work
since this seems like a narrower place to scope our conversation. I read this to mean:
You don’t know of any in principle barrier to solving this problem,
You believe the solution is underconstrained by available evidence.
I find the second point hard to believe, and don’t really see anywhere you have evidenced it.
As a maybe-relevant aside to that, wrt.
You’re saying that AI could take the garbage and by mere application of thought turn it into something useful. That’s not in line with the actual history of the development of useful AI outputs.
I think you’re talking of ‘mere application of thought’ like it’s not the distinguishing feature humanity has. I don’t care what’s ‘in line with the actual history’ of AI, I care what a literal superintelligence could do, and this includes a bunch of possibilities like:
Making inhumanly close observation of all existing data,
Noticing new, inhumanly-complex regularities in said data,
Proving new simplifying regularities from theory,
Inventing new algorithms for heuristic simulation,
Finding restricted domains where easier regularities hold,
Bifurcating problem space and operating over each plausible set,
Sending an interesting email to a research lab to get choice high-ROI data.
We can ignore the last one for this conversation. I still don’t understand why the others are deemed unreasonable ways of making progress on this task.
I appreciated the comments on time complexity but am skipping it because I don’t expect at this point that it lies at the crux.
One quick intuition pump: do you think a team of 10,000 of the smartest human engineers and scientists could do this if they had perfect photographic memory, were immortal, and could think for a billion years?
To keep the situation analogous to an AI needing to do this quickly, we’ll suppose this team of humans is subject to the same restrictions on compute for simulations (e.g. over the entire billion years they only get 10^30 FLOP) and also can only run whatever experiments the AI would get to run (effectively no interesting experiments).
I feel uncertain about whether this team would succeed, but it does seem pretty plausible that they would be able to succeed. Perhaps I think they’re 40% likely to succeed?
Now, suppose the superintelligence is like this, but even more capable.
Separately, I don’t think it’s very important to know what an extremely powerful superintelligence could do, because prior to the point where you have an extremely powerful superintelligence, humanity will already be obsoleted by weaker AIs. So, I think Yudkowsky’s arguments about nanotech are mostly unimportant for other reasons.
But, if you did think “well, sure the AI might be arbitrarily smart, but if we don’t give it access to the nukes what can it even do to us?” then I think that there are many sources of concern and nanotech is certainly one of them.
I don’t understand where your confidence is coming from here, but fair enough. It wasn’t clear to me if your take was more like “wildly, wildly superintelligent AI will be considerably weaker than a team of humans thinking for a billion years” or more like “literally impossible without either experiments or >>10^30 FLOP”.
I generally have an intuition like “it’s really, really hard to rule out physically possible things out without very strong evidence, by default things have a reasonable chance of being possible (e.g. 50%) when sufficient intelligence is applied if they are physically possible”. It seems you don’t share this intuition, fair enough.
(I feel like this applies for nearly all human inventions? Like if you had access to a huge amount of video of the world from 1900 and all written books that existed at this point, and had the affordances I described with a team of 10,000 people, 10^30 FLOP, and a billion years, it seems to me like there is a good chance you’d be able to one-shot reinvent ~all inventions of modern humanity (not doing everything in the same way, in many cases you’d massively over engineer to handle one-shot). Planes seem pretty easy? Rockets seem doable?)
Please don’t use That Alien Message as an intuition pump.
I think this is an ok, but not amazing intuition pump for what wildly, wildly superintelligent AI could be like.
The best thing you can do is rid yourself of the notion that superhuman AI would have arbitrary capabilities
I separately think it’s not very important to think about the abilities of wildly, wildly superintelligent AI for most purposes (as I noted in my comment). So I agree that imagining arbitrary capabilities is probablematic. (For some evidence that this isn’t post-hoc justification, see this post on which I’m an author.)
PS: We’ve had AGI since 2017. That’d better be compatible with your world view if you want accurate predictions.
Uhhhh, I’m not sure I agree with this as it doesn’t seem like nearly all jobs are easily fully automatable by AI. Perhaps you use a definition of AGI which is much weaker like “able to speak slightly coherant english (GPT-1?) and classify images”?
My assumption is that when people say AGI here they mean Bostrom’s ASI, and they got linked because Eliezer believed (and believes still?) that AGI will FOOM into ASI almost immediately, which it has not.
In case this wasn’t clear from early discussion, I disagree with Eliezer on a number of topics, including takeoff speeds. In particular I disagree about the time from AI that is economically transformative to AI that is much, much more powerful.
I think you’ll probably find it healthier and more productive to not think of LW as an amorphous collective and instead note that there are a variety of different people who post on the forum with a variety of different views. (I sometimes have made this mistake in the past and I find it healthy to clarify at least internally.)
E.g. instead of saying “LW has bad views about X” say “a high fraction of people who comment on LW have bad views about X” or “a high fraction of karma votes seem to be from people with bad views about X”. Then, you should maybe double check the extent to which a given claim is actualy right : ). For instance, I don’t think almost immediate FOOM is the typical view on LW when aggregating by most metrics, a somewhat longer duration takeoff is now a more common view I think.
By the way, where’s this number coming from? (10^30 FLOP) You keep repeating it.
Extremely rough and slightly conservatively small ball park number for how many FLOP will be used to create powerful AIs. The idea being that this will represent roughly how many FLOP could plausibly be available at the time.
GPT-4 is ~10^26 FLOP, I expect GPT-7 is maybe 10^30 FLOP.
Perhaps this is a bit too much because the scheming AI will have access to far few FLOP than exist at the time, but I expect this isn’t cruxy, so I just did a vague guess.
I greatly appreciate the effort in this reply, but I think it’s increasingly unclear to me how to make efficient progress on our disagreements, so I’m going to hop.
.
Could you quote or else clearly reference a specific argument from the post you found convincing on that topic?
.
Well yes, nobody thinks that existing techniques suffice to build de-novo self-replicating nano machines, but that means it’s not very informative to comment on the fallibility of this or that package or the time complexity of some currently known best approach without grounding in the necessity of that approach.
One has to argue instead based on the fundamental underlying shape of the problem, and saying accurate simulation is O(n⁷) is not particularly more informative to that than saying accurate protein folding is NP. I think if the claim is that you can’t make directionally informative predictions via simulation for things meaningfully larger than helium then one is taking the argument beyond where it can be validly applied. If the claim is not that, it would be good to hear it clearly stated.
.
And what reason do you have for thinking it can’t be usefully approximated in some sufficiently productive domain, that wouldn’t also invalidly apply to protein folding? I think it’s not useful to just restate that there exist reasons you know of, I’m aiming to actually elicit those arguments here.
.
Thanks, I appreciate the attempt to clarify. I do though think there’s some fundamental disagreement about what we’re arguing over here that’s making it less productive than it could be. For example,
I think both:
Lack of human progress doesn’t necessarily mean the problem is intrinsically unsolvable by advanced AI. Humans often take a bunch of time before proving things.
It seems not at all the case that algorithmic progress isn’t happening, so it’s hardly a given that we’re no closer to a solution unless you first circularly assume that there’s no solution to arrive at.
If you’re starting out with an argument that we’re not there yet, this makes me think more that there’s some fundamental disagreement about how we should reason about ASI, more than your belief being backed by a justification that would be convincing to me had only I succeeded at eliciting it. Claiming that a thing is hard is at most a reason not to rule out that it’s impossible. It’s not a reason on its own to believe that it is impossible.
With regard to complexity,
I failed to understand the specific difference with protein folding. Protein folding is NP-hard, which is significantly harder than O(n³).
I failed to find the source for the claim that O(n³) or O(n⁴) are optimal. Actually I’m pretty confused how this is even a likely concept; surely if the O(n³) algorithm is widely useful then the O(n⁴) proof can’t be that strong of a bound on practical usefulness? So why is this not true of the O(n³) proof as well?
It’s maybe true that protein folding is easier to computationally verify solutions to, but first, can you prove this, and second, on what basis are you claiming that existing knowledge is necessarily insufficient to develop better heuristics than the ones we already have? The claim doesn’t seem to complete to me.
Please note that I’ve not been making the claim that ASI could necessarily solve this problem. I have been making the claim that the arguments in this post don’t usefully support the claim that it can’t. It is true that largely on priors I expect it should be able to, but my priors also aren’t particularly useful ones to this debate and I have tried to avoid making comments that are dependent on them.
.
If you say “Indeed it’s provable that you can’t have a faster algorithm than those O(n^3) and O(n^4) approximations which cover all relevant edge cases accurately” I am quite likely to go on a digression where I try to figure out what proof you’re pointing at and why you think it’s a fundamental barrier, and it seems now that per a couple of your comments you don’t believe it’s a fundamental barrier, but at the same time it doesn’t feel like any position has been moved, so I’m left rather foggy about where progress has been made.
I think it’s very useful that you say
since this seems like a narrower place to scope our conversation. I read this to mean:
You don’t know of any in principle barrier to solving this problem,
You believe the solution is underconstrained by available evidence.
I find the second point hard to believe, and don’t really see anywhere you have evidenced it.
As a maybe-relevant aside to that, wrt.
I think you’re talking of ‘mere application of thought’ like it’s not the distinguishing feature humanity has. I don’t care what’s ‘in line with the actual history’ of AI, I care what a literal superintelligence could do, and this includes a bunch of possibilities like:
Making inhumanly close observation of all existing data,
Noticing new, inhumanly-complex regularities in said data,
Proving new simplifying regularities from theory,
Inventing new algorithms for heuristic simulation,
Finding restricted domains where easier regularities hold,
Bifurcating problem space and operating over each plausible set,
Sending an interesting email to a research lab to get choice high-ROI data.
We can ignore the last one for this conversation. I still don’t understand why the others are deemed unreasonable ways of making progress on this task.
I appreciated the comments on time complexity but am skipping it because I don’t expect at this point that it lies at the crux.
.
One quick intuition pump: do you think a team of 10,000 of the smartest human engineers and scientists could do this if they had perfect photographic memory, were immortal, and could think for a billion years?
To keep the situation analogous to an AI needing to do this quickly, we’ll suppose this team of humans is subject to the same restrictions on compute for simulations (e.g. over the entire billion years they only get 10^30 FLOP) and also can only run whatever experiments the AI would get to run (effectively no interesting experiments).
I feel uncertain about whether this team would succeed, but it does seem pretty plausible that they would be able to succeed. Perhaps I think they’re 40% likely to succeed?
Now, suppose the superintelligence is like this, but even more capable.
See also That Alien Message
Separately, I don’t think it’s very important to know what an extremely powerful superintelligence could do, because prior to the point where you have an extremely powerful superintelligence, humanity will already be obsoleted by weaker AIs. So, I think Yudkowsky’s arguments about nanotech are mostly unimportant for other reasons.
But, if you did think “well, sure the AI might be arbitrarily smart, but if we don’t give it access to the nukes what can it even do to us?” then I think that there are many sources of concern and nanotech is certainly one of them.
.
I don’t understand where your confidence is coming from here, but fair enough. It wasn’t clear to me if your take was more like “wildly, wildly superintelligent AI will be considerably weaker than a team of humans thinking for a billion years” or more like “literally impossible without either experiments or >>10^30 FLOP”.
I generally have an intuition like “it’s really, really hard to rule out physically possible things out without very strong evidence, by default things have a reasonable chance of being possible (e.g. 50%) when sufficient intelligence is applied if they are physically possible”. It seems you don’t share this intuition, fair enough.
(I feel like this applies for nearly all human inventions? Like if you had access to a huge amount of video of the world from 1900 and all written books that existed at this point, and had the affordances I described with a team of 10,000 people, 10^30 FLOP, and a billion years, it seems to me like there is a good chance you’d be able to one-shot reinvent ~all inventions of modern humanity (not doing everything in the same way, in many cases you’d massively over engineer to handle one-shot). Planes seem pretty easy? Rockets seem doable?)
I think this is an ok, but not amazing intuition pump for what wildly, wildly superintelligent AI could be like.
I separately think it’s not very important to think about the abilities of wildly, wildly superintelligent AI for most purposes (as I noted in my comment). So I agree that imagining arbitrary capabilities is probablematic. (For some evidence that this isn’t post-hoc justification, see this post on which I’m an author.)
Uhhhh, I’m not sure I agree with this as it doesn’t seem like nearly all jobs are easily fully automatable by AI. Perhaps you use a definition of AGI which is much weaker like “able to speak slightly coherant english (GPT-1?) and classify images”?
.
FWIW, I do taboo this term and thus didn’t use it in this conversation until you introduced it.
.
Definition in the OpenAI Charter:
A post on the topic by Richard (AGI = beats most human experts).
In case this wasn’t clear from early discussion, I disagree with Eliezer on a number of topics, including takeoff speeds. In particular I disagree about the time from AI that is economically transformative to AI that is much, much more powerful.
I think you’ll probably find it healthier and more productive to not think of LW as an amorphous collective and instead note that there are a variety of different people who post on the forum with a variety of different views. (I sometimes have made this mistake in the past and I find it healthy to clarify at least internally.)
E.g. instead of saying “LW has bad views about X” say “a high fraction of people who comment on LW have bad views about X” or “a high fraction of karma votes seem to be from people with bad views about X”. Then, you should maybe double check the extent to which a given claim is actualy right : ). For instance, I don’t think almost immediate FOOM is the typical view on LW when aggregating by most metrics, a somewhat longer duration takeoff is now a more common view I think.
Also, I’m going to peace out of this discussion FYI.
Extremely rough and slightly conservatively small ball park number for how many FLOP will be used to create powerful AIs. The idea being that this will represent roughly how many FLOP could plausibly be available at the time.
GPT-4 is ~10^26 FLOP, I expect GPT-7 is maybe 10^30 FLOP.
Perhaps this is a bit too much because the scheming AI will have access to far few FLOP than exist at the time, but I expect this isn’t cruxy, so I just did a vague guess.
I wasn’t trying to justify anything, just noting my stance.
I greatly appreciate the effort in this reply, but I think it’s increasingly unclear to me how to make efficient progress on our disagreements, so I’m going to hop.