One quick intuition pump: do you think a team of 10,000 of the smartest human engineers and scientists could do this if they had perfect photographic memory, were immortal, and could think for a billion years?
By merely thinking about it, and not running any experiments? No, absolutely not. I don’t you understood my post if you assume I’d think otherwise.
Try this: I’m holding a certain number of fingers behind my back. You and a team of 10,000 of the smartest human engineers and scientists have a billion years to decide, without looking, what your guess will be as to how many fingers I’m holding behind my back. But you only get one chance to guess at the end of that billion years.
Please don’t use That Alien Message as an intuition pump. There’s a tremendous amount wrong with the sci-fi story. Not least of which is that it completely violates the same constraint you put into your own post about constraining computation. I suggest doing your own analysis of how many thought-seconds the AI would have in-between frames of video, especially if you assume it to be running as a large inference model.
The best thing you can do is rid yourself of the notion that superhuman AI would have arbitrary capabilities. That is where EY went wrong, and a lot of the LW crowd too. If you permit dividing by zero or multiplying by infinity, then you can easily convince yourself of anything. AI isn’t magic, and AGI isn’t a free pass to believe anything.
PS: We’ve had AGI since 2017. That’d better be compatible with your world view if you want accurate predictions.
I don’t understand where your confidence is coming from here, but fair enough. It wasn’t clear to me if your take was more like “wildly, wildly superintelligent AI will be considerably weaker than a team of humans thinking for a billion years” or more like “literally impossible without either experiments or >>10^30 FLOP”.
I generally have an intuition like “it’s really, really hard to rule out physically possible things out without very strong evidence, by default things have a reasonable chance of being possible (e.g. 50%) when sufficient intelligence is applied if they are physically possible”. It seems you don’t share this intuition, fair enough.
(I feel like this applies for nearly all human inventions? Like if you had access to a huge amount of video of the world from 1900 and all written books that existed at this point, and had the affordances I described with a team of 10,000 people, 10^30 FLOP, and a billion years, it seems to me like there is a good chance you’d be able to one-shot reinvent ~all inventions of modern humanity (not doing everything in the same way, in many cases you’d massively over engineer to handle one-shot). Planes seem pretty easy? Rockets seem doable?)
Please don’t use That Alien Message as an intuition pump.
I think this is an ok, but not amazing intuition pump for what wildly, wildly superintelligent AI could be like.
The best thing you can do is rid yourself of the notion that superhuman AI would have arbitrary capabilities
I separately think it’s not very important to think about the abilities of wildly, wildly superintelligent AI for most purposes (as I noted in my comment). So I agree that imagining arbitrary capabilities is probablematic. (For some evidence that this isn’t post-hoc justification, see this post on which I’m an author.)
PS: We’ve had AGI since 2017. That’d better be compatible with your world view if you want accurate predictions.
Uhhhh, I’m not sure I agree with this as it doesn’t seem like nearly all jobs are easily fully automatable by AI. Perhaps you use a definition of AGI which is much weaker like “able to speak slightly coherant english (GPT-1?) and classify images”?
By the way, where’s this number coming from? You keep repeating it. That amount of calculation is equivalent to running the largest supercomputer in existence for 30k years. You hypothetical AI scheming breakout is not going to have access to that much compute. Be reasonable.
I generally have an intuition like “it’s really, really hard to rule out physically possible things out without very strong evidence, by default things have a reasonable chance of being possible (e.g. 50%) when sufficient intelligence is applied if they are physically possible”
Ok let’s try a different tract. You want to come up with a molecular mechanics model that can efficiently predict the outcome of reactions, so that you can get about designing one-shot nanotechnology bootstrapping. What would success look like?
You can’t actually do a full simulation to get ground truth for training a better molecular mechanics model. So how would you know the model you came up will work as intended? You can back-test against published results in the literature, but surprise surprise, a big chunk of scientific papers don’t replicate. Shoddy lab technique, publication pressure, and a niche domain combine to create conditions where papers are rushed and sometimes not 100% truthful. Even without deliberate fraud (which also happens), you run into problems such as synthesis steps not working as advertised, images used from different experimental runs than the one described in the paper, etc.
Except you don’t know that. You’re not allowed to do experiments! Maybe you guess that replication will be an issue, although why that would be a hypothesis in the first place without first seeing failures in the lab first isn’t clear to me. But let’s say you do. Which experiments should you discount? Which should you assume to be correct? If you’re allowed to start choosing which reported results you believe and which you don’t, you’ve lost the plot. There could be millions of possible heuristics which partially match the literature and there’s no way to tell the difference.
So what would success look like? How would you know you have the right molecular mechanics model that gives accurate predictions?
You can’t. Not any more than Descartes could think his way to absolute truth.
Also for what it’s worth you’ve made a couple of logical errors here. You are considering human inventions which already exist, then saying that you could one-shot invent them in 1900. That’s hindsight bias, but also selection bias. Nanotechnology doesn’t exist. Even if it would work if created, there’s no existence proof that there exists an accessible path to achieving it. Like superheavy atoms in the island of stability, or micro black holes, there just might not be a pathway to make them from present day capabilities. (Obviously I don’t believe this as I’m running a company attempting to bootstrap Drexlarian nanotechnology, but I feel it’s essential to point out the logical error.)
(Re: Alien Message) I think this is an ok, but not amazing intuition pump for what wildly, wildly superintelligent AI could be like.
Why? You’ve gone into circular logic here.
I pointed out that the Alien Message story makes fundamental errors with respect to computational capability being wildly out of scale, so actual super intelligent AIs aren’t going to be anything like the one in the story.
Maybe a Jupiter-sized matryoshka brain made of computronium would exhibit this level of super intelligence. I’m not saying it’s not physically possible. But in terms of sketching out and bounding the capabilities of near-term AI/ASI, it’s a fucking terrible intuition pump.
Uhhhh, I’m not sure I agree with this as it doesn’t seem like nearly all jobs are easily fully automatable by AI. Perhaps you use a definition of AGI which is much weaker like “able to speak slightly coherant english (GPT-1?) and classify images”?
The transformer architecture introduced in 2017 is:
Artificial: man-made
General: able to train over arbitrary unstructured input, from which it infers models that can be applied in arbitrarily ways to find solutions of problems drawn from domains outside of its training data.
Intelligent: able to construct efficient solutions to new problems it hasn’t seen.
Artificial General Intelligence. A.G.I.
If you’re thinking “yeah, but..” then I suggest you taboo the term AGI. This is literally all that it the word means.
If you want to quibble over dates then maybe we can agree on 2022 with the introduction of ChatGPT, a truly universal (AKA general) interface to mature transformer technology. Either way we’re already well within the era of artificial general intelligence.
(Maybe EY’s very public meltdown a year ago is making more sense now? But rest easy, EY’s predictions about AI x-risk have consistently been wildly off the mark.)
You highlighted “disagree” on the part about AGI’s definition. I don’t know how to respond to that directly, so I’ll do so here. Here’s the story about how the term “AGI” was coined, by the guy who published the literal book on AGI and ran the AGI conference series for the past two decades:
LW seems to have adopted some other vague, ill-defined, threatening meaning for the acronym “AGI” that is never specified. My assumption is that when people say AGI here they mean Bostrom’s ASI, and they got linked because Eliezer believed (and believes still?) that AGI will FOOM into ASI almost immediately, which it has not.
Anyway it irks me that the term has been coopted here. AGI is a term of art in the pre-ML era of AI research with a clearly defined meaning.
My assumption is that when people say AGI here they mean Bostrom’s ASI, and they got linked because Eliezer believed (and believes still?) that AGI will FOOM into ASI almost immediately, which it has not.
In case this wasn’t clear from early discussion, I disagree with Eliezer on a number of topics, including takeoff speeds. In particular I disagree about the time from AI that is economically transformative to AI that is much, much more powerful.
I think you’ll probably find it healthier and more productive to not think of LW as an amorphous collective and instead note that there are a variety of different people who post on the forum with a variety of different views. (I sometimes have made this mistake in the past and I find it healthy to clarify at least internally.)
E.g. instead of saying “LW has bad views about X” say “a high fraction of people who comment on LW have bad views about X” or “a high fraction of karma votes seem to be from people with bad views about X”. Then, you should maybe double check the extent to which a given claim is actualy right : ). For instance, I don’t think almost immediate FOOM is the typical view on LW when aggregating by most metrics, a somewhat longer duration takeoff is now a more common view I think.
By the way, where’s this number coming from? (10^30 FLOP) You keep repeating it.
Extremely rough and slightly conservatively small ball park number for how many FLOP will be used to create powerful AIs. The idea being that this will represent roughly how many FLOP could plausibly be available at the time.
GPT-4 is ~10^26 FLOP, I expect GPT-7 is maybe 10^30 FLOP.
Perhaps this is a bit too much because the scheming AI will have access to far few FLOP than exist at the time, but I expect this isn’t cruxy, so I just did a vague guess.
By merely thinking about it, and not running any experiments? No, absolutely not. I don’t you understood my post if you assume I’d think otherwise.
Try this: I’m holding a certain number of fingers behind my back. You and a team of 10,000 of the smartest human engineers and scientists have a billion years to decide, without looking, what your guess will be as to how many fingers I’m holding behind my back. But you only get one chance to guess at the end of that billion years.
That’s a more comparable example.
Please don’t use That Alien Message as an intuition pump. There’s a tremendous amount wrong with the sci-fi story. Not least of which is that it completely violates the same constraint you put into your own post about constraining computation. I suggest doing your own analysis of how many thought-seconds the AI would have in-between frames of video, especially if you assume it to be running as a large inference model.
The best thing you can do is rid yourself of the notion that superhuman AI would have arbitrary capabilities. That is where EY went wrong, and a lot of the LW crowd too. If you permit dividing by zero or multiplying by infinity, then you can easily convince yourself of anything. AI isn’t magic, and AGI isn’t a free pass to believe anything.
PS: We’ve had AGI since 2017. That’d better be compatible with your world view if you want accurate predictions.
I don’t understand where your confidence is coming from here, but fair enough. It wasn’t clear to me if your take was more like “wildly, wildly superintelligent AI will be considerably weaker than a team of humans thinking for a billion years” or more like “literally impossible without either experiments or >>10^30 FLOP”.
I generally have an intuition like “it’s really, really hard to rule out physically possible things out without very strong evidence, by default things have a reasonable chance of being possible (e.g. 50%) when sufficient intelligence is applied if they are physically possible”. It seems you don’t share this intuition, fair enough.
(I feel like this applies for nearly all human inventions? Like if you had access to a huge amount of video of the world from 1900 and all written books that existed at this point, and had the affordances I described with a team of 10,000 people, 10^30 FLOP, and a billion years, it seems to me like there is a good chance you’d be able to one-shot reinvent ~all inventions of modern humanity (not doing everything in the same way, in many cases you’d massively over engineer to handle one-shot). Planes seem pretty easy? Rockets seem doable?)
I think this is an ok, but not amazing intuition pump for what wildly, wildly superintelligent AI could be like.
I separately think it’s not very important to think about the abilities of wildly, wildly superintelligent AI for most purposes (as I noted in my comment). So I agree that imagining arbitrary capabilities is probablematic. (For some evidence that this isn’t post-hoc justification, see this post on which I’m an author.)
Uhhhh, I’m not sure I agree with this as it doesn’t seem like nearly all jobs are easily fully automatable by AI. Perhaps you use a definition of AGI which is much weaker like “able to speak slightly coherant english (GPT-1?) and classify images”?
By the way, where’s this number coming from? You keep repeating it. That amount of calculation is equivalent to running the largest supercomputer in existence for 30k years. You hypothetical AI scheming breakout is not going to have access to that much compute. Be reasonable.
Ok let’s try a different tract. You want to come up with a molecular mechanics model that can efficiently predict the outcome of reactions, so that you can get about designing one-shot nanotechnology bootstrapping. What would success look like?
You can’t actually do a full simulation to get ground truth for training a better molecular mechanics model. So how would you know the model you came up will work as intended? You can back-test against published results in the literature, but surprise surprise, a big chunk of scientific papers don’t replicate. Shoddy lab technique, publication pressure, and a niche domain combine to create conditions where papers are rushed and sometimes not 100% truthful. Even without deliberate fraud (which also happens), you run into problems such as synthesis steps not working as advertised, images used from different experimental runs than the one described in the paper, etc.
Except you don’t know that. You’re not allowed to do experiments! Maybe you guess that replication will be an issue, although why that would be a hypothesis in the first place without first seeing failures in the lab first isn’t clear to me. But let’s say you do. Which experiments should you discount? Which should you assume to be correct? If you’re allowed to start choosing which reported results you believe and which you don’t, you’ve lost the plot. There could be millions of possible heuristics which partially match the literature and there’s no way to tell the difference.
So what would success look like? How would you know you have the right molecular mechanics model that gives accurate predictions?
You can’t. Not any more than Descartes could think his way to absolute truth.
Also for what it’s worth you’ve made a couple of logical errors here. You are considering human inventions which already exist, then saying that you could one-shot invent them in 1900. That’s hindsight bias, but also selection bias. Nanotechnology doesn’t exist. Even if it would work if created, there’s no existence proof that there exists an accessible path to achieving it. Like superheavy atoms in the island of stability, or micro black holes, there just might not be a pathway to make them from present day capabilities. (Obviously I don’t believe this as I’m running a company attempting to bootstrap Drexlarian nanotechnology, but I feel it’s essential to point out the logical error.)
Why? You’ve gone into circular logic here.
I pointed out that the Alien Message story makes fundamental errors with respect to computational capability being wildly out of scale, so actual super intelligent AIs aren’t going to be anything like the one in the story.
Maybe a Jupiter-sized matryoshka brain made of computronium would exhibit this level of super intelligence. I’m not saying it’s not physically possible. But in terms of sketching out and bounding the capabilities of near-term AI/ASI, it’s a fucking terrible intuition pump.
The transformer architecture introduced in 2017 is:
Artificial: man-made
General: able to train over arbitrary unstructured input, from which it infers models that can be applied in arbitrarily ways to find solutions of problems drawn from domains outside of its training data.
Intelligent: able to construct efficient solutions to new problems it hasn’t seen.
Artificial General Intelligence. A.G.I.
If you’re thinking “yeah, but..” then I suggest you taboo the term AGI. This is literally all that it the word means.
If you want to quibble over dates then maybe we can agree on 2022 with the introduction of ChatGPT, a truly universal (AKA general) interface to mature transformer technology. Either way we’re already well within the era of artificial general intelligence.
(Maybe EY’s very public meltdown a year ago is making more sense now? But rest easy, EY’s predictions about AI x-risk have consistently been wildly off the mark.)
FWIW, I do taboo this term and thus didn’t use it in this conversation until you introduced it.
You highlighted “disagree” on the part about AGI’s definition. I don’t know how to respond to that directly, so I’ll do so here. Here’s the story about how the term “AGI” was coined, by the guy who published the literal book on AGI and ran the AGI conference series for the past two decades:
https://web.archive.org/web/20181228083048/http://goertzel.org/who-coined-the-term-agi/
LW seems to have adopted some other vague, ill-defined, threatening meaning for the acronym “AGI” that is never specified. My assumption is that when people say AGI here they mean Bostrom’s ASI, and they got linked because Eliezer believed (and believes still?) that AGI will FOOM into ASI almost immediately, which it has not.
Anyway it irks me that the term has been coopted here. AGI is a term of art in the pre-ML era of AI research with a clearly defined meaning.
Definition in the OpenAI Charter:
A post on the topic by Richard (AGI = beats most human experts).
In case this wasn’t clear from early discussion, I disagree with Eliezer on a number of topics, including takeoff speeds. In particular I disagree about the time from AI that is economically transformative to AI that is much, much more powerful.
I think you’ll probably find it healthier and more productive to not think of LW as an amorphous collective and instead note that there are a variety of different people who post on the forum with a variety of different views. (I sometimes have made this mistake in the past and I find it healthy to clarify at least internally.)
E.g. instead of saying “LW has bad views about X” say “a high fraction of people who comment on LW have bad views about X” or “a high fraction of karma votes seem to be from people with bad views about X”. Then, you should maybe double check the extent to which a given claim is actualy right : ). For instance, I don’t think almost immediate FOOM is the typical view on LW when aggregating by most metrics, a somewhat longer duration takeoff is now a more common view I think.
Also, I’m going to peace out of this discussion FYI.
Extremely rough and slightly conservatively small ball park number for how many FLOP will be used to create powerful AIs. The idea being that this will represent roughly how many FLOP could plausibly be available at the time.
GPT-4 is ~10^26 FLOP, I expect GPT-7 is maybe 10^30 FLOP.
Perhaps this is a bit too much because the scheming AI will have access to far few FLOP than exist at the time, but I expect this isn’t cruxy, so I just did a vague guess.
I wasn’t trying to justify anything, just noting my stance.