By the way, where’s this number coming from? You keep repeating it. That amount of calculation is equivalent to running the largest supercomputer in existence for 30k years. You hypothetical AI scheming breakout is not going to have access to that much compute. Be reasonable.
I generally have an intuition like “it’s really, really hard to rule out physically possible things out without very strong evidence, by default things have a reasonable chance of being possible (e.g. 50%) when sufficient intelligence is applied if they are physically possible”
Ok let’s try a different tract. You want to come up with a molecular mechanics model that can efficiently predict the outcome of reactions, so that you can get about designing one-shot nanotechnology bootstrapping. What would success look like?
You can’t actually do a full simulation to get ground truth for training a better molecular mechanics model. So how would you know the model you came up will work as intended? You can back-test against published results in the literature, but surprise surprise, a big chunk of scientific papers don’t replicate. Shoddy lab technique, publication pressure, and a niche domain combine to create conditions where papers are rushed and sometimes not 100% truthful. Even without deliberate fraud (which also happens), you run into problems such as synthesis steps not working as advertised, images used from different experimental runs than the one described in the paper, etc.
Except you don’t know that. You’re not allowed to do experiments! Maybe you guess that replication will be an issue, although why that would be a hypothesis in the first place without first seeing failures in the lab first isn’t clear to me. But let’s say you do. Which experiments should you discount? Which should you assume to be correct? If you’re allowed to start choosing which reported results you believe and which you don’t, you’ve lost the plot. There could be millions of possible heuristics which partially match the literature and there’s no way to tell the difference.
So what would success look like? How would you know you have the right molecular mechanics model that gives accurate predictions?
You can’t. Not any more than Descartes could think his way to absolute truth.
Also for what it’s worth you’ve made a couple of logical errors here. You are considering human inventions which already exist, then saying that you could one-shot invent them in 1900. That’s hindsight bias, but also selection bias. Nanotechnology doesn’t exist. Even if it would work if created, there’s no existence proof that there exists an accessible path to achieving it. Like superheavy atoms in the island of stability, or micro black holes, there just might not be a pathway to make them from present day capabilities. (Obviously I don’t believe this as I’m running a company attempting to bootstrap Drexlarian nanotechnology, but I feel it’s essential to point out the logical error.)
(Re: Alien Message) I think this is an ok, but not amazing intuition pump for what wildly, wildly superintelligent AI could be like.
Why? You’ve gone into circular logic here.
I pointed out that the Alien Message story makes fundamental errors with respect to computational capability being wildly out of scale, so actual super intelligent AIs aren’t going to be anything like the one in the story.
Maybe a Jupiter-sized matryoshka brain made of computronium would exhibit this level of super intelligence. I’m not saying it’s not physically possible. But in terms of sketching out and bounding the capabilities of near-term AI/ASI, it’s a fucking terrible intuition pump.
Uhhhh, I’m not sure I agree with this as it doesn’t seem like nearly all jobs are easily fully automatable by AI. Perhaps you use a definition of AGI which is much weaker like “able to speak slightly coherant english (GPT-1?) and classify images”?
The transformer architecture introduced in 2017 is:
Artificial: man-made
General: able to train over arbitrary unstructured input, from which it infers models that can be applied in arbitrarily ways to find solutions of problems drawn from domains outside of its training data.
Intelligent: able to construct efficient solutions to new problems it hasn’t seen.
Artificial General Intelligence. A.G.I.
If you’re thinking “yeah, but..” then I suggest you taboo the term AGI. This is literally all that it the word means.
If you want to quibble over dates then maybe we can agree on 2022 with the introduction of ChatGPT, a truly universal (AKA general) interface to mature transformer technology. Either way we’re already well within the era of artificial general intelligence.
(Maybe EY’s very public meltdown a year ago is making more sense now? But rest easy, EY’s predictions about AI x-risk have consistently been wildly off the mark.)
You highlighted “disagree” on the part about AGI’s definition. I don’t know how to respond to that directly, so I’ll do so here. Here’s the story about how the term “AGI” was coined, by the guy who published the literal book on AGI and ran the AGI conference series for the past two decades:
LW seems to have adopted some other vague, ill-defined, threatening meaning for the acronym “AGI” that is never specified. My assumption is that when people say AGI here they mean Bostrom’s ASI, and they got linked because Eliezer believed (and believes still?) that AGI will FOOM into ASI almost immediately, which it has not.
Anyway it irks me that the term has been coopted here. AGI is a term of art in the pre-ML era of AI research with a clearly defined meaning.
My assumption is that when people say AGI here they mean Bostrom’s ASI, and they got linked because Eliezer believed (and believes still?) that AGI will FOOM into ASI almost immediately, which it has not.
In case this wasn’t clear from early discussion, I disagree with Eliezer on a number of topics, including takeoff speeds. In particular I disagree about the time from AI that is economically transformative to AI that is much, much more powerful.
I think you’ll probably find it healthier and more productive to not think of LW as an amorphous collective and instead note that there are a variety of different people who post on the forum with a variety of different views. (I sometimes have made this mistake in the past and I find it healthy to clarify at least internally.)
E.g. instead of saying “LW has bad views about X” say “a high fraction of people who comment on LW have bad views about X” or “a high fraction of karma votes seem to be from people with bad views about X”. Then, you should maybe double check the extent to which a given claim is actualy right : ). For instance, I don’t think almost immediate FOOM is the typical view on LW when aggregating by most metrics, a somewhat longer duration takeoff is now a more common view I think.
By the way, where’s this number coming from? (10^30 FLOP) You keep repeating it.
Extremely rough and slightly conservatively small ball park number for how many FLOP will be used to create powerful AIs. The idea being that this will represent roughly how many FLOP could plausibly be available at the time.
GPT-4 is ~10^26 FLOP, I expect GPT-7 is maybe 10^30 FLOP.
Perhaps this is a bit too much because the scheming AI will have access to far few FLOP than exist at the time, but I expect this isn’t cruxy, so I just did a vague guess.
By the way, where’s this number coming from? You keep repeating it. That amount of calculation is equivalent to running the largest supercomputer in existence for 30k years. You hypothetical AI scheming breakout is not going to have access to that much compute. Be reasonable.
Ok let’s try a different tract. You want to come up with a molecular mechanics model that can efficiently predict the outcome of reactions, so that you can get about designing one-shot nanotechnology bootstrapping. What would success look like?
You can’t actually do a full simulation to get ground truth for training a better molecular mechanics model. So how would you know the model you came up will work as intended? You can back-test against published results in the literature, but surprise surprise, a big chunk of scientific papers don’t replicate. Shoddy lab technique, publication pressure, and a niche domain combine to create conditions where papers are rushed and sometimes not 100% truthful. Even without deliberate fraud (which also happens), you run into problems such as synthesis steps not working as advertised, images used from different experimental runs than the one described in the paper, etc.
Except you don’t know that. You’re not allowed to do experiments! Maybe you guess that replication will be an issue, although why that would be a hypothesis in the first place without first seeing failures in the lab first isn’t clear to me. But let’s say you do. Which experiments should you discount? Which should you assume to be correct? If you’re allowed to start choosing which reported results you believe and which you don’t, you’ve lost the plot. There could be millions of possible heuristics which partially match the literature and there’s no way to tell the difference.
So what would success look like? How would you know you have the right molecular mechanics model that gives accurate predictions?
You can’t. Not any more than Descartes could think his way to absolute truth.
Also for what it’s worth you’ve made a couple of logical errors here. You are considering human inventions which already exist, then saying that you could one-shot invent them in 1900. That’s hindsight bias, but also selection bias. Nanotechnology doesn’t exist. Even if it would work if created, there’s no existence proof that there exists an accessible path to achieving it. Like superheavy atoms in the island of stability, or micro black holes, there just might not be a pathway to make them from present day capabilities. (Obviously I don’t believe this as I’m running a company attempting to bootstrap Drexlarian nanotechnology, but I feel it’s essential to point out the logical error.)
Why? You’ve gone into circular logic here.
I pointed out that the Alien Message story makes fundamental errors with respect to computational capability being wildly out of scale, so actual super intelligent AIs aren’t going to be anything like the one in the story.
Maybe a Jupiter-sized matryoshka brain made of computronium would exhibit this level of super intelligence. I’m not saying it’s not physically possible. But in terms of sketching out and bounding the capabilities of near-term AI/ASI, it’s a fucking terrible intuition pump.
The transformer architecture introduced in 2017 is:
Artificial: man-made
General: able to train over arbitrary unstructured input, from which it infers models that can be applied in arbitrarily ways to find solutions of problems drawn from domains outside of its training data.
Intelligent: able to construct efficient solutions to new problems it hasn’t seen.
Artificial General Intelligence. A.G.I.
If you’re thinking “yeah, but..” then I suggest you taboo the term AGI. This is literally all that it the word means.
If you want to quibble over dates then maybe we can agree on 2022 with the introduction of ChatGPT, a truly universal (AKA general) interface to mature transformer technology. Either way we’re already well within the era of artificial general intelligence.
(Maybe EY’s very public meltdown a year ago is making more sense now? But rest easy, EY’s predictions about AI x-risk have consistently been wildly off the mark.)
FWIW, I do taboo this term and thus didn’t use it in this conversation until you introduced it.
You highlighted “disagree” on the part about AGI’s definition. I don’t know how to respond to that directly, so I’ll do so here. Here’s the story about how the term “AGI” was coined, by the guy who published the literal book on AGI and ran the AGI conference series for the past two decades:
https://web.archive.org/web/20181228083048/http://goertzel.org/who-coined-the-term-agi/
LW seems to have adopted some other vague, ill-defined, threatening meaning for the acronym “AGI” that is never specified. My assumption is that when people say AGI here they mean Bostrom’s ASI, and they got linked because Eliezer believed (and believes still?) that AGI will FOOM into ASI almost immediately, which it has not.
Anyway it irks me that the term has been coopted here. AGI is a term of art in the pre-ML era of AI research with a clearly defined meaning.
Definition in the OpenAI Charter:
A post on the topic by Richard (AGI = beats most human experts).
In case this wasn’t clear from early discussion, I disagree with Eliezer on a number of topics, including takeoff speeds. In particular I disagree about the time from AI that is economically transformative to AI that is much, much more powerful.
I think you’ll probably find it healthier and more productive to not think of LW as an amorphous collective and instead note that there are a variety of different people who post on the forum with a variety of different views. (I sometimes have made this mistake in the past and I find it healthy to clarify at least internally.)
E.g. instead of saying “LW has bad views about X” say “a high fraction of people who comment on LW have bad views about X” or “a high fraction of karma votes seem to be from people with bad views about X”. Then, you should maybe double check the extent to which a given claim is actualy right : ). For instance, I don’t think almost immediate FOOM is the typical view on LW when aggregating by most metrics, a somewhat longer duration takeoff is now a more common view I think.
Also, I’m going to peace out of this discussion FYI.
Extremely rough and slightly conservatively small ball park number for how many FLOP will be used to create powerful AIs. The idea being that this will represent roughly how many FLOP could plausibly be available at the time.
GPT-4 is ~10^26 FLOP, I expect GPT-7 is maybe 10^30 FLOP.
Perhaps this is a bit too much because the scheming AI will have access to far few FLOP than exist at the time, but I expect this isn’t cruxy, so I just did a vague guess.
I wasn’t trying to justify anything, just noting my stance.