Why are we worried about ASI if current techniques will not lead to intelligence explosion?
There’s often a bait and switch in these communities, where I ask this and people say “even if takeoff is slow, there is still these other problems …” and then list a bunch of small problems, not too different from other tech, which can be dealt with in normal ways.
“This post rules out the intelligence explosion but still talks about x-risk; what the hell?” (which is false; the post specifically rules IN the intelligence explosion)
OR
“This post refers to the intelligence explosion, which I find improbable or impossible, and I want to hear a version of the argument that doesn’t appeal to the intelligence explosion, while still focusing on x-risk. I don’t think you can make such an argument, because longer timelines means the world is safe by default, because we’ll figure it out as we go.”
Which do you mean (or is it some third thing?), and what kind of engagement are you looking for?
It is clear that language models are not “recursively self improving” in any fast sense. They improve with more data in a pretty predictable way in S curves that top out at a pretty disappointing peak. They are useful to do AI research in a limited capacity, some of which hits back at the growth rate (like better training design) but the loops are at long human time-scales. I am not sure it’s even fast enough to give us an industrial revolution.
I have an intuition that most naiive ways of quickly tightening the loop just causes the machine to break and not be very powerful at all.
So okay we have this promising technology that do IMO math, write rap lyrics, moralize, assert consciousness, and make people fall in love with it—but it can’t run a McDonald’s franchise or fly drones into tanks on the battlefield (yet?) Is “general intelligence” a good model for this technology? It is very spiky “intelligence”. It does not rush past all human capability. It has approached human capability gradually and in an uneven way. It is good at the soft feelsy stuff and bad at a lot of the hard power stuff. I think this is the best possible combination of alignment vs power/agency that we could have hoped for back in 2015 to 2019. But people here are still freaking like gpt-2 just came out.
A crux for me is, will language models win over a different paradigm? I do think it is “winning” right now, being more general and actually economically useful kinda. So it would have to be a new exotic paradigm.
Another crux for me is, how good at is it at new science? Not just helping AI researchers with their emails. How good will it be at improving rate of AI research, as well as finding new drugs, better weapons, and other crazy new secrets (at least) like the discovery of atomic power? I think it is not good at this and will not be that good at this. It is best when there is a lot of high quality data and already fast iteration times (programming) but suffers in most fields of science, especially new science, where that is not the case. I relent that if language models will get to the superweapons then it makes sense to treat this like an issue of national/global security.
Intuitively I am more worried about the language models accelerating memetic technology. New religion/spirituality/movements, psychological operations, propaganda. This seems clearly where they are most powerful. I can see a future where we fight culture wars forever, but also one where we genuinely raise humanity to a better state of being as all information technologies have done before (ha). This is not something that hits back at the AI intelligence growth rate very much.
Besides tending the culture, I also think a promising direction for “alignment” (though maybe you want to call it a different name, being a different field) is paying attention to the relationships between individual humans and AI and the pattern of care and interdependence that arises. The closest analogue is raising children and managing other close human relationships.
Why are we worried about ASI if current techniques will not lead to intelligence explosion?
There’s often a bait and switch in these communities, where I ask this and people say “even if takeoff is slow, there is still these other problems …” and then list a bunch of small problems, not too different from other tech, which can be dealt with in normal ways.
Hey Sinclair.
I’m not sure if you mean to say:
“This post rules out the intelligence explosion but still talks about x-risk; what the hell?” (which is false; the post specifically rules IN the intelligence explosion)
OR
“This post refers to the intelligence explosion, which I find improbable or impossible, and I want to hear a version of the argument that doesn’t appeal to the intelligence explosion, while still focusing on x-risk. I don’t think you can make such an argument, because longer timelines means the world is safe by default, because we’ll figure it out as we go.”
Which do you mean (or is it some third thing?), and what kind of engagement are you looking for?
More the latter.
It is clear that language models are not “recursively self improving” in any fast sense. They improve with more data in a pretty predictable way in S curves that top out at a pretty disappointing peak. They are useful to do AI research in a limited capacity, some of which hits back at the growth rate (like better training design) but the loops are at long human time-scales. I am not sure it’s even fast enough to give us an industrial revolution.
I have an intuition that most naiive ways of quickly tightening the loop just causes the machine to break and not be very powerful at all.
So okay we have this promising technology that do IMO math, write rap lyrics, moralize, assert consciousness, and make people fall in love with it—but it can’t run a McDonald’s franchise or fly drones into tanks on the battlefield (yet?)
Is “general intelligence” a good model for this technology? It is very spiky “intelligence”. It does not rush past all human capability. It has approached human capability gradually and in an uneven way.
It is good at the soft feelsy stuff and bad at a lot of the hard power stuff. I think this is the best possible combination of alignment vs power/agency that we could have hoped for back in 2015 to 2019. But people here are still freaking like gpt-2 just came out.
A crux for me is, will language models win over a different paradigm? I do think it is “winning” right now, being more general and actually economically useful kinda. So it would have to be a new exotic paradigm.
Another crux for me is, how good at is it at new science? Not just helping AI researchers with their emails. How good will it be at improving rate of AI research, as well as finding new drugs, better weapons, and other crazy new secrets (at least) like the discovery of atomic power?
I think it is not good at this and will not be that good at this. It is best when there is a lot of high quality data and already fast iteration times (programming) but suffers in most fields of science, especially new science, where that is not the case.
I relent that if language models will get to the superweapons then it makes sense to treat this like an issue of national/global security.
Intuitively I am more worried about the language models accelerating memetic technology. New religion/spirituality/movements, psychological operations, propaganda. This seems clearly where they are most powerful. I can see a future where we fight culture wars forever, but also one where we genuinely raise humanity to a better state of being as all information technologies have done before (ha).
This is not something that hits back at the AI intelligence growth rate very much.
Besides tending the culture, I also think a promising direction for “alignment” (though maybe you want to call it a different name, being a different field) is paying attention to the relationships between individual humans and AI and the pattern of care and interdependence that arises. The closest analogue is raising children and managing other close human relationships.