This probability is the probability of non AI apocalypse. (Large asteroid impacts, nuclear war, alien invasion, vacuum collapse, etc). Basically assuming nothing stopping humans from continuing to improve AI, the chance of “not much impact” is precisely 0. It’s 0 because either it already had an impact or will in the very near future with just slight and obvious improvements to the AI systems that already exist. What sort of future history would have “no significant impact” and HOW? This is like asking after the first Trinity fission weapon test what the probability by 2022 there would be “no significant impact” from nuclear weapons. It’s 0 - already the atmosphere of the earth was contaminated we just didn’t know it.
This is very possible. Complex deception and unstoppable plans to conquer the planet and so on require specific setups for the agent, like “long term reward”. Actual models have myopia inherently, due to how they are trained and limitations on their computational resources. This means a “paperclip production agent” is probably more likely to spend all it’s compute optimizing for small variables like air temperature differences and other parameters to accelerate the robots producing paperclips than to invest in a multi year plan to take over the planet that will let it tile the solar system in paperclip plants after it wins a world war.
I think it isn’t productive to say “let’s not talk of how we would improve capabilities.”. Modeling how future systems are likely to actually work helps to model how you might restrict their behavior effectively.
What sort of future history would have “no significant impact” and HOW? This is like asking after the first Trinity fission weapon test what the probability by 2022 there would be “no significant impact” from nuclear weapons. It’s 0 - already the atmosphere of the earth was contaminated we just didn’t know it.
Zero is not a probability. What if Japan had surrendered before the weapons could be deployed, and the Manhattan project had never been completed? I could totally believe in a one in one hundred thousand probability that nuclear weapons just never saw proliferation, maybe more.
What if Japan had surrendered before the weapons could be deployed, and the Manhattan project had never been completed? I could totally believe in a one in one hundred thousand probability that nuclear weapons just never saw proliferation, maybe more
Specifically, I am referring to after the Trinity test. They had assembled a device and released about 25 kilotons. I am claiming for AI, the “Trinity test” already happened—llms, game playing RL agents that beat humans, and all the other 2022 AI results show that larger yields are possible. Trinity had “no direct significance” on the world in that it didn’t blow up a city, and the weapon wasn’t deployable on a missile, but it showed both were readily achievable (“make a reactor and produce plutonium or separate out u235”) and fission yield was real.
In our world, we don’t have AI better than humans at everything, and they aren’t yet affecting real world products much, but I would argue that the RL agent results show that the equivalent to “fission yield”, “superintelligence”, is possible and also readily achievable. (big neural network, big compute, big training data set = superintelligence)
After the Trinity test, if Japan had surrendered, the information had already leaked, and motivated agents—the all the world powers—would have started power seeking to make nukes. What sort of plausible history has them not doing it? A worldwide agreement that they won’t? What happens when, in a nuke free world, one country attacks another. Won’t the target start rush developing fission weapons? Won’t the other powers learn of this and join in?
It’s unstable. A world where all the powers agree not to build nukes is not a stable one, any pertubation will push it to history states closer to our real timeline.
I would argue that such a world with agreements not to build AGI is similarly not stable.
Right. Because either you believe AI has already made an impact (it has, see recsys for production use of AI that matters) or it will imminently.
The true probability when metaculus resolves as yes isn’t actually zero but the chance you get forecaster credit if you are on the wrong side of the bet IS.
I definitely agree that his confidence in the idea that AI is significant is unjustifiable, but 0 is a probability, it’s just the extreme of improbability into impossibility.
And that’s coming from me, where I do believe that AI being significant has a pretty high probability.
Right. And I am saying it is impossible, except for the classes of scenarios I mentioned, due to the fact that transformative AI is an attractor state.
There are many possible histories, and many possible algorithms humans could try, or current AI recursively self improving could try.
But the optimization arrow is always in the direction of more powerful AI, and this is recursive. Given sufficient compute it’s always the outcome.
It’s kinda like saying “the explosives on a fission bomb have detonated and the nuclear core is to design spec. What is the probability it doesn’t detonate”.
Essentially 0. It’s impossible. I will acknowledge there is actually a possibility that the physics work out where it fails to have any fission gain and stops, but it is probably so small it won’t happen in the lifespan of the observable universe.
Can you explain why it’s “unjustifiable”? What is a plausible future history, even a possible one, free of apocalypse, where humans plus existing AI systems fail to develop transformative systems by 2100.
I think that I don’t have a plausible story, and I think very high 90%+ confidence in significant impact is reasonable.
But the issue I have is roughly that probabilities of literally 100% or a little lower is unjustifiable due to the fact that we must always have some probability for (Our model is totally wrong.)
I do think very high confidence is justifiable, though.
I accept that having some remaining probability mass for “unknown unknowns” is reasonable. And you can certainly talk about ideas that didn’t work out even though they had advantages and existed 60 years ago. Jetpacks, that sort of thing.
But if you do more than a cursory analysis you will see the gain from a jetpack is you save a bit of time at the risk of your life, absurd fuel consumption, high cost, and deafening noise to your neighbors. Gain isn’t worth it.
The potential gain of better AI unlocks most of the resources of the solar system (via automated machinery that can manufacture more automated machinery) and makes world conquest feasible. It’s literally a “get the technology or lose” situation. All it takes is a belief that another power is close to having AI able to operate self replicating machinery and you either invest in the same tech or lose your entire country. Sort of how right now Google believes either they release a counter to BingGPT or lose their company.
So yeah I don’t see a justification for even 10 percent doubt.
0 is a perfectly valid probability estimate. Obviously the chance that an event observed to have not in fact happens is..ok fair. Maybe not zero. You could be mistaken about ground truth reality having happened.
So for instance if AIs take over and wipe everyone’s memories and put them in a simulation, the observed probability in 2100 is that AI didn’t do anything.
If we have short term myopic misaligned AI is still misaligned. It looks like social media algorithms promoting clickbait, like self driving cars turning themselves off half a second before an inevitable crash. Like chatbot recommendation systems telling you what you want to hear, never mind if it’s true.
This is a world where AI is widely used, and is full of non-world-destroying bugs. Self driving cars have been patched and twiddled until they usually work. But on Tuesdays when the moon is waning, they will tend to move to change lanes the rightmost lane, and no one knows why.
This is a suboptimal world that is also one where humans could survive. You’re describing dangers that are usually not worse than what we already live (and die) with. (the SDCs can easily be measurably a lot better than human drivers and human pilots, even if they do have odd tendencies)
Note the “SDC turning itself off half a second before a crash” is actually Working as Intended. If a crash is detected as inevitable—because the high level policy failed to work—you want a microcontroller running plain C code to order maximum braking force. Current systems more or less do work this way.
I think that it is wrong. If instead of dropping nukes on mostly wooden cities they used them against enemy troops (or ships, or even cities which aren’t built out of bamboo), the conclusion would be that a nuke is a not that powerful and cost-inefficient weapon.
As for “significant impact”—what impact counts as “significant”? Here are some technologies which on my opinion had no significant impact so far:
human spaceflight
superconductors
genetic engineering
It is totally possible that AI goes into the same bag.
superconductors—there is a large amount of scientific and research equipment that relies on superconductors. Simplest is NMR magnets. Chemists would not be as productive, but you can argue they would have used alternative technologies not fully exploited in a universe without superconductivity but all the other exploitable natural phenomena we have. So semi correct.
genetic engineering—were you aware that most of the USA corn crop is GMO? And other deliberate changes? The mRNA vaccines were all based on it?
I suspect you meant to make narrower claims:
(1) power transmission and maglev transport superconductivity
(2) human genetic engineering
I would agree completely with your narrower claims.
And would then ask for you to examine the “business case” in the era these things were first discovered. Explain how :
Human spaceflight
superconducting power transmission/maglev transport
3. human genetic engineering
Would ever, even at the heyday after discovery, provide ROI. Think of ROI in terms of real resources and labor instead of money if it helps. Assume the government is willing to loan limitless money at low interest if it helps, just it does have to produce ROI.
Finally, look at the business case for AI.
These are not the same classes of technology. Human spaceflight has zero ROI. Superconductors for the 2 purposes increase efficiency, but often only on the order of 10-20%, so unless the cost of energy saved > cost of equipment it has zero ROI. And human genetic engineering takes too long to give any ROI, even with low interest rates, you have to wait basically for the edited humans to become adults and be productive, and you pay a human cost for all the failures that has enormous reputational costs to any institution.
AI has explosive, self amplifying ROI that is a real business case for effectively “unfathomably large money/material resources”. This is with very conservative assumptions about what AI can be trusted to do. (aka if you ONLY trusted narrow, sandboxed AIs to perform a well defined task, and shut down if anything is more than slightly outside the training environment)
My claim is different—that there is no defined threshold for significance, but on the spectrum from useless to world-changing some technologies which looked very promising decades ago still lie closer to lower end. So it is possible that in 2053 AI products would be about as important as MRI scanners and GMO crops in 2023.
Ok. But how. GMO crops at their theoretical limit cannot fix carbon any faster than thermodynamics will allow. Given all the parts the edited genes spec for come from nature’s codon space, this is what, 100 percent gain at the limit?
So you might get double the growth rate, probably with tradeoffs that make the crop more fragile and more expensive to grow.
MRI well, it lets you crudely see inside the human body in a different way the x-rays. It lets you watch helplessly as tumors kill someone- it provides no tooling to do anything about it. Presumably with the right dyes and alternate techniques like ct scanning you can learn about the same information.
Please try to phrase how AI, with it’s demonstrated abilities, lumps into the above. Does it not let you build self replicating robots? Why?
If you mean human genetic enhancement like designer babies, then sure. Not much impact because ethical concerns prevent it. However, the advent of tech like CRISPR allows for significant impact like gene therapy, though this is still an emerging field. (Just ask someone with sickle cell disease if they think a cure would be significant.)
Lalartu’s claim was that the technology offered no major benefit so far.
Note that treating a few people with severe genetic disease provides no ROI.
This is because those people are rare (most will have died), and there is simply not the market to support the expensive effort to develop a treatment. This is why gene therapy efforts are limited.
Treating diseases isn’t much of a positive feedback loop but claiming “no ROI” strikes me as extremely callous towards those afflicted. Maybe it doesn’t affect enough people to be sufficiently “significant” in this context but it’s certainly not zero return on investment unless reducing human suffering has no value.
Unfortunately, for our purposes it kinda is. There are 2 issues:
Most people don’t have diseases that can be cured or prevented this way
CRISPR is actually quite limited, and in particular the requirement that it only affects your children basically makes it a dealbreaker for human genetic engineering, especially if you’re trying to make superpowered people.
Genetic engineering for humans needs to be both seriously better and they need to be able to edit the somatic cells as well as the gametes cells, or it doesn’t matter.
I don’t dispute this and there are publicly funded efforts that at a small scale, do help people where there isn’t ROI. A few people with blindness or paralysis have received brain implants. A few people have received gene therapies. But the overall thread is it significant. Is the technology mainstream, with massive amounts of sales and R&D effort going into improving it? Is it benefitting most living humans? And the answer is no and no. The brain implants and gene therapies are not very good: they are frankly crap, for the reason that there is not enough resources to make them better.
And from a utilitarian perspective this is correct: in a world where you have very finite resources, most of those resources should be spent on activities that give ROI, as in more resources than you started with. This may sound “callous” but having more resources allows more people to benefit overall from a general sense.
This is why AI and AGI is so different : it trivially gives ROI. Just the current llms produce more value per dollar, on the subset of tasks they are able to do, than any educated human, even from the cheapest countries.
This probability is the probability of non AI apocalypse. (Large asteroid impacts, nuclear war, alien invasion, vacuum collapse, etc). Basically assuming nothing stopping humans from continuing to improve AI, the chance of “not much impact” is precisely 0. It’s 0 because either it already had an impact or will in the very near future with just slight and obvious improvements to the AI systems that already exist. What sort of future history would have “no significant impact” and HOW? This is like asking after the first Trinity fission weapon test what the probability by 2022 there would be “no significant impact” from nuclear weapons. It’s 0 - already the atmosphere of the earth was contaminated we just didn’t know it.
This is very possible. Complex deception and unstoppable plans to conquer the planet and so on require specific setups for the agent, like “long term reward”. Actual models have myopia inherently, due to how they are trained and limitations on their computational resources. This means a “paperclip production agent” is probably more likely to spend all it’s compute optimizing for small variables like air temperature differences and other parameters to accelerate the robots producing paperclips than to invest in a multi year plan to take over the planet that will let it tile the solar system in paperclip plants after it wins a world war.
I think it isn’t productive to say “let’s not talk of how we would improve capabilities.”. Modeling how future systems are likely to actually work helps to model how you might restrict their behavior effectively.
Zero is not a probability. What if Japan had surrendered before the weapons could be deployed, and the Manhattan project had never been completed? I could totally believe in a one in one hundred thousand probability that nuclear weapons just never saw proliferation, maybe more.
What if Japan had surrendered before the weapons could be deployed, and the Manhattan project had never been completed? I could totally believe in a one in one hundred thousand probability that nuclear weapons just never saw proliferation, maybe more
Specifically, I am referring to after the Trinity test. They had assembled a device and released about 25 kilotons. I am claiming for AI, the “Trinity test” already happened—llms, game playing RL agents that beat humans, and all the other 2022 AI results show that larger yields are possible. Trinity had “no direct significance” on the world in that it didn’t blow up a city, and the weapon wasn’t deployable on a missile, but it showed both were readily achievable (“make a reactor and produce plutonium or separate out u235”) and fission yield was real.
In our world, we don’t have AI better than humans at everything, and they aren’t yet affecting real world products much, but I would argue that the RL agent results show that the equivalent to “fission yield”, “superintelligence”, is possible and also readily achievable. (big neural network, big compute, big training data set = superintelligence)
After the Trinity test, if Japan had surrendered, the information had already leaked, and motivated agents—the all the world powers—would have started power seeking to make nukes. What sort of plausible history has them not doing it? A worldwide agreement that they won’t? What happens when, in a nuke free world, one country attacks another. Won’t the target start rush developing fission weapons? Won’t the other powers learn of this and join in?
It’s unstable. A world where all the powers agree not to build nukes is not a stable one, any pertubation will push it to history states closer to our real timeline.
I would argue that such a world with agreements not to build AGI is similarly not stable.
What if you just s/0/epsilon/g ?
I’d imagine Gerald’s “probability 0” is something like Metaculus’s “resolved as yes”—that is, the even in question has already happened.
Right. Because either you believe AI has already made an impact (it has, see recsys for production use of AI that matters) or it will imminently.
The true probability when metaculus resolves as yes isn’t actually zero but the chance you get forecaster credit if you are on the wrong side of the bet IS.
I definitely agree that his confidence in the idea that AI is significant is unjustifiable, but 0 is a probability, it’s just the extreme of improbability into impossibility.
And that’s coming from me, where I do believe that AI being significant has a pretty high probability.
Right. And I am saying it is impossible, except for the classes of scenarios I mentioned, due to the fact that transformative AI is an attractor state.
There are many possible histories, and many possible algorithms humans could try, or current AI recursively self improving could try.
But the optimization arrow is always in the direction of more powerful AI, and this is recursive. Given sufficient compute it’s always the outcome.
It’s kinda like saying “the explosives on a fission bomb have detonated and the nuclear core is to design spec. What is the probability it doesn’t detonate”.
Essentially 0. It’s impossible. I will acknowledge there is actually a possibility that the physics work out where it fails to have any fission gain and stops, but it is probably so small it won’t happen in the lifespan of the observable universe.
Can you explain why it’s “unjustifiable”? What is a plausible future history, even a possible one, free of apocalypse, where humans plus existing AI systems fail to develop transformative systems by 2100.
I think that I don’t have a plausible story, and I think very high 90%+ confidence in significant impact is reasonable.
But the issue I have is roughly that probabilities of literally 100% or a little lower is unjustifiable due to the fact that we must always have some probability for (Our model is totally wrong.)
I do think very high confidence is justifiable, though.
I accept that having some remaining probability mass for “unknown unknowns” is reasonable. And you can certainly talk about ideas that didn’t work out even though they had advantages and existed 60 years ago. Jetpacks, that sort of thing.
But if you do more than a cursory analysis you will see the gain from a jetpack is you save a bit of time at the risk of your life, absurd fuel consumption, high cost, and deafening noise to your neighbors. Gain isn’t worth it.
The potential gain of better AI unlocks most of the resources of the solar system (via automated machinery that can manufacture more automated machinery) and makes world conquest feasible. It’s literally a “get the technology or lose” situation. All it takes is a belief that another power is close to having AI able to operate self replicating machinery and you either invest in the same tech or lose your entire country. Sort of how right now Google believes either they release a counter to BingGPT or lose their company.
So yeah I don’t see a justification for even 10 percent doubt.
0 is a perfectly valid probability estimate. Obviously the chance that an event observed to have not in fact happens is..ok fair. Maybe not zero. You could be mistaken about ground truth reality having happened.
So for instance if AIs take over and wipe everyone’s memories and put them in a simulation, the observed probability in 2100 is that AI didn’t do anything.
If we have short term myopic misaligned AI is still misaligned. It looks like social media algorithms promoting clickbait, like self driving cars turning themselves off half a second before an inevitable crash. Like chatbot recommendation systems telling you what you want to hear, never mind if it’s true.
This is a world where AI is widely used, and is full of non-world-destroying bugs. Self driving cars have been patched and twiddled until they usually work. But on Tuesdays when the moon is waning, they will tend to move to change lanes the rightmost lane, and no one knows why.
This is a suboptimal world that is also one where humans could survive. You’re describing dangers that are usually not worse than what we already live (and die) with. (the SDCs can easily be measurably a lot better than human drivers and human pilots, even if they do have odd tendencies)
Note the “SDC turning itself off half a second before a crash” is actually Working as Intended. If a crash is detected as inevitable—because the high level policy failed to work—you want a microcontroller running plain C code to order maximum braking force. Current systems more or less do work this way.
I wasn’t claiming this was a particularly bad world. I was just disagreeing with the idea that myopic AI=Aligned AI.
The turning itself off thing, I was thinking of the tetris bot that learned to pause the game.
I think that it is wrong. If instead of dropping nukes on mostly wooden cities they used them against enemy troops (or ships, or even cities which aren’t built out of bamboo), the conclusion would be that a nuke is a not that powerful and cost-inefficient weapon.
As for “significant impact”—what impact counts as “significant”? Here are some technologies which on my opinion had no significant impact so far:
human spaceflight
superconductors
genetic engineering
It is totally possible that AI goes into the same bag.
human spaceflight—you’re correct
superconductors—there is a large amount of scientific and research equipment that relies on superconductors. Simplest is NMR magnets. Chemists would not be as productive, but you can argue they would have used alternative technologies not fully exploited in a universe without superconductivity but all the other exploitable natural phenomena we have. So semi correct.
genetic engineering—were you aware that most of the USA corn crop is GMO? And other deliberate changes? The mRNA vaccines were all based on it?
I suspect you meant to make narrower claims:
(1) power transmission and maglev transport superconductivity
(2) human genetic engineering
I would agree completely with your narrower claims.
And would then ask for you to examine the “business case” in the era these things were first discovered. Explain how :
Human spaceflight
superconducting power transmission/maglev transport
3. human genetic engineering
Would ever, even at the heyday after discovery, provide ROI. Think of ROI in terms of real resources and labor instead of money if it helps. Assume the government is willing to loan limitless money at low interest if it helps, just it does have to produce ROI.
Finally, look at the business case for AI.
These are not the same classes of technology. Human spaceflight has zero ROI. Superconductors for the 2 purposes increase efficiency, but often only on the order of 10-20%, so unless the cost of energy saved > cost of equipment it has zero ROI. And human genetic engineering takes too long to give any ROI, even with low interest rates, you have to wait basically for the edited humans to become adults and be productive, and you pay a human cost for all the failures that has enormous reputational costs to any institution.
AI has explosive, self amplifying ROI that is a real business case for effectively “unfathomably large money/material resources”. This is with very conservative assumptions about what AI can be trusted to do. (aka if you ONLY trusted narrow, sandboxed AIs to perform a well defined task, and shut down if anything is more than slightly outside the training environment)
My claim is different—that there is no defined threshold for significance, but on the spectrum from useless to world-changing some technologies which looked very promising decades ago still lie closer to lower end. So it is possible that in 2053 AI products would be about as important as MRI scanners and GMO crops in 2023.
Ok. But how. GMO crops at their theoretical limit cannot fix carbon any faster than thermodynamics will allow. Given all the parts the edited genes spec for come from nature’s codon space, this is what, 100 percent gain at the limit?
So you might get double the growth rate, probably with tradeoffs that make the crop more fragile and more expensive to grow.
MRI well, it lets you crudely see inside the human body in a different way the x-rays. It lets you watch helplessly as tumors kill someone- it provides no tooling to do anything about it. Presumably with the right dyes and alternate techniques like ct scanning you can learn about the same information.
Please try to phrase how AI, with it’s demonstrated abilities, lumps into the above. Does it not let you build self replicating robots? Why?
“human genetic engineering”
If you mean human genetic enhancement like designer babies, then sure. Not much impact because ethical concerns prevent it. However, the advent of tech like CRISPR allows for significant impact like gene therapy, though this is still an emerging field. (Just ask someone with sickle cell disease if they think a cure would be significant.)
Lalartu’s claim was that the technology offered no major benefit so far.
Note that treating a few people with severe genetic disease provides no ROI.
This is because those people are rare (most will have died), and there is simply not the market to support the expensive effort to develop a treatment. This is why gene therapy efforts are limited.
Treating diseases isn’t much of a positive feedback loop but claiming “no ROI” strikes me as extremely callous towards those afflicted. Maybe it doesn’t affect enough people to be sufficiently “significant” in this context but it’s certainly not zero return on investment unless reducing human suffering has no value.
Unfortunately, for our purposes it kinda is. There are 2 issues:
Most people don’t have diseases that can be cured or prevented this way
CRISPR is actually quite limited, and in particular the requirement that it only affects your children basically makes it a dealbreaker for human genetic engineering, especially if you’re trying to make superpowered people.
Genetic engineering for humans needs to be both seriously better and they need to be able to edit the somatic cells as well as the gametes cells, or it doesn’t matter.
I don’t dispute this and there are publicly funded efforts that at a small scale, do help people where there isn’t ROI. A few people with blindness or paralysis have received brain implants. A few people have received gene therapies. But the overall thread is it significant. Is the technology mainstream, with massive amounts of sales and R&D effort going into improving it? Is it benefitting most living humans? And the answer is no and no. The brain implants and gene therapies are not very good: they are frankly crap, for the reason that there is not enough resources to make them better.
And from a utilitarian perspective this is correct: in a world where you have very finite resources, most of those resources should be spent on activities that give ROI, as in more resources than you started with. This may sound “callous” but having more resources allows more people to benefit overall from a general sense.
This is why AI and AGI is so different : it trivially gives ROI. Just the current llms produce more value per dollar, on the subset of tasks they are able to do, than any educated human, even from the cheapest countries.