I think even the wealthy supporters of it are more complex: I was surprised that Palantir’s Peter Thiel came out discussing how AI “must not be allowed to surpass the human spirit” even as he clearly is looking to use AI in military operations. This all suggests significant controls incoming, even from those looking to benefit from it.
I agree with controls. I have an issue with wasted time on bureaucratic review and think it could burn the lead the western countries have.
Basically, “do z y z” to prove your model is good, design it according to “this known good framework” is ok with me.
“We have closed reviews for this year” is not. “We have issued too many AI research licenses this year” is not. “We have denied your application because we made mistakes in our review and will not update on evidence” is not.
All of these occur from a power imbalance. The entity requesting authorization is liable for any errors, but the government makes itself immune from accountability. (For example the government should be on the hook for lost revenue from the future products actual revenue for each day the review is delayed. The government should be required to buy companies at fair market value if it denies them an AI research license. Etc)
You are using the poisoned banana theory and do not believe we can easily build controllable ASI systems by restricting their inputs to in test distribution examples and resetting state often, correct?
I just wanted to establish your cruxes. Because if you could build safe ASI easily would this change your opinion on the correct policy?
No, I wouldn’t want it even if it was possible since by nature it is a replacement of humanity. I’d only accept Elon’s vision of AI bolted onto humans, so it effectively is part of us and thus can be said to be an evolution rather than replacement.
My main crux is that humanity has to be largely biological due to holobiont theory. There’s a lot of flexibility around that but anything that threatens that is a nonstarter.
Ok, that’s reasonable. Do you foresee, in worlds where ASI turns out to be easily controllable, ones where governments set up “grim triggers” like you advocate for or do you think, in worlds conditional on ASI being easily controllable/taskable, that such policies would not be enacted by the superpowers with nuclear weapons?
Obviously, without grim triggers, you end up with the scenario you despise: immortal humans and their ASI tools controlling essentially all power and wealth.
This is I think kind of a flaw in your viewpoint. Over the arrow of time, AI/AGI/ASI adopters and contributors are going to have almost all of the effective votes. Your stated preferences mean over time your faction will lose power and relevance.
For an example of this see autonomous weapons bans. Or a general example is the emh.
Please note I am trying to be neutral here. Your preferences are perfectly respectable and understandable, it’s just that some preferences may have more real world utility than others.
This frames things as an inevitability which is almost certainly wrong, but more specifically opposition to a technology leads to alternatives being developed. E.g. widespread nuclear control led to alternatives being pursued for energy.
Being controllable is unlikely even if it is tractable by human controllers: it still represents power which means it’ll be treated as a threat by established actors and its terroristic implications mean there is moral valence to police it.
In a world with controls, grim triggers or otherwise, AI would have to develop along different lines and likely in ways that are more human compatible. In a world of intense grim triggers, it may be that is too costly to continue to develop beyond a point. “Don’t build ASI or we nuke” is completely reasonable if both “build ASI” and “nuking” is negative, but the former is more negative.
Autonomous weapons actually are an excellent example of delay: despite excellent evidence of the superiority of drones, pilots have continued to mothball it for at least 40 years and so have governments in spite of wartime benefits.
The argument seems to similar to the flaw in the “billion year” argument: we may die eventually, but life only persists by resisting death, long enough for it to replicate.
As far as real world utility, notwithstanding some recent successes, going down without fighting for myself and my children is quite silly.
I think the error here is you may be comparing technologies on different benefit scales than I am.
Nuclear power can be cheaper than paying for fossil fuel to burn in a generator, if the nuclear reactor is cheaply built and has a small operating staff. Your benefit is a small decrease in price per kWh.
As we both know, cheaply built and lightly staffed nuclear plants are a hazard and governments have made them illegal. Safe plants, that are expensively built with lots of staff and time spent on reviewing the plans for approval and redoing faulty work during construction, are more expensive than fossil fuel and now renewables, and are generally not worth building.
Until extremely recently, AI controlled aircraft did not exist. The general public has for decades had a misinterpretation of what “autopilot” systems are capable of. Until a few months ago, none of those systems could actually pilot their aircraft, they solely act as simple controllers to head towards waypoints, etc. (Some can control the main flight controls during a landing but many of the steps must be performed by the pilot)
The benefit of an AI controlled aircraft is you don’t have to pay a pilot.
Drones were not superior until extremely recently. You may be misinformed to the capabilities of systems like the predator 1 and 2 drones, which were not capable of air combat maneuvering and had no software algorithms available in that era capable of it. Also combat aircraft have been firing autonomous missiles at each other since the Korean war.
Note both benefits are linear. You get say n percent cheaper electricity where n is less than 50 percent, or n percent cheaper to operate aircraft, where n is less than 20 percent.
The benefits of AGI is exponential. Eventually the benefits scale to millions, then billions, then trillions of times the physical resources, etc, that you started with.
It’s extremely divergent. Once a faction gets even a doubling or 2 it’s over, nukes won’t stop them.
Assumption: by doubling I mean say a nation with a GDP of 10 trillion gets AGI and now has 20 or 40 trillion GDP. Their territory is covered with billions of new AGI based robotic factories and clinics and so on. Your nuclear bombardment does not destroy enough copies of the equipment to prevent them from recovering.
I’ll look for the article later but basically the Air Force has found pilotless aircraft to be useful for around thirty years but organized rejection has led to most such programs meeting an early death.
The rest is a lot of AGI is magic without considering the actual costs of computation or noncomputable situations. Nukes would just scale up: it costs much less to destroy than it is to build and the significance of modern economics is indeed that they require networks which do not take shocks well. Everything else basically is “ASI is magic.”
We would need some more context on what you are referring to. For loitering over an undefended target and dropping bombs, yes, drones are superior and the us air force has allowed the US army to operate those drones instead. I do not think the us air force has had the belief that operating high end aircraft such as stealth and supersonic fighter bombers was within the capability of drone software over the last 30 years, with things shifting recently. Remember, in 2012 the first modern deep learning experiments were tried, prior to this AI was mostly a curiosity.
If “the bomb” can wipe out a country with automated factories and missile defense systems, why fear AGI/ASI? I see a bit of cognitive dissonance in your latest point similar to Gary Marcus. Gary Marcus has consistently argued that current llms are just a trick, real AGI is very far away, and that near term systems are no threat, yet also argues for AI pauses. This feels like an incoherent view that you are also expressing. Either AGI/ASI is, as you put it, in fact magic and you need to pound the red button early and often, or you can delay committing national suicide until later. I look forward to a clarification of your beliefs.
I don’t think it is magic but it is still sufficiently disgusting to treat it with equal threat now. Red button now.
Its not a good idea to treat a disease right before it kills you: prevention is the way to go.
So no, I don’t think it is magic. But I do think just as the world agreed against human cloning long before there was a human clone, now is the time to act.
So gathering up your beliefs, you believe ASI/AGI to be a threat, but not so dangerous a threat you need to use nuclear weapons until an enemy nation with it is extremely far along, which will take, according to your beliefs, many years since it’s not that good.
But you find the very idea of non human intelligence in use by humans or possibly serving itself so disgusting that you want nuclear weapons used the instant anyone steps out of compliance with international rules you wish to impose. (Note this is historically unprecedented, arms control treaties have been voluntary and did not have immediate thermonuclear war as the penalty for violating them)
And since your beliefs are emotionally based on “disgust”, I assume there is no updating based on actual measurements? That is, if ASI turns out to be safer than you currently think, you still want immediate nukes, and vice versa?
What percentage of the population of world superpower decision makers do you feel share your belief? Just a rough guess is fine.
The point is that sanctions should be applied as necessary to discourage AGI, however, approximate grim triggers should apply as needed to prevent dystopia.
As the other commentators have mentioned, my reaction is not unusual and thus this is why the concerns of doom have been widespread.
I think even the wealthy supporters of it are more complex: I was surprised that Palantir’s Peter Thiel came out discussing how AI “must not be allowed to surpass the human spirit” even as he clearly is looking to use AI in military operations. This all suggests significant controls incoming, even from those looking to benefit from it.
Googling for “must not be allowed to surpass the human spirit” and Palantir finds no hits.
He discussed it here:
https://youtu.be/Ufm85wHJk5A?list=PLQk-vCAGvjtcMI77ChZ-SPP—cx6BWBWm
I agree with controls. I have an issue with wasted time on bureaucratic review and think it could burn the lead the western countries have.
Basically, “do z y z” to prove your model is good, design it according to “this known good framework” is ok with me.
“We have closed reviews for this year” is not. “We have issued too many AI research licenses this year” is not. “We have denied your application because we made mistakes in our review and will not update on evidence” is not.
All of these occur from a power imbalance. The entity requesting authorization is liable for any errors, but the government makes itself immune from accountability. (For example the government should be on the hook for lost revenue from the future products actual revenue for each day the review is delayed. The government should be required to buy companies at fair market value if it denies them an AI research license. Etc)
Lead is irrelevant to human extinction, obviously. The first to die is still dead.
In a democratic world, those affected have a say in how they should be inflicted with AI and how much they want to die or suffer.
The government represents the people.
You are using the poisoned banana theory and do not believe we can easily build controllable ASI systems by restricting their inputs to in test distribution examples and resetting state often, correct?
I just wanted to establish your cruxes. Because if you could build safe ASI easily would this change your opinion on the correct policy?
No, I wouldn’t want it even if it was possible since by nature it is a replacement of humanity. I’d only accept Elon’s vision of AI bolted onto humans, so it effectively is part of us and thus can be said to be an evolution rather than replacement.
My main crux is that humanity has to be largely biological due to holobiont theory. There’s a lot of flexibility around that but anything that threatens that is a nonstarter.
Ok, that’s reasonable. Do you foresee, in worlds where ASI turns out to be easily controllable, ones where governments set up “grim triggers” like you advocate for or do you think, in worlds conditional on ASI being easily controllable/taskable, that such policies would not be enacted by the superpowers with nuclear weapons?
Obviously, without grim triggers, you end up with the scenario you despise: immortal humans and their ASI tools controlling essentially all power and wealth.
This is I think kind of a flaw in your viewpoint. Over the arrow of time, AI/AGI/ASI adopters and contributors are going to have almost all of the effective votes. Your stated preferences mean over time your faction will lose power and relevance.
For an example of this see autonomous weapons bans. Or a general example is the emh.
Please note I am trying to be neutral here. Your preferences are perfectly respectable and understandable, it’s just that some preferences may have more real world utility than others.
This frames things as an inevitability which is almost certainly wrong, but more specifically opposition to a technology leads to alternatives being developed. E.g. widespread nuclear control led to alternatives being pursued for energy.
Being controllable is unlikely even if it is tractable by human controllers: it still represents power which means it’ll be treated as a threat by established actors and its terroristic implications mean there is moral valence to police it.
In a world with controls, grim triggers or otherwise, AI would have to develop along different lines and likely in ways that are more human compatible. In a world of intense grim triggers, it may be that is too costly to continue to develop beyond a point. “Don’t build ASI or we nuke” is completely reasonable if both “build ASI” and “nuking” is negative, but the former is more negative.
Autonomous weapons actually are an excellent example of delay: despite excellent evidence of the superiority of drones, pilots have continued to mothball it for at least 40 years and so have governments in spite of wartime benefits.
The argument seems to similar to the flaw in the “billion year” argument: we may die eventually, but life only persists by resisting death, long enough for it to replicate.
As far as real world utility, notwithstanding some recent successes, going down without fighting for myself and my children is quite silly.
I think the error here is you may be comparing technologies on different benefit scales than I am.
Nuclear power can be cheaper than paying for fossil fuel to burn in a generator, if the nuclear reactor is cheaply built and has a small operating staff. Your benefit is a small decrease in price per kWh.
As we both know, cheaply built and lightly staffed nuclear plants are a hazard and governments have made them illegal. Safe plants, that are expensively built with lots of staff and time spent on reviewing the plans for approval and redoing faulty work during construction, are more expensive than fossil fuel and now renewables, and are generally not worth building.
Until extremely recently, AI controlled aircraft did not exist. The general public has for decades had a misinterpretation of what “autopilot” systems are capable of. Until a few months ago, none of those systems could actually pilot their aircraft, they solely act as simple controllers to head towards waypoints, etc. (Some can control the main flight controls during a landing but many of the steps must be performed by the pilot)
The benefit of an AI controlled aircraft is you don’t have to pay a pilot.
Drones were not superior until extremely recently. You may be misinformed to the capabilities of systems like the predator 1 and 2 drones, which were not capable of air combat maneuvering and had no software algorithms available in that era capable of it. Also combat aircraft have been firing autonomous missiles at each other since the Korean war.
Note both benefits are linear. You get say n percent cheaper electricity where n is less than 50 percent, or n percent cheaper to operate aircraft, where n is less than 20 percent.
The benefits of AGI is exponential. Eventually the benefits scale to millions, then billions, then trillions of times the physical resources, etc, that you started with.
It’s extremely divergent. Once a faction gets even a doubling or 2 it’s over, nukes won’t stop them.
Assumption: by doubling I mean say a nation with a GDP of 10 trillion gets AGI and now has 20 or 40 trillion GDP. Their territory is covered with billions of new AGI based robotic factories and clinics and so on. Your nuclear bombardment does not destroy enough copies of the equipment to prevent them from recovering.
I’ll look for the article later but basically the Air Force has found pilotless aircraft to be useful for around thirty years but organized rejection has led to most such programs meeting an early death.
The rest is a lot of AGI is magic without considering the actual costs of computation or noncomputable situations. Nukes would just scale up: it costs much less to destroy than it is to build and the significance of modern economics is indeed that they require networks which do not take shocks well. Everything else basically is “ASI is magic.”
I would bet on the bomb.
Two points :
We would need some more context on what you are referring to. For loitering over an undefended target and dropping bombs, yes, drones are superior and the us air force has allowed the US army to operate those drones instead. I do not think the us air force has had the belief that operating high end aircraft such as stealth and supersonic fighter bombers was within the capability of drone software over the last 30 years, with things shifting recently. Remember, in 2012 the first modern deep learning experiments were tried, prior to this AI was mostly a curiosity.
If “the bomb” can wipe out a country with automated factories and missile defense systems, why fear AGI/ASI? I see a bit of cognitive dissonance in your latest point similar to Gary Marcus. Gary Marcus has consistently argued that current llms are just a trick, real AGI is very far away, and that near term systems are no threat, yet also argues for AI pauses. This feels like an incoherent view that you are also expressing. Either AGI/ASI is, as you put it, in fact magic and you need to pound the red button early and often, or you can delay committing national suicide until later. I look forward to a clarification of your beliefs.
I don’t think it is magic but it is still sufficiently disgusting to treat it with equal threat now. Red button now.
Its not a good idea to treat a disease right before it kills you: prevention is the way to go.
So no, I don’t think it is magic. But I do think just as the world agreed against human cloning long before there was a human clone, now is the time to act.
So gathering up your beliefs, you believe ASI/AGI to be a threat, but not so dangerous a threat you need to use nuclear weapons until an enemy nation with it is extremely far along, which will take, according to your beliefs, many years since it’s not that good.
But you find the very idea of non human intelligence in use by humans or possibly serving itself so disgusting that you want nuclear weapons used the instant anyone steps out of compliance with international rules you wish to impose. (Note this is historically unprecedented, arms control treaties have been voluntary and did not have immediate thermonuclear war as the penalty for violating them)
And since your beliefs are emotionally based on “disgust”, I assume there is no updating based on actual measurements? That is, if ASI turns out to be safer than you currently think, you still want immediate nukes, and vice versa?
What percentage of the population of world superpower decision makers do you feel share your belief? Just a rough guess is fine.
The point is that sanctions should be applied as necessary to discourage AGI, however, approximate grim triggers should apply as needed to prevent dystopia.
As the other commentators have mentioned, my reaction is not unusual and thus this is why the concerns of doom have been widespread.
So the answer is: enough.