Understandable position well articulated. An important issue I have with conservatism (and many AI safety positions) is that it assumes the kind of world that arguably doesn’t exist. That is one that is somewhat stable, safe, and sustainable. The assumption is that without AI the good things we currently know would continue for long enough to matter. If instead we view ourselves in an unlikely timeline where the most likely outcome from the last 100 years is that we have had a full on nuke war then that changes the perspective. Considering all the close calls, if there in hindsight was a 75% chance of nuke war from 1960 till now and we are just lucky then that changes much.
Given that such odds probably havn’t changed i.e. great power conflict with China taking the place of Russia in the next 75 years will give similar dangers, then our current situation is not one to preserve, but instead change as soon as we can. You talk about Russian Roulette but perhaps want to preserve a situation where we arguably already play it every 100 years with 5 bullets in the chamber. That is not including new threats—does pre LLM AI/social media cause collapse after time? Does LLM + dictators cause a permanent 1984 style world given time?
If you believe that humanity is an a very dangerous phase of unavoidable change then it is about getting out of that situation with the highest chance, rather then attempting to preserve the current seemingly safe-ish situation. ASI is one way, large scale space colonization (different stars) is another.
Russell—I take your point that in most alternative timelines, we would already be dead, decades ago, due to nuclear war, and I often make that point in discussing AI risk, to hammer home the point that humanity does not have any magical ‘character armor’ that will protect us from extinction, and that nobody is coming to save us if we’re dumb enough to develop AGI/ASI.
However, I disagree with the claim that ‘our current situation is not one to preserve’. I know people in the military/intelligence communities who work full time on nuclear safety, nuclear non-proliferation, counter-terrorism, etc. There are tens of thousands of smart people across dozens of agencies across many countries who spend their entire lives reducing the risks of nuclear war. They’re not just activists making noise from outside the centers of power. They’re inside the government, with high security clearances, respected expertise, and real influence. I’m not saying the risk of nuclear war has gone to zero, but it is taken very seriously by all the major world governments.
By contrast, AI safety remains something of a fringe issue, with virtually no representation inside governments, corporations, media, academia, or any other power centers. That’s the thing that needs to change.
We don’t need a ‘hail Mary’ where we develop AGI/ASI and then hope that it can reduce nuclear risk more than it increases all other risks.
I am glad there are people working on nuclear safety and absolutely agree there should be more AI safety inside governments. I also think pre-LLM AI tech should get more attention—Peter Thiel I think makes the point that software has very little regulation compared to much physical things yet it can have enormous influence. I’m sure I don’t need to persuade you that the current dating situation is not ideal. What can be practically done about it all things considered however is not so clear.
However those nuke safety people aren’t working inside Russia as far as I am aware? My point is that we still don’t know what such risk is as of now, nor do we have much of an estimate in the coming decades. The justifiable uncertainty is huge. My position when considering a pause/stop depends on weighing up things we can really only guess at.
To consider when say delaying ASI 50+ years we need to know:
What is the chance of nuke war/lethal pandemic etc in that time? 2%, 90%?
What will LLM tech and similar do to our society? Specifically what is the chance that it will degrade our society in some way that when we do choose to go ahead with ASI we get “imagine a boot stamping on a human face – for ever.” While pure x-risk may be higher with immediate ASI, I think S-risk will be higher with a delay. In the past, centralization and dictators would eventually fail. Now imagine if a place like N Korea gave everyone a permanent bracelet that recorded everything they said paired to an LLM that also understood their hand gestures and facial expressions. They additionally let pre-ASI AI run their society so that central planning actually could work. I think that there is no coming back from that.
Now even if such a country is economically weaker than a free one, if there is a % chance each decade that free societies fall into such an attractor, then eventually the majority of economic output ends up in such a system. They then “solve” alignment getting an ASI that does their bidding.
What is the current X-risk, and what would it be after 50 years of alignment research?
I believe that pre GPT-3/3.5, further time spent on alignment would be essentially a dead end. Without actual data we get into diminishing returns, and likely false confidence on results and paradigms. However it is clear that X-risk could be a lot lower if done right. To me that means actually building ever more powerful AI in very limited and controlled situations. So yes a well managed delay could greatly reduce X-risk.
There are 4 very important unknowns here, potentially 5 if you separate out S risk. How to decide? Is +2% more S-risk acceptable if you take X risk from 50% to 5%? Different numbers for these situations will give very different recommendations. If the current world was going well, then sure its easy to see that a pause/stop is the best option.
What to do?
From this it is clear that work on actually making the current world safer is very valuable. That is protecting institutions that work, anticipating future threats and making the world more robust against them. Unfortunately that doesn’t mean that keeping the current situation as long as possible is the best all things considered.
If someone thought there is a high chance that ASI is coming soon or that even with the best efforts the current world can’t be made sufficiently safe, then they would want to work on making ASI go well, for example mechanistic interpretability research or other practical alignment work.
Expressing such uncertainty on my part probably won’t get me invited to make speeches and can come across as a lack of moral clarity. However it is my position and I don’t think behavior based on the outcome of those uncertainties should be up for moral stigmatization.
These are not my numbers but lets say you have 50% for nuke war/similar event, then 50% for S-risk from the surviving worlds over the next 100 years with no ASI, but 20% X risk/1% S risk from ASI < 5 years. Your actions and priorities are then clear and morally defensible from your probabilities. Some e/acc people may genuinely have these beliefs.
Edited later for my reference Does pursuing WBE change this? Perhaps if you think we can delay ASI but just 20 years to get WBE and believe that they will be better aligned. If you get ASI first and then use them to create WBE that can be seen perhaps as a pivotal act. Stop pure AI but only create WBE is not a strategy I have seen pushed seriously. It doesn’t seem possible without first having massive GPU control etc as its pretty clear without constraints pure AI will be made first. For example if you have the tech to scan enough of a brain, then you are pretty much guaranteed to be able to make ASI from what you have learnt before you have scanned the whole brain.
Understandable position well articulated.
An important issue I have with conservatism (and many AI safety positions) is that it assumes the kind of world that arguably doesn’t exist. That is one that is somewhat stable, safe, and sustainable. The assumption is that without AI the good things we currently know would continue for long enough to matter.
If instead we view ourselves in an unlikely timeline where the most likely outcome from the last 100 years is that we have had a full on nuke war then that changes the perspective. Considering all the close calls, if there in hindsight was a 75% chance of nuke war from 1960 till now and we are just lucky then that changes much.
Given that such odds probably havn’t changed i.e. great power conflict with China taking the place of Russia in the next 75 years will give similar dangers, then our current situation is not one to preserve, but instead change as soon as we can. You talk about Russian Roulette but perhaps want to preserve a situation where we arguably already play it every 100 years with 5 bullets in the chamber. That is not including new threats—does pre LLM AI/social media cause collapse after time? Does LLM + dictators cause a permanent 1984 style world given time?
If you believe that humanity is an a very dangerous phase of unavoidable change then it is about getting out of that situation with the highest chance, rather then attempting to preserve the current seemingly safe-ish situation. ASI is one way, large scale space colonization (different stars) is another.
Russell—I take your point that in most alternative timelines, we would already be dead, decades ago, due to nuclear war, and I often make that point in discussing AI risk, to hammer home the point that humanity does not have any magical ‘character armor’ that will protect us from extinction, and that nobody is coming to save us if we’re dumb enough to develop AGI/ASI.
However, I disagree with the claim that ‘our current situation is not one to preserve’. I know people in the military/intelligence communities who work full time on nuclear safety, nuclear non-proliferation, counter-terrorism, etc. There are tens of thousands of smart people across dozens of agencies across many countries who spend their entire lives reducing the risks of nuclear war. They’re not just activists making noise from outside the centers of power. They’re inside the government, with high security clearances, respected expertise, and real influence. I’m not saying the risk of nuclear war has gone to zero, but it is taken very seriously by all the major world governments.
By contrast, AI safety remains something of a fringe issue, with virtually no representation inside governments, corporations, media, academia, or any other power centers. That’s the thing that needs to change.
We don’t need a ‘hail Mary’ where we develop AGI/ASI and then hope that it can reduce nuclear risk more than it increases all other risks.
I am glad there are people working on nuclear safety and absolutely agree there should be more AI safety inside governments.
I also think pre-LLM AI tech should get more attention—Peter Thiel I think makes the point that software has very little regulation compared to much physical things yet it can have enormous influence. I’m sure I don’t need to persuade you that the current dating situation is not ideal. What can be practically done about it all things considered however is not so clear.
However those nuke safety people aren’t working inside Russia as far as I am aware? My point is that we still don’t know what such risk is as of now, nor do we have much of an estimate in the coming decades. The justifiable uncertainty is huge. My position when considering a pause/stop depends on weighing up things we can really only guess at.
To consider when say delaying ASI 50+ years we need to know:
What is the chance of nuke war/lethal pandemic etc in that time? 2%, 90%?
What will LLM tech and similar do to our society?
Specifically what is the chance that it will degrade our society in some way that when we do choose to go ahead with ASI we get “imagine a boot stamping on a human face – for ever.” While pure x-risk may be higher with immediate ASI, I think S-risk will be higher with a delay. In the past, centralization and dictators would eventually fail. Now imagine if a place like N Korea gave everyone a permanent bracelet that recorded everything they said paired to an LLM that also understood their hand gestures and facial expressions. They additionally let pre-ASI AI run their society so that central planning actually could work. I think that there is no coming back from that.
Now even if such a country is economically weaker than a free one, if there is a % chance each decade that free societies fall into such an attractor, then eventually the majority of economic output ends up in such a system. They then “solve” alignment getting an ASI that does their bidding.
What is the current X-risk, and what would it be after 50 years of alignment research?
I believe that pre GPT-3/3.5, further time spent on alignment would be essentially a dead end. Without actual data we get into diminishing returns, and likely false confidence on results and paradigms. However it is clear that X-risk could be a lot lower if done right. To me that means actually building ever more powerful AI in very limited and controlled situations. So yes a well managed delay could greatly reduce X-risk.
There are 4 very important unknowns here, potentially 5 if you separate out S risk. How to decide? Is +2% more S-risk acceptable if you take X risk from 50% to 5%? Different numbers for these situations will give very different recommendations. If the current world was going well, then sure its easy to see that a pause/stop is the best option.
What to do?
From this it is clear that work on actually making the current world safer is very valuable. That is protecting institutions that work, anticipating future threats and making the world more robust against them. Unfortunately that doesn’t mean that keeping the current situation as long as possible is the best all things considered.
If someone thought there is a high chance that ASI is coming soon or that even with the best efforts the current world can’t be made sufficiently safe, then they would want to work on making ASI go well, for example mechanistic interpretability research or other practical alignment work.
Expressing such uncertainty on my part probably won’t get me invited to make speeches and can come across as a lack of moral clarity. However it is my position and I don’t think behavior based on the outcome of those uncertainties should be up for moral stigmatization.
These are not my numbers but lets say you have 50% for nuke war/similar event, then 50% for S-risk from the surviving worlds over the next 100 years with no ASI, but 20% X risk/1% S risk from ASI < 5 years. Your actions and priorities are then clear and morally defensible from your probabilities. Some e/acc people may genuinely have these beliefs.
Edited later for my reference
Does pursuing WBE change this? Perhaps if you think we can delay ASI but just 20 years to get WBE and believe that they will be better aligned. If you get ASI first and then use them to create WBE that can be seen perhaps as a pivotal act. Stop pure AI but only create WBE is not a strategy I have seen pushed seriously. It doesn’t seem possible without first having massive GPU control etc as its pretty clear without constraints pure AI will be made first. For example if you have the tech to scan enough of a brain, then you are pretty much guaranteed to be able to make ASI from what you have learnt before you have scanned the whole brain.