To the point of contention, I believe it is actually fairly well illustrated on the website, that it is a fallacy in itself to make it up to such nuances and to demand proof of any particular of the possible outcomes in the future to shape one’s actions, if it pertains to the question of whether or not you should do your bests to mitigate the risk in ensuring your basic survival.
An unsurvivable AGI outcome is just one of the many possible scenarios. Although you can speculate about the details of how it could play out (partial extermination, full extermination, no extermination) and what means AGI might use, the whole point of thinking logically about the issue in terms of ensuring your survival is to recognize that those things are ultimately unknowable. What you have to do is to simply develop a strategy that deals with all eventualities as best as possible.
I don’t know if it is clear to you, but those are the basic scenarios of AGI development taken into consideration on the website, which are basically everything you can consider:
AGI will be docile, that is super-intelligent but fully obedient to humans -> society can collapse due to abuse or misuse (ranging from mild to severe) -> society can collapse due to system shock (ranging from mild to severe) -> society can flourish due to benefits
AGI will become fully independent -> accidental extinction due to insanity (ranging from mild to severe) -> deliberate extinction to eliminate humans as a threat (ranging from mild to severe) -> coincidental extinction due to disregard for human life (ranging from mild to severe) -> AGI will become god or a guardian angle and help or digitalize human existence -> AGI will leave on a space ship and simply leave us alone, because we are insignificant and there is an infinite amount of planets to exploit resources from
Experimental stages of AGI lead to some kind of destruction
AGI will not develop in the near future, because it is too complicated or outlawed
AGIs will fight against each other and may cause some kind of detrimental chaos
Obviously you can also mix those scenarios, as they all derive from each other, and they all can be rather permanent or temporary in nature, mild or severe as well. For example you could have a collapsing stock market, due to system shock of hyper-intelligent but fully human-controlled AI systems, all the while independent AGI systems are on the rise 2 years later and then waging digital wars against each other and some of them try to exterminate humans, while others try to protect them or enslave them. While this seems somewhat ridiculous to consider, it is just to illustrate that there is a wide range of possible outcomes with a wide range of details, and no single definitive outcome (e.g. full extinction of every last human on earth from AGI) can be determined with any degree of certainty.
In the end in most scenarios, society will recover in time and some amount of human life will still be present afterwards. But even if there was just a single scenario with the off chance of human survival, then this would be enough to work towards by literally spending all your time and resources on it. Anything else can only be described as suicidal.
Personally though, I believe the chances of overall human survival are very high, and the chances of hostile AGI are rather low. This is exactly how it is reflected by expert opinion as well. But there is a lot of in-between risk that you need to consider, most of which concerns intermediary stages of AGI, and this is where spending reasonable amounts of money (e.g. 2000 Euros) come into play.
So I was hoping you can help me to improve the page and tell me how that was not clear from reading it through and maybe how I can write it differently, without adding thousands of words to the page.
This guide was not written for LessWrong btw., but for common people who are smart enough to follow the arguments and are willing to protect themselves if necessary in light of this new information and proper consideration and risk management.
But even if there was just a single scenario with the off chance of human survival, then this would be enough to work towards by literally spending all your time and resources on it. Anything else can only be described as suicidal.
Well, sure, I see the logic in that. Unlike you, however, my probability that (even if I started preparing in earnest now) I would survive an AGI that has taken control of most human infrastructure and has the capacity to invent new technologies is so low that my best course of action (best out of a bad lot) is to put all my efforts into preventing the creation of the AGI.
I arrive at that low probability by asking myself what would I do (what subgoals I would pursue) if I assigned zero intrinsic value to human life and human preferences and I could invent powerful new technologies (and discover new scientific laws that no human knows about) and my ability to plan were truly superhuman. I.e., I put myself in the “shoes” of the AGI (and I reflected on that vantage point over many days).
In other words, by the time AGI research has progressed to the stage that things like my supply of food, water and electricity get disrupted by an AGI, it is almost certainly already too late for me on my models.
OK. Which is the audience you are hoping to influence? Are you hoping to alert preppers to the dangers of AI or are you hoping to get people who already know AI is dangerous (like many of us here) to prepare themselves for shortages and mobs of desperate unprepared people? (It would be a bad idea IMO to try to engage both audiences with a single set of web pages.)
The audience is the general public. That is anyone who has the attention span and is smart enough to read what I wrote, without feeling the need to disregard the idea out of comfort, personal conviction, laziness, etc. I was toying with the idea of writing a much shorter version for stupid people. But then I think that is just an exercise in futility. And also I don’t really like the idea of stupid people gaining such an existential advantage.
Unfortunately I am not allowed to create another post the next 7 days, due to low karma. I have written a new post however that I will post soon. It is quite long. If you are interested you can read it here:
Maybe you could tell me how much it is on track, from your experience with this community. And I would also like to understand any reasons how my perspective could be flawed. I was reading quite a bit on here and watching some videos. Although influenced a lot by your replies, it is not meant to attack you, but those who follow by authority and influence of Eliezer and don’t think for themselves instead.
I don’t write and will not write those these things to please people though. If people cannot jump over their own shadow and process dissonance in a healthy constructive way, then so be it.
Thanks for your reply.
To the point of contention, I believe it is actually fairly well illustrated on the website, that it is a fallacy in itself to make it up to such nuances and to demand proof of any particular of the possible outcomes in the future to shape one’s actions, if it pertains to the question of whether or not you should do your bests to mitigate the risk in ensuring your basic survival.
An unsurvivable AGI outcome is just one of the many possible scenarios. Although you can speculate about the details of how it could play out (partial extermination, full extermination, no extermination) and what means AGI might use, the whole point of thinking logically about the issue in terms of ensuring your survival is to recognize that those things are ultimately unknowable. What you have to do is to simply develop a strategy that deals with all eventualities as best as possible.
I don’t know if it is clear to you, but those are the basic scenarios of AGI development taken into consideration on the website, which are basically everything you can consider:
AGI will be docile, that is super-intelligent but fully obedient to humans
-> society can collapse due to abuse or misuse (ranging from mild to severe)
-> society can collapse due to system shock (ranging from mild to severe)
-> society can flourish due to benefits
AGI will become fully independent
-> accidental extinction due to insanity (ranging from mild to severe)
-> deliberate extinction to eliminate humans as a threat (ranging from mild to severe)
-> coincidental extinction due to disregard for human life (ranging from mild to severe)
-> AGI will become god or a guardian angle and help or digitalize human existence
-> AGI will leave on a space ship and simply leave us alone, because we are insignificant and there is an infinite amount of planets to exploit resources from
Experimental stages of AGI lead to some kind of destruction
AGI will not develop in the near future, because it is too complicated or outlawed
AGIs will fight against each other and may cause some kind of detrimental chaos
Obviously you can also mix those scenarios, as they all derive from each other, and they all can be rather permanent or temporary in nature, mild or severe as well. For example you could have a collapsing stock market, due to system shock of hyper-intelligent but fully human-controlled AI systems, all the while independent AGI systems are on the rise 2 years later and then waging digital wars against each other and some of them try to exterminate humans, while others try to protect them or enslave them. While this seems somewhat ridiculous to consider, it is just to illustrate that there is a wide range of possible outcomes with a wide range of details, and no single definitive outcome (e.g. full extinction of every last human on earth from AGI) can be determined with any degree of certainty.
In the end in most scenarios, society will recover in time and some amount of human life will still be present afterwards. But even if there was just a single scenario with the off chance of human survival, then this would be enough to work towards by literally spending all your time and resources on it. Anything else can only be described as suicidal.
Personally though, I believe the chances of overall human survival are very high, and the chances of hostile AGI are rather low. This is exactly how it is reflected by expert opinion as well. But there is a lot of in-between risk that you need to consider, most of which concerns intermediary stages of AGI, and this is where spending reasonable amounts of money (e.g. 2000 Euros) come into play.
So I was hoping you can help me to improve the page and tell me how that was not clear from reading it through and maybe how I can write it differently, without adding thousands of words to the page.
This guide was not written for LessWrong btw., but for common people who are smart enough to follow the arguments and are willing to protect themselves if necessary in light of this new information and proper consideration and risk management.
Well, sure, I see the logic in that. Unlike you, however, my probability that (even if I started preparing in earnest now) I would survive an AGI that has taken control of most human infrastructure and has the capacity to invent new technologies is so low that my best course of action (best out of a bad lot) is to put all my efforts into preventing the creation of the AGI.
I arrive at that low probability by asking myself what would I do (what subgoals I would pursue) if I assigned zero intrinsic value to human life and human preferences and I could invent powerful new technologies (and discover new scientific laws that no human knows about) and my ability to plan were truly superhuman. I.e., I put myself in the “shoes” of the AGI (and I reflected on that vantage point over many days).
In other words, by the time AGI research has progressed to the stage that things like my supply of food, water and electricity get disrupted by an AGI, it is almost certainly already too late for me on my models.
OK. Which is the audience you are hoping to influence? Are you hoping to alert preppers to the dangers of AI or are you hoping to get people who already know AI is dangerous (like many of us here) to prepare themselves for shortages and mobs of desperate unprepared people? (It would be a bad idea IMO to try to engage both audiences with a single set of web pages.)
The audience is the general public. That is anyone who has the attention span and is smart enough to read what I wrote, without feeling the need to disregard the idea out of comfort, personal conviction, laziness, etc. I was toying with the idea of writing a much shorter version for stupid people. But then I think that is just an exercise in futility. And also I don’t really like the idea of stupid people gaining such an existential advantage.
Unfortunately I am not allowed to create another post the next 7 days, due to low karma. I have written a new post however that I will post soon. It is quite long. If you are interested you can read it here:
https://pastebin.com/7WR0P8ZM
Maybe you could tell me how much it is on track, from your experience with this community. And I would also like to understand any reasons how my perspective could be flawed. I was reading quite a bit on here and watching some videos. Although influenced a lot by your replies, it is not meant to attack you, but those who follow by authority and influence of Eliezer and don’t think for themselves instead.
I don’t write and will not write those these things to please people though. If people cannot jump over their own shadow and process dissonance in a healthy constructive way, then so be it.