This is a good and interesting point, but it definitely isn’t necessary for learning. As an example, I get why pointing even an unloaded gun at someone you do not intend to kill is generally a bad idea, despite never having had any gun accidents or close calls. I think it’s worth trying to become better at seeing the reasons for these sorts of things without having to go through first-hand experience. This is especially relevant when it comes to reasoning about the dangers of superintelligence, as we will very likely only get one chance.
I think the idea actually works pretty well with superintelligence (with one big exception if you assume we all die). Lots of people don’t understand how/why superintelligence could kill us all. They naively think that creating a superintelligence would be a great idea. If we all died, then they would understand why alignment is a necessary complexity. The only problem with this is that we are all dead.
This is a good and interesting point, but it definitely isn’t necessary for learning. As an example, I get why pointing even an unloaded gun at someone you do not intend to kill is generally a bad idea, despite never having had any gun accidents or close calls. I think it’s worth trying to become better at seeing the reasons for these sorts of things without having to go through first-hand experience. This is especially relevant when it comes to reasoning about the dangers of superintelligence, as we will very likely only get one chance.
I think the idea actually works pretty well with superintelligence (with one big exception if you assume we all die). Lots of people don’t understand how/why superintelligence could kill us all. They naively think that creating a superintelligence would be a great idea. If we all died, then they would understand why alignment is a necessary complexity. The only problem with this is that we are all dead.