A) It doesn’t even have to be at superhuman levels, per se. While that would be a lot more dangerous, a very scalable human level AGI could also be a big problem. Though I suppose that scalability would totally classify as superhuman…
C) killing everyone is the extreme case (or one of them). A less extreme, but more general version is the removal of agency. Where humanities potential is taken over by AIs. Something like humans being moved to nature reserves where they can live in picturesque villages but have no influence on the future.
That being said, the fundamental points seem quite good, though I’d quibble with:
3 - all kinds of final stands and kamikaze are done without hope of them working, in order to go out with a flash of glory type thing. Or at least to die with a sword in your hand.
4 - it’s justifiable if you don’t have any other context. You can argue from the expected value that if there are 1000 researchers, then they should choose what they want to work on on the basis of some combination of:
the probability of solving the problem
the probability of it happening
the severity of it happening
So even if it’s very unlikely, the expected negative utility would be very large, so it’s worth spending 1 or 2 researchers, just in case
5 - this sounds rational. But humans often aren’t and will be concerned about things that are available rather than whether they’re helpful. People like to approach people in trouble and ask if they can help, even if they’re pretty sure there is nothing to do
6 - you could argue that caring about the future is also an extension of caring for self, if you believe that your children (or ideas) are in some way an extension of yourself
7 - I’d say that this is totally orthogonal to how much someone cares about existential risks, or anything, really. I might be misunderstanding what you mean by “worry” here. But even if you mean something like “are worried enough about the potential of this problem to actively try to make it better” then you can find examples everywhere of people irrationally worried about things and then attempting irrational methods to solve the problem. This point is good to include in a list of things that someone who wants to fix things should possess, but even then it’s not needed—you can make things better totally by random. It’s not effective, but sometimes works
A) It doesn’t even have to be at superhuman levels, per se. While that would be a lot more dangerous, a very scalable human level AGI could also be a big problem. Though I suppose that scalability would totally classify as superhuman…
C) killing everyone is the extreme case (or one of them). A less extreme, but more general version is the removal of agency. Where humanities potential is taken over by AIs. Something like humans being moved to nature reserves where they can live in picturesque villages but have no influence on the future.
That being said, the fundamental points seem quite good, though I’d quibble with:
3 - all kinds of final stands and kamikaze are done without hope of them working, in order to go out with a flash of glory type thing. Or at least to die with a sword in your hand.
4 - it’s justifiable if you don’t have any other context. You can argue from the expected value that if there are 1000 researchers, then they should choose what they want to work on on the basis of some combination of:
the probability of solving the problem
the probability of it happening
the severity of it happening
So even if it’s very unlikely, the expected negative utility would be very large, so it’s worth spending 1 or 2 researchers, just in case
5 - this sounds rational. But humans often aren’t and will be concerned about things that are available rather than whether they’re helpful. People like to approach people in trouble and ask if they can help, even if they’re pretty sure there is nothing to do
6 - you could argue that caring about the future is also an extension of caring for self, if you believe that your children (or ideas) are in some way an extension of yourself
7 - I’d say that this is totally orthogonal to how much someone cares about existential risks, or anything, really. I might be misunderstanding what you mean by “worry” here. But even if you mean something like “are worried enough about the potential of this problem to actively try to make it better” then you can find examples everywhere of people irrationally worried about things and then attempting irrational methods to solve the problem. This point is good to include in a list of things that someone who wants to fix things should possess, but even then it’s not needed—you can make things better totally by random. It’s not effective, but sometimes works