I agree that it’s psychologically very difficult, and that “is my work a net positive” is also hard to answer.
But I don’t think it’s necessarily about millions of dollars and personal glory. I think the biggest difficulty is the extreme social conflict and awkwardness you would have telling researchers who are very personally close to you to simply shut down their project full of hard work, and tell them to oh do something else that probably won’t make money and in the end we’ll probably go bankrupt.
As for millions of dollars, the top executives have enough money they won’t feel the difference.
As for “still probably die,” well from a rational self interest point of view they should spend the last years they have left on vacation, rather than stressing out at a lab.
As for personal glory, it’s complicated. I think they genuinely believe there is a very decent chance of survival, in which case “doing the hard unpleasant thing” will result in far more glory in the post-singularity world. I agree it may be a factor in the short term.
I think questions like “is my work a net positive?” “Is my ex-girlfriend more correct about our breakup than me?” and “Is the political party I like running the economy better?” are some of the most important questions in life. But all humans are delusional about these most important questions in life, and no matter how smart you are, wondering about these most important questions will simply give your delusions more time find reassurances that you aren’t delusional.
The only way out is to look at how other smart rational people are delusional, and how futile their attempts at self questioning are, and infer that holy shit this could be happening to me too without me realizing it.
Not sure I get your overall position. But I don’t believe all humans are delusional about the most important questions in their lives. See here for an analysis of pressures on people that can cause them to be insane on a topic. I think you can create inverse pressures in yourself, and you can also have no pressures and simply use curiosity and truth-seeking heuristics. It’s not magic to not be delusional. It just requires doing the same sorts of cognition you use to fix a kitchen sink.
Admittedly, I got a bit lost writing the comment. What I should’ve wrote was: “not being delusional is either easy or hard.”
If it’s easy, you should be able to convince them to stop being delusional, since it’s their rational self interest.
If it’s hard, you should be able to show them how hard and extremely insidious it is, and how one cannot expect oneself to succeed, so one should be far more uncertain/concerned about delusion.
I agree that it’s psychologically very difficult, and that “is my work a net positive” is also hard to answer.
But I don’t think it’s necessarily about millions of dollars and personal glory. I think the biggest difficulty is the extreme social conflict and awkwardness you would have telling researchers who are very personally close to you to simply shut down their project full of hard work, and tell them to oh do something else that probably won’t make money and in the end we’ll probably go bankrupt.
As for millions of dollars, the top executives have enough money they won’t feel the difference.
As for “still probably die,” well from a rational self interest point of view they should spend the last years they have left on vacation, rather than stressing out at a lab.
As for personal glory, it’s complicated. I think they genuinely believe there is a very decent chance of survival, in which case “doing the hard unpleasant thing” will result in far more glory in the post-singularity world. I agree it may be a factor in the short term.
I think questions like “is my work a net positive?” “Is my ex-girlfriend more correct about our breakup than me?” and “Is the political party I like running the economy better?” are some of the most important questions in life. But all humans are delusional about these most important questions in life, and no matter how smart you are, wondering about these most important questions will simply give your delusions more time find reassurances that you aren’t delusional.
The only way out is to look at how other smart rational people are delusional, and how futile their attempts at self questioning are, and infer that holy shit this could be happening to me too without me realizing it.
Not sure I get your overall position. But I don’t believe all humans are delusional about the most important questions in their lives. See here for an analysis of pressures on people that can cause them to be insane on a topic. I think you can create inverse pressures in yourself, and you can also have no pressures and simply use curiosity and truth-seeking heuristics. It’s not magic to not be delusional. It just requires doing the same sorts of cognition you use to fix a kitchen sink.
Admittedly, I got a bit lost writing the comment. What I should’ve wrote was: “not being delusional is either easy or hard.”
If it’s easy, you should be able to convince them to stop being delusional, since it’s their rational self interest.
If it’s hard, you should be able to show them how hard and extremely insidious it is, and how one cannot expect oneself to succeed, so one should be far more uncertain/concerned about delusion.