Thank you for a very thorough post. I think your writing has served me as a more organized account of some of my own impressions opposing longtermisim.
I agree with CrimsonChin in that I think there’s a lot of your post many longtermists would agree with, including the practicality of focusing on short-term sub-goals. Also, I personally believe that initiatives like global health, poverty reduction, etc. probably improve the prospects of the far future, even if their expected value seems less than X-risk mitigation.
Nonetheless, I still think we should be motivated by the immensity of the future even if it is off set by tiny probabilities and there are huge margins of error, because the lower bounds of these estimates appear to me as sufficiently high to be very compelling. The post How Many Lives Does X-Risk Work Save From Nonexistance On Average demonstrates my thinking on this by having estimates of future lives that vary by dozens of orders of magnitude(!) but still arrives at very high expected values for X-Risk work even on the lower bounds.
Even I don’t really feel anything when I read such massive numbers, and I acknowledge how large the intervals of these estimates are, but I wouldn’t say they “make no sense to me” or that ‘To the extent we can quantify existential risks in the far future, we can only say something like “extremely likely,’ ‘possible,’ or ‘can’t be ruled out.’”
For what it’s worth, I use to essentially be an egoist, and was unmoved by all of the charities I had ever encountered. It seemed to me that humanity was on a good trajectory and my personal impact would be negligible. It was only after I started thinking about really large numbers, like the duration of the universe, the age of humanity, the number of potential minds in the universe (credit to SFIA), how neglected these figures were, and moral uncertainty, that I started to feel like I could and should act for others.
There are definitely many, possibly most, contexts where incredibly large or small numbers can be safely disregarded. I wouldn’t be moved by them in adversarial situations, like Pascal’s Mugging, or when doing my day-to-day moral decision making. But for question’s like “What should I really care deeply about?” I think they should be considered.
As for Pascal’s Wager, It calls for picking a very specific God to worship out of a space of infinite possible contradictory gods, and this infinitely small probability of success cancels out the infinite reward of heaven over hell or non-existance. Dissimilarly, Longtermism isn’t committed to any specific action regarding the far future, just the well being of entities in the future generally. I expect that most longtermists would gladly pivot away from a specific cause area (like AI alignment) if they were shown some other cause (E.g. a planet-killing asteroid certainly colliding with Earth in 100 years) was more likely to similarly adversely impact the far future.
Thank you for making these threads. I have been reading LW off and on for several years and this will be my first post.
My question: Is purposely leaning into creating a human wire-header an easier alignment target to hit than the more commonly touted goal of creating an aligned superintelligence that prevents the emergence of other potentially dangerous superintelligence, yet somehow reliably leaves humanity mostly in the driver’s seat?
If the current forecast on aligning superintelligent AI is so dire, is there a point where it would make sense to just settle for ceding control and steering towards creating a superintelligence very likely to engage in wire heading humans (or post-humans)? I’m imagining the AI being tasked with tiling the universe with as many conscious entities as possible, with each experiencing as much pleasure as possible, and maybe with a bias towards maximizing pleasure over number of conscious entities as those goals constrain each other. I don’t want to handwave this as being easy, I’m just curious if there’s been much though to removing the constraint of “don’t wirehead humans.”
Background on me: One of my main departures from what I’ve come across so far is that I don’t share as much concern about hedonism or wire heading. It seems to me a superintelligence would grasp that humans in their current form require things like novelty, purpose, and belonging and wouldn’t just naively pump humans—as they currently are—full of drugs and call it a day. I don’t see why these “higher values” couldn’t be simulated or stimulated too. If I learned my current life was a simulation, but one that was being managed by an incredibly successful AI that could reliably keep the simulation running unperturbed, I would not want to exit my current life in order to seize more control in the “real world.”
Honestly, if an AI could replace humans with verifiably conscious simple entities engineered to experience greater satisfaction than current humans and without ever feeling boredom, I’m hard pressed to thing of anything more deserving of the universe’s resources.
The main concern I have with this so far is that the goal of “maximizing pleasure” could be very close in idea-space to “maximizing suffering,” but it’s still really hard for me to see a superintelligence cable of becoming a singleton making such an error or why it would deliberately switch to maximizing suffering.