Saying the quiet part out loud: trading off x-risk for personal immortality
Statement: I want to deliberately balance the caution and the recklessness in developing AGI, such that it gets created in the last possible moment so that I and my close ones do not die.
This Statement confuses me. There are several observations I can make about it. There are also many questions I want to ask but have no idea how to answer. The goal of this post is to deconfuse myself, and to get feedback on the points that I raised (or failed to raise) below.
First observation: The Statement is directly relevant to LW interests.
It ties together the issues of immortality and AI risk, both of which are topics people here are interested in. There are countless threads, posts and discussions about high-level approaches to AI safety, both in the context of “is” (predictions) and “ought” (policy). At the same time, there is still a strong emphasis on the individual action, deliberating on which choices to make to improve the to marginal effects of living life in a certain way. The same is true for immortality. It has been discussed to death, both from the high-level and from the individual, how-do-I-sign-up-for-Alcor point of view. The Statement has been approached from the “is”, but not from the “ought” perspective. At the same time:
Second observation: No one talks about the Statement.
I have never met anyone who expressed this opinion, neither in-person nor online, even after being a part (although, somewhat on the periphery) of the rationalist community for several years. Not only that, I have not been able to find any post or comment thread on LW or SSC/ACX that discusses it, argues for or against it, or really gives it any attention whatsoever. I am confused by this since the Statement seems to be fairly straightforward.
One reason might be the:
Third observation: Believing in the Statement is low status, as it constitutes an almost-taboo opinion.
Not only no one is discussing it, but the few times when I expressed the Statement in person (at EA-infiltrated rationalists meetups), it was treated with suspicion or hostility. Although to be honest, I’m not sure how much this is me potentially misinterpreting the reactions. I got the impression that it is seen as sociopathic. Maybe it is?
Fourth observation: Believing in the Statement is incompatible with long-termism, and it runs counter to significantly valuing future civilisation in general.
Fifth observation: Believing in the Statement is compatible with folk morality and revealed preferences of most of the population.
Most people value their lives, and the lives of those around them to a much greater extent than those far away from them. This is even more true for the future lives. The revealed-preference discount factor is bounded away from 1.
Sixth observation: The Statement is internally consistent.
I don’t see any problems with it on the purely logical level. Rational egoism (or variants thereof) constitutes a valid ethical theory, although it is potentially prone to self-defeat.
Seventh observation: Because openly admitting to believing in the Statement is disadvantageous, it is possible that many people in fact hold this opinion secretly.
I have no idea how plausible this is. Judging this point is one of my main goals in writing this post. The comments are a good place for debating the meta-level points, but, if I am right about the cost of holding this opinion—not so much for counting its supporters. An alternative is this anonymous poll I created—please vote if you’re reading this.
Eighth observation: The Statement has the potential to explain some of the variance of attitudes to AI risk-taking.
One way of interpreting this observation might be that people arguing against pausing/for accelerating AI developments are intentionally hiding their motivations. But, using the standard Elephant-in-the-brain arguments about people’s tendency to self-delude about their true reasons for acting or believing in certain things, it seems possible that some of the differences between Pause-aligned and e/acc-aligned crowd come down to this issue. Again, I have no idea how plausible this is. (People in e/acc are—usually—already anonymous, which is a point against my theory).
Ninth observation: The Statement is not self-defeating in the sense of failure of ethical theories to induce coordination in the society/escaping prisoner dilemmas.
Although the Statement relies on the ethical judgement of gambling the entire future of our civilisation on my meagre preferences for my life and life of people around me, everyone’s choices are to a greater or lesser extent aligned (depending on the similarity of life expectancy). Most probably we, as the “current population” survive together or die together—with everyone dying being the status quo. Trading off future lives does not create coordination problems (ignoring the possibility of acausal trade), leaving only the conflict between different age groups.
This leads me to the...
Tenth observation: Openly resolving the topic of the Statement leads to a weird political situation.
As we said, the Statement implicitly refers to the Pareto frontier of AI existential risk-taking, which is parametrised by the life expectancy of the individual. Therefore, assuming a sufficiently powerful political system, it is a negotiation about the cut-off point—who gets left behind and who gets to live indefinitely. There does not seem to be much negotiation space—fairly compensating the first group seems impossible. Thus, this seems like a high-conflict situation. In a real-life not-so-powerful system, it might lead to unilateral action of people left below the cut-off, but they might not have enough resources to carry out the research and development alone. Also, the probabilistic nature of that decision would probably not trigger any strong outrage response.
Eleventh observation: Much of the “working EA theory” might break when applied to the Statement.
How much do people posture altruism, and how much do they really care? EA signalling can be a good trade (10% of your income in exchange for access to the network and being well-liked overall). This was the argument SSC was making in one of the posts: the beauty of the EA today is that the inside doesn’t really matter—being motivated by true altruism and being motivated by status are both still bringing the same result at the end. This can, however, break if the stakes are raised sufficiently high. (I personally approach the problem of death with a pascal-wager-like attitude, but even with a discount rate of 0.9, the choice comes down to a 10x life value).
Twelfth observation: The Statement does not trivialise either way: the optimal amount of neither recklessness nor caution is zero.
The latter is a consensus (at least on LW). The former is not. The only take I ever heard advocating for this was that cryonics + certainly aligned AI is worth more in expectation than the chance of aligned AI in our lifetimes. For obvious reasons, I consider this to be a very weak argument.
Thirteenth observation: For a median believer of the Statement, we might be passing the point where the balance is being too heavily weighted towards pause.
The fundamental difference between the pause side and the accelerate side is that pause has a much bigger social momentum to it. Governments are good at delaying and introducing new regulations—not so much at speeding up and removing them (at least not recently). I can’t find a source, but I think EY admitted this in one of the interviews—the only good thing about the government is that it will grind everything to a complete stop. After seeing literally all major governments take interest (US executive order, UK summit), the academia heavily leaning towards pause, and the general public being more and more scared of AI—I worry that the pause is going to be too entrenched, will have too much political momentum to stop and reverse course. I don’t know how much should I consider the market forces being a counterweight.
Fourteenth observation: Taking into account s-risks in addition to x-risk might change the tradeoff.
A final argument about which I am unsure—it seems plausible that the Statement should not be endorsed by anyone because of the potential of unaligned AI to cause suffering. But I do not think s-risks are sufficiently probable to pause indefinitely.
So: Am I the only one thinking those thoughts? Am I missing any obvious insights? Is that really a taboo view? Was it debated before and resolved definitely one way or another? Is this some standard philosophical problem that already has a Google-able name? Is there any downside to openly discussing this whole thing that I don’t see?
At this point timelines look short enough that you likely increase your personal odds of survival more by increasing the chance that AI goes well than by speeding up timelines. Also I don’t see why you think cryonics doesn’t make sense as alternative option.
Cryonics is likely a very tough sell to the “close ones”.
I wish I could convince my grandpa to sign up for cryonics, but he’s a 95 yo Indian doctor in India, where facilities for cryopreservation only extends to organs and eggs, so it’s moot regardless of the fact that I can’t convince him.
I expect my parents to survive to the Singularity, whether or not it kills us in the process. Same for me, and given my limited income, I’m not spending it on cryonics given that a hostile AGI will kill even the ones frozen away.
You probably also irreversibly lose a ton of information with cryonics.
The timelines certainly still looked short enough a couple of months ago. But what prompted me to write this was the 13th observation: the seemingly snowballing Pause movement, which, once it reaches certain threshold, has a potential to significantly stifle the development of AI. Analogies: human genetic enhancement, nuclear energy. I’m not sure whether this is already past the point of countering the opposite forces (useful applications, Moore’s law), but I’m also not sure that it isn’t (or won’t be soon).
Cryonics is a very speculative tech. We don’t understand how much information is lost in the process, scientific evidence seems lacking overall—consensus being it’s in the ~few percent success probability region, future AI (future society) would have to want to revive humans instead of creating new ones, etc.
> consensus being it’s in the ~few percent success probability region
Consensus among who? I haven’t been able to find a class of experts I’d defer to. We have Alcor, who are too partial, we have the society of cryobiology who openly refuse to learn anything about the process and threaten to exile any member who does, and I have random members of the rationalist community who have no obligation to be right and just want to sound grounded.
In order for a pause to work it has to happen everywhere. Nuclear power is widely deployed in e.g. France, so you need a stronger political force than the one causing nuclear power not to proliferate.
AI is also more like the “keys to the kingdom” here.
The benefit of nuclear power isn’t that huge, you still have fossil fuels which are cheap (even if they cause climate change in the long run).
The benefits of genetic editing/eugenics is also pretty nebulous and may take decades to realize.
On the other hand, one country having an aligned ASI offers an overwhelming advantage. World dominance goes from fiction to mundane reality. I think these sort of treaties also advertise this fact so I think it’s likely they won’t work. All governments are probably seriously considering what I mentioned above.
Why is the U.S. blocking Chinas access to GPUs? That seems like the most plausible explanation.
High levels GPUs are needed for basically anything mundane today. No need to bring in AGI worries to make it a strategic ressource.
I think the timing and perf focus of it all makes it clear it’s related to foundation models.
If you think cryonics has a very high likelihood of working then sure. I don’t think the arguments that cryonics is likely to work are that good though. I don’t think Eliezer has even made arguments to that effect. They were mostly “hey doing cryonics is better than not doing it because not doing cryonics is just death!”
I think Eliezer is pretty confident that cryonics will work. For myself, I’m not sure, haven’t really looked into it that deeply, but the a priori argument makes sense and I feel like this is the kind of thing people would be irrationally biased against due to its speculative nature(similar to the AGI skepticism that many had until recently) so I’d give it decent odds.
I was responding to this point. The “cryonics is better than nothing” argument doesn’t make cryonics an alternative option to immortality by friendly AI. If Bob thinks cryonics has a 10% chance of making him immortal and thinks AI will have a 20% chance of making him immortal and an 80% chance of destroying the world, then the superhuman AI route is more likely to lead to Bob’s immortality than cryonics.
I didn’t say that “cryonics is better than nothing”, I said I think it has decent odds of success. To spell it out, I think the success probability is higher than the increased probability of friendly AI in my lifetime from acceleration(which is the relevant comparison) while imposing fewer costs on future generations. And I think that if you made it your life’s work, you could probably improve those odds, up to 80% perhaps(conditional on future people wanting to revive you)
I do not want the timing of AGI development to be based on my personal life-expectancy. Going further I think most decent people would be willing to sacrifice their own life to prevent their civilization from going extinct, and I think it would be a very honorable thing to do [1]. Every single generation to come is at stake, so I don’t think my own life bears much on whether to defend all 1050+ of theirs.
Not that I am aware of any such opportunity, to be clear.
Note that the idea that >10^50 lives are at stake is typically premised on the notion that there will be a value lock-in event, after which we will successfully colonize the reachable universe. If there is no value lock-in event, then even if we solve AI alignment, values will drift in the long-term, and the stars will eventually be colonized by something that does not share our values. From this perspective, success in AI alignment would merely delay the arrival of a regime of alien values, rather than prevent it entirely. If true, this would imply that positive interventions now are not as astronomically valuable as you might have otherwise thought.
My guess is that the idea of a value lock-in sounded more plausible back in the days when people were more confident there will be (1) a single unified AI that takes control over the world with effectively no real competition forever, and (2) this AI will have an explicit, global utility function over the whole universe that remains unchanging permanently. However, both of these assumptions seem dubious to me currently.
The relevance of value drift here sounds to me analogous to the following exchange.
The concern about AI misalignment is itself a concern about value drift. People are worried that near-term AIs will not share our values. The point I’m making is that even if we solve this problem for the first generation of smarter-than-human AIs, that doesn’t guarantee that AIs will permanently share our values in every subsequent generation. In your analogy, a large change in the status quo (death) is compared to an arguably smaller and more acceptable change over the long term (biological development). By contrast, I’m comparing a very bad thing to another similarly very bad thing. This analogy seems mostly valid only to the extent you reject the premise that extreme value drift is plausible in the long-term, and I’m not sure why you would reject that premise.
Putting “cultural change” and “an alien species comes along and murders us into extinction” into the same bucket seems like a mistake to me. I understand that in each literally one set of values replaces another. But in the latter case, the route by which those values change is something like “an alien was grown in order to maximize a bunch of functions that we were able to define like stock price and next-token prediction and eventually overthrew us”, and I think that is qualitatively different than “people got richer and wealthier and so what they wanted changed” in a way that is likely to be ~worthless from our perspective.
From reading elsewhere my current model (which you may falsify) is that you think that those values will be sufficiently close that they will still be very valuable from our perspective, or about as valuable as people from 2000 years ago would think us today. I don’t buy the first claim; I think the second claim is more interesting but I’m not really confident that it’s relevant or true.
(Consider this an open offer to dialogue about this point sometime, perhaps at my dialogues party this weekend.)
Value drift encompasses a lot more than cultural change. If you think humans messing up on alignment could mean something as dramatic as “an alien species comes along and murders us”, surely you should think that the future could continue to include even more, similarly dramatic shifts. Why would we assume that once we solve value alignment for the first generation of AIs, values would then be locked in perfectly forever for all subsequent generations?
I don’t understand how this new analogy is supposed to apply to the argument, but if I wanted to modify the analogy to get my point across, I’d make Alice 90 years old. Then, I’d point out that, at such an age, getting hit by a car and dying painlessly genuinely isn’t extremely bad, since the alternative is to face death within the next several years with high probability anyway.
If you actually believe that at some point, even with aligned AI, the forces of value drift are so powerful that we are still unlikely to survive, then you’ve just shifted the problem to avoiding this underspecified secondary catastrophe in addition to the first. The astronomical loss is still there.
Human survival is different from the universe being colonized under the direction of human values. Humans could survive, for example, as a tiny pocket of a cosmic civilization.
Indeed, but the question is, “What was the counterfactual?” If solving AI alignment merely delays an astronomical loss, then it is not astronomically important to solve the problem. (It could still be very important in this case, but just not so important that we should think of it as saving 10^50 lives.)
Can you specify the problem(s) you think we might need to solve, in addition to alignment, in order to avoid this sterility outcome?
We would need to solve the general problem of avoiding value drift. Value drift is the phenomenon of changing cultural, social, personal, biological and political values over time. We have observed it in human history during every generation. Older generations vary on average in what they want and care about compared to younger generations. More broadly, species have evolved over time, with constant change on Earth as a result of competition, variation, and natural selection. While over short periods of time, value drift tends to look small, over long periods of time, it can seem enormous.
I don’t know what a good solution to this problem would look like, and some proposed solutions—such as a permanent, very strict global regime of coordination to prevent cultural, biological, and artificial evolution—might be worse than the disease they aim to cure. However, without a solution, our distant descendants will likely be very different from us in ways that we consider morally relevant.
How do you expect this value drift to occur in an environment where humans don’t actually have competitive mental/physical capital? Presumably if humans “drift” beyond some desired bounds, the coalition of people or AIs with the actual compute and martial superiority who are allowing the humans to have these social and political fights will intervene. It wouldn’t matter if there was one AI or twenty, because the N different optimizers are still around and optimizing for the same thing, and the equilibrium between them wouldn’t be meaningfully shifted by culture wars between the newly formed humans.
Value drift happens in almost any environment in which there is variation and selection among entities in the world over time. I’m mostly just saying that things will likely continue to change continuously over the long-term, and a big part of that is that the behavioral tendencies, desires and physical makeup of the relevant entities in the world will continue to evolve too, absent some strong countervailing force to prevent that. This feature of our world does not require that humans continue to have competitive mental and physical capital. On the broadest level, the change I’m referring to took place before humans ever existed.
I do not understand how you expect this general tendency to continue against the will of one or more nearby gpu clusters.
Some people enjoy arguing philosophical points, and there is nothing wrong with that.
Do you believe that the considerations you have just described have any practical relevance to someone who believes that the probability of AI research’s ending all human life some time in the next 60 years is .95 and wants to make a career out of pessimizing that probability?
Yes, I believe this point has practical relevance. If what I’m saying is true, then I do not believe that solving AI alignment has astronomical value (in the sense of saving 10^50 lives). If solving AI alignment does not have astronomical counterfactual value, then its value becomes more comparable to the value of other positive outcomes, like curing aging for people who currently exist. This poses a challenge for those who claim that delaying AI is obviously for the greater good as long as it increases the chance of successful alignment, since that could also cause billions of currently existing people to die.
While I agree, I want to take the opportunity to poke at something I often see in models like this one.
I think if you ask most people about this choice, they’d answer like you predict.
I think if you gave people a choice of buttons to push, one of which is this self-sacrifice button and the other being… uh… whatever the alternative is, there’d be some more hesitance but maybe not a ton.
But I suspect most people do not in fact understand what it means to die, or what they’re agreeing to when they agree to sacrifice themselves for something.
I think this is precisely what changes when someone gets a terminal diagnosis. They come to understand in a lived way what it means that they, right there, inside that body and seeing from behind those eyes, are going to experience death. Their mortality isn’t an abstraction for them anymore. They stop thinking “I’m going to die someday” like they’re talking about a video came character and instead get that it means something way way deeper.
If you adjust the situation so that the person believes the argument that they need to die for the sake of their civilization, and then you hand them a gun with which to shoot themselves…
…I think you’d find the capacity to be “honorable” here dropping dramatically.
But not people saying they would shoot themselves in this hypothetical scenario! Because for the person thinking the thought experiment, the thought experiment doesn’t embed the thought-experimenter.
I don’t think this bears on the discussion of immortality vs. AI risk. The actions here are abstract enough to be more like the button-pushing case.
I just keep seeing this question of embedded agency getting skipped over and I think it’s deeply important.
I’ve seen this kind of opinion before (on Twitter, and maybe reddit?), and I strongly suspect that the average person would react with extreme revulsion to it. It most closely resembles “cartoon villain morality”, in being a direct tradeoff between everyone’s lives and someone’s immortality. People strongly value the possibility of their children and grandchildren being able to have further children of their own, and for things in the world to continue on. And of course, the statement plays so well into stereotypes of politically-oriented age differences: Old people not sufficiently caring about what happens after they die, so they’ll take decisions that let young people deal with catastrophes, young people thinking they’ll never die and being so selfish that they discount the broader world outside themselves, etc. If anything, this is a “please speak directly into the microphone” situation, where the framing would pull people very strongly in the direction of stopping AGI.
I think the current situation is/was greatly distorted by signalling games that people play. Once everyone realises that this is an actual choice, there is a chance they change their opinions to reflect the true tradeoff. (This depends a lot on network effects, shifting Overton window etc., I’m not claiming that 100% of the effect would be rational consideration. But I think rational consideration biases the process to in a non-negligible way.). But yes, one of the pieces of evidence is how old people don’t seem to particularly care about the future of civilisation.
I don’t think this is true. Stereotype evidence: your average conservative old person loves their country and is mad that college campuses are liberal because it means elites will ruin it.
Everyone’s immortality. They don’t typically make cartoon villains like that.
The framing of OP is specifically about racing just slowly enough that some specific people make the cut[1], which is absolutely a tradeoff between everyone’s lives and those specific people’s immortality. OP is explicitly indifferent about the effects on other people, including those who die sooner and those whose chances might be ruined by moving too fast for safety.
which rests on a possibly-questionable received wisdom that AGI is a sufficient and necessary route to immediate immortality
I agree with the Statement. As strongly as I can agree with anything. I think the hope of current humans achieving… if not immortality, then very substantially increased longevity… without AI doing the work for us, is at most a rounding error. And ASI that was even close to aligned, that found it worth reserving even a billionth part of the value of the universe for humans, would treat this as the obvious most urgent problem and solve death pretty much if there’s any physically possible way of doing so. And when I look inside, I find that I simply don’t care about a glorious transhumanist future that doesn’t include me or any of the particular other humans I care about. I do somewhat prefer being kind / helpful / benificent to people I’ve never met, very slightly prefer that even for people who don’t exist yet, but it’s far too weak a preference to trade off against any noticeable change to the odds of me and everyone I care about dying. If that makes me a “sociopath” in the view of someone or other, oh well.
I’ve been a supporter of MIRI, AI alignment, etc. for a long time, not because I share that much with EA in terms of values, but because the path to the future having any value has seemed for a long time to route through our building aligned ASI, which I consider as hard as MIRI does. But when the “pivotal act” framing started being discussed, rather than actually aligning ASI, I noticed a crack developing between my values and MIRI’s, and the past year with advocacy for “shut it all down” and so on has blown that crack wide open. I no longer feel like a future I value has any group trying to pursue it. Everyone outside of AI alignment is either just confused and flailing around with unpredictable effects, or is badly mistaken and actively pushing towards turning us all into paperclips, but those in AI alignment are either extremely unrealistically optimistic about plans that I’m pretty sure, for reasons that MIRI has argued, won’t work; or, like current MIRI, they say things like that I should stake my personal presence in the glorious transhumanist future on cryonics (and what of my friends and family members who I could never convince to sign up? What of the fact that, IMO, current cryonics practice probably doesn’t even prevent info-theoretical death, let alone give one a good shot at actually being revived at some point in the future?)
I happen to also think that most plans for preventing ASI from happening soon, that aren’t “shut it all down” in a very indiscriminate way, just won’t work—that is, I think we’ll get ASI (and probably all die) pretty soon anyway. And I think “shut it all down” is very unlikely to be societally selected as our plan for how to deal with AI in the near term, let alone effectively implemented. There are forms of certain actors choosing to go slower on their paths to ASI that I would support, but only if those actors are doing that specifically to attempt to solve alignment before ASI, and only if it won’t slow them down so much that someone else just makes unaligned ASI first anyway. And of course we should forcibly stop anyone who is on the path to making ASI without even trying to align it (because they’re mistaken about the default result of building ASI without aligning it, or because they think humanity’s extinction is good actually), although I’m not sure how capable we are of stopping them. But I want an organization that is facing up to the real, tremendous difficulty of making the first ASI aligned, and trying to do that anyway, because no other option actually has a result that they (or I) find acceptable. (By the way, MIRI is right that “do your alignment homework for you” is probably the literal worst possible task to give to one’s newly developed AGI, so e.g. OpenAI’s alignment plan seems deeply delusional to me and thus OpenAI is not the org for which I’m looking.)
I’d like someone from MIRI to read this. If no one replies here, I may send them a copy, or something based on this.
Thank you for writing this. I usually struggle to find resonating thoughts, but this indeed resonates. Not all of it, but many key points have a reflection that I’m going to share:
Biological immortality (radical life extension) without ASI (and reasonably soon) looks hardly achievable. It’s a difficult topic, but for me even Michael Levin’s talks are not inspiring enough. (I would rather prefer to become a substrate-independent mind, but, again, imagine all the R&D without substantial super-human help.)
I’m a rational egoist (more or less), so I want to see the future and have pretty much nothing to say about the world without me. Enjoying not being alone on the planet is just a personal preference. (I mean, the origin system is good, nice planets and stuff, but what if I want to GTFO?) Also, I don’t trust imaginary agents (gods, evolution, future generations, AGIs), however creating some of them may be rational.
Let’s say that early Yudkowsky has influenced my transhumanist views. To be honest, I feel somewhat betrayed. Here my position is close to what Max More says. Basically, I value the opportunities, even if I don’t like all the risks.
I agree that AI progress is really hard to stop. The scaling leaves possible algorithmic breakthroughs underexplored. There is so much to be done, I believe. The tech world will still be working on it even with mediocre hardware. So we are going to ASI anyway.
And all the alignment plans… Well, yes, they tend to be questionable. For me, creating human-like agency in AI (to negotiate with) is more about capabilities, but that’s a different story.
I respectfully disagree on the first point. I am a doctor myself and given observable increase in investment in life extension (largely in well funded stealth startups or Google Calico), I have ~70% confidence that in the absence of superhuman AGI or other x-risks in the near term, we have a shot at getting to longevity escape velocity in 20 years.
While my p(doom) for AGI is about 30% now, down from a peak of 70% maybe 2 years ago after the demonstration that it didn’t take complex or abstruse techniques to reasonably align our best AI (LLMs), I can’t fully endorse acceleration on that front because I expect the tradeoff in life expectancy to be net negative.
YMMV, it’s not like I’m overly confident myself at 70% for life expectancy being uncapped, and it’s not like we’re probably going to find out either. It just doesn’t look like a fundamentally intractable problem in isolation.
I also see multiple technological pathways that would get us to longevity escape velocity that seem plausible without AGI in that timeframe.
If nothing else, with advances in tissue engineering I expect we will be able to regrow and replace every organ in the body except the brain by mid-century.
But I also think a lot of what’s needed is culturally/politically/legally fraught in various ways. I think if we don’t get life-extending tech, it will be because we made rules inadvertently preventing it or pretending it’s better not to have it.
Is the claim here a 70% chance of longevity escape velocity by 2043? It’s a bit hard to parse.
If that is indeed the claim, I find it very surprising, and I’m curious about what evidence you’re using to make that claim? Also, is that LEV for like, a billionaire, a middle class person in a developed nation, or everyone?
Yes, you can reformat it in that form if you prefer.
This is a gestalt impression based off my best impressions of the pace of ongoing research (significantly ramped up compared to where investment was 20 years ago), human neurology, synthetic organs and finally non-biological alternatives like cybernetic enhancement. I will emphasize that LEV != actual biological immortality, but it leads to at least a cure for aging if nothing else.
Aging, while complicated and likely multifactorial, doesn’t seem intractable to analysis or mitigation. We have independent research projects tackling individual aspects, but as I’ve stated, most of them are in stealth mode even if they’re well-funded, and solving any individual mechanism is insufficient because of how aging itself is an exponential process.
To help, I’m going to tackle the top causes of aging in the West-
Heart disease- This is highly amenable to outright replacement of the organ, be it with a cybernetic replacement or one grown in-vitro. Obesity, which contributes heavily to cardiovascular disease and morbidity, is already being tackled by the discovery of GLP-1 antagonists like semaglutide, and I fully expect that the obesity epidemic that is dragging down life expectancy in the West will be over well before then.
Cancer- Another reason for optimism, CAR-T therapy is incredibly promising, as are other targeted therapies. So are vaccines for diseases like HPV that themselves cause cancer (said vaccine already exists, I’m talking more generally).
Unintentional injuries- The world has grown grossly safer, and only will continue to do so, especially as things get more automated.
Respiratory diseases- Once again, reason for optimism that biological replacements will be cheap enough that we won’t have to rely on limited numbers of donors for transplants.
Stroke and cerebrovascular disease- I’ll discuss the brain separately, but while this is a harder subject to tackle, mitigating obesity helps immensely.
Alzheimers- Same disclaimer as above
Diabetes- Our insulin pumps and formulations only get better and cheaper, and many of the drawbacks of artificial insulin supplementation will vanish (our pancreas is currently better at quickly and responsively adjusting blood sugar levels by releasing insulin than we are). Once again, a target for outright replacement of the organ.
These are ranked in descending order.
The brain remains incredibly difficult to regenerate, so if we run into something intractable to the hypothetical capabilities 20 years hence, this will likely be the biggest hurdle. Even then, I’m cautiously optimistic we’ll figure something out, or reduce the incidence of dementia.
Beyond organic replacement, I’m bullish on gene therapy, most hereditary disease will be eliminated, and eventually somatic gene therapy will be able to work on the scale of the entire body, and I would be highly surprised if this wasn’t possible in 20 years.
I expect regenerative medicine to be widely available, beyond our current limited attempts at arresting the progression of illness or settling for replacements from human donors. There’s a grab bag of individual therapies like thymic replacement that I won’t get into.
As for the costs associated with this, I claim no particular expertise, but in general, most such treatments are amenable to economies of scale, and I don’t expect them to remain out of reach for long. Organ replacement will likely get a lot cheaper once they’re being vat grown, and I put a decent amount of probability that ~universally acceptable organs can be created by careful management of the expression of HLA antigens such that they’re unlikely to be rejected outright. Worst case, patient tissue such as pluripotent stem cells will be used to fill out inert scaffolding like we do today.
As a doctor, I can clearly see the premium people put on any additional extension of their lives when mortality is staring them in the face, and while price will likely be prohibitive for getting everyone on the globe to avail of such options, I expect even middle class Westerners with insurance to be able to keep up.
Like I said, this is a gestalt impression of a very broad field, and 70% isn’t an immense declaration of confidence. Besides, it’s mostly moot in the first place, we’re very likely certainly getting AGI of some form by 2043.
To further put numbers on it, I think that in a world where AI is arrested at a level not significantly higher than GPT-4, I, being under the age of 30, have a ~80% chance of making it to LEV in my lifespan, with an approximately 5% drop for every additional decade older you are at the present.
You, being a relatively wealthy person in a modernized country? Do you think you’ll be able to afford the LEV by that time, or only that some of the wealthiest people will?
I’m a doctor in India right now, and will likely be a doctor in the UK by then, assuming I’m not economically obsolete. And yes, I expect that if we do have therapies that help provide LEV, they will be affordable in my specific circumstances as well as most LW readers, if not globally. UK doctors are far poorer compared to the their US kin.
Most biological therapies are relatively amenable to economies of scale, and while there are others that might be too bespoke to manage the same, that won’t last indefinitely. I can’t imagine anything with as much demand as a therapy that is proven to delay aging nigh indefinitely, for an illustrative example look at what Ozempic and Co are achieving already, every pharma industry leader and their dog wants to get in on the action, and the prices will keep dropping for a good while.
It might even make economic sense for countries to subsidize the treatment (IIRC, it wouldn’t take much more for GLP-1 drugs to reach the point where they’re a net savings for insurers or governments in terms of reducing obesity related health expenditures). After all, aging is why we end up succumbing to so many diseases in our senescence, not the reverse.
Specifically, gene therapy will likely be the best bet for scaling, if a simple drug doesn’t come about (seems unlikely to me, I doubt there’s such low hanging fruit, even if the net result of LEV might rely on multiple different treatments in parallel with none achieving it by themself).
Thanks for your reply. “70% confidence that… we have a shot” is slightly ambiguous—I’d say that most shots one has are missed, but I’m guessing that isn’t what you meant, and that you instead meant 70% chance of success.
70% feels way too high to me, but I do find it quite plausible that calling it a rounding error is wrong. However, with a 20 year timeline, a lot of people I care about will almost definitely still die, who could have not died if death were Solved, which group with very much not negligible probability includes myself. And as you note downthread, the brain is a really deep problem with prosaic life extension. Overall I don’t see how anything along these lines can be fast enough and certain enough to be a crux on AI for me, but I’m glad people are working on it more than is immediately apparent to the casual observer. (I’m a type 1 diabetic and would have died at 8 years old if I’d lived before insulin was discovered and made medically available, so the value of prosaic life extension is very much not lost on me.)
T1DM is a nasty disease, and much like you, I’m more than glad to live in the present day when we have tools to tackle it, even if other diseases still persist. There’s no other time I’d rather be alive, even if I die soon, it’s going to be interesting, and we’ll either solve ~all our problems or die trying.
I understand. My mother has chronic liver disease, and my grandpa is 95 years old, even if he’s healthy for his age (a low bar!). In the former case, I think she has a decent chance of making it to 2043 in the absence of a Singularity, even if it’s not as high as I would like. As for my grandfather, at that age just living to see the next birthday quickly becomes something you can’t take for granted. I certainly cherish all the time I can spend him with him, and hope it all goes favorably for us all.
As for me, I went from envying the very young, because I thought they were shoe-ins for making it to biological immortality, to pitying them more these days, because they haven’t had at least the quarter decade of life I’ve had in the event AGI turns out malign.
Hey, at least I’m glad we’re not in the Worst Possible Timeline, given that awareness of AI x-risk has gone mainstream. That has to count for something.
P.S. Having this set of values and beliefs is very hard on one’s epistemics. I think it’s a writ-large version of what Eliezer has stated as “thinking about AI timelines is bad for one’s epistemics”. Here are some examples:
(1) Although I’ve never been at all tempted by e/acc techno-optimism (on this topic specifically) / alignment isn’t a problem at all / alignment by default, boy, it sure would be nice to hear about a strategy for alignment that didn’t sound almost definitely doomed for one reason or another. Even though Eliezer can (accurately, IMO) shoot down a couple of new alignment strategies before getting out of bed in the morning. So far I’ve never found myself actually doing it, but it’s impossible not to notice that if I just weren’t as good at finding problems or as willing to acknowledge problems found by others, then some alignment strategies I’ve seen might have looked non-doomed, at least at first...
(2) I don’t expect any kind of deliberate slowdown of making AGI to be all that effective even on its own terms, with the single exception of indiscriminate “tear it all down”, which I think is unlikely to get within the Overton window, at least in a robust way that would stop development even in countries that don’t agree (forcing someone to sabotage / invade / bomb them). Although such actions might buy us a few years, it seems overdetermined to me that they still leave us doomed, and in fact they appear to cut away some of the actually-helpful options that might otherwise be available (the current crop of companies attempting to develop AGI definitely aren’t the least concerned with existential risk of all actors who’d develop AGI if they could, for one thing). Compute thresholds of any kind, in particular, I expect to lead to much greater focus on doing more with the same compute resources rather than doing more by using more compute resources, and I expect there’s a lot of low-hanging fruit there since that isn’t where people have been focusing, and that the thresholds would need to decrease very much very fast to actually prevent AGI, and decreasing the thresholds below the power of a 2023 gaming rig is untenable. I’m not aware of any place in this argument where I’m allowing “if deliberate slowdowns were effective on their own terms, I’d still consider the result very bad” to bias my judgment. But is it? I can’t really prove it isn’t...
(3) The “pivotal act” framing seems unhelpful to me. It seems strongly impossible to me for humans to make an AI that’s able to pass strawberry alignment that has so little understanding of agency that it couldn’t, if it wanted to, seize control of the world. (That kind of AI is probably logically possible, but I don’t think humans have any real possibility of building one.) An AI that can’t even pass strawberry alignment clearly can’t be safely handed “melt all the GPUs” or any other task that requires strongly superhuman capabilities (and if “melt all the GPUs” were a good idea, and it didn’t require strongly superhuman capabilities, then people should just directly do that). So, it seems to me that the only good result that could come from aiming for a pivotal act would be that the ASI you’re using to execute it is actually aligned with humans and “goes rogue” to implement our glorious transhuman future; and it seems to me that if that’s what you want, it would be better to aim for that directly rather than trying to fit it through this weirdly-shaped “pivotal act” hole.
But… if this is wrong, and a narrow AGI could safely do a pivotal act, I’d very likely consider the resulting world very bad anyway, because we’d be in a world where unaligned ASI has been reliably prevented from coming into existence, and if the way that was done wasn’t by already having aligned ASI, then by far the obvious way for that to happen is to reliably prevent any ASI from coming into existence. But IMO we need aligned ASI to solve death. Does any of that affect how compelling I find the case for narrow pivotal-act AI on its own terms? Who knows...
Apparent moral alien here. Hi! I’m pretty astonished how many people apparently believe something like that statement. In my world, it’s an off-the-deep-end perverse and sociopathic sort of value to hold. Of course, it’s consistent. I can work with sociopaths, because to a first approximation I can work with anyone. But if there’s enough delay, cooperation is going to become strained, because I’m quite happy with me dying, and you dying, and your loved ones dying (ideally painlessly having lived fulfilling lives!), if it turns out to be necessary in order for there to be a valuable future at all.
Despite introducing myself as an alien, I currently think most humans don’t espouse your statement, because most humans’ moral circle is big enough to preclude it. Other humans reject it on the basis of philosophical open/empty individualism. Others reject it for superstitious reasons or due to ethical injunctions. Others still for the deceptive reasons you mentioned. There are basically a lot of reasons either not buy it, or be quiet about it if you do!
As an aside, it appears that in certain circles, the deceptive/prudent reasons you mention to not espouse this view are inverted: a sort of neoliberal/objectivist-celebratory crowd would tend to praise this sort of thing.
A separate aside: there is a strong implicit assumption that AGI is a necessary prerequisite to functional immortality, which isn’t at all obvious to me. I think it’s an insufficiently-justified received wisdom from the days of sloppier, lower-stakes futurism.
Finally, I’m much less confident than you that a locked-in pause is likely happening!
I thought your analysis was generally pretty good otherwise.
Many humans, given a choice between
A) they and their loved ones (actually everyone on earth) will live forever with an X-risk p
B) this happens after they and everyone they love dies with an X-risk less than p
Would choose A.
Abortion has a sort of similar parallel but with economic risk instead of X risk, and obviously no immortality yet many are pro choice.
I think valuing the lives of future humans you don’t know of over the lives of yourselves and your loved ones is the alien choice here.
I think any human with time-consistent preferences prefers A to B for some margin? The question is how much margin.
Don’t understand your abortion analogy at all I’m afraid.
I did introduce myself as ‘apparent moral alien’! - though for the reasons in the rest of my comment I don’t think I’m all that rare. Until quite recently I’d have been confident I was in a supermajority, but I’m less sure of that now, and I weakly entertain the hypothesis it’s a minority.
You are ending a potential future life in exchange for preserving a quality of present life.
Yet human routinely sacrifice their own lives for the good of other (see : firefighters, soldiers, high mountain emergency rescuers, etc.). The X-risk argument is more abstract but basically the same.
A lot of our moral frameworks breakdown once immortality is a real choice. Sacrificing your own life for the own good can be reframed as dying a little earlier.
Many of these people go in knowing there’s a small chance of death. A lot of them would probably change their minds if it was a suicide mission (except a minority).
If the 5 year mortality rate of firefighting was 100%, how many would still do that job?
It’s good of you to say that so plainly. I’ll return the favor and say that I’d run a substantial risk of death if it meant getting rid of you and people like you, to make the world a safer place for my loved ones.
Are you aware I’m a full time AI safety researcher? I don’t think you want to ‘get rid’ of us. Perhaps if you could politically silence me or intimidate me while having me carry on technical work? Naturally I don’t endorse this plan even for highly fanatical egoists.
Separately, even if I were just my political influence (along with my reference class), I (tentatively) don’t believe you’re as fanatical as your comment claims.
I kind of agree with the statement, but I don’t quite agree with the implications: my timelines are already quite short and I think that if we contract things more, we pay a high cost in risk for little benifit unless you expect yourself and many of your loved ones to die before then. In which case, what’s the point of discussing it?
Now if timelines were long, then yeah, it would make sense to accelerate AI insofar as it contributed to the continued surival of me and my loved ones. That could be through creating a FAI, or it could be via improved longevity tech, better disease treatment, improved cryonics, mind uploading etc. Which many people in this community are quite in favour of, even now.
I think you underestimate how weird people find the desire for immortality. Why does pretty much no-one sign up for cryonics? Because it is unlikely to work? Nah, the odd’s are maybe a few percent for current tech. Plenty of people should be willing to take a bet for continued life at those odds. Is it because of cost? Again, no. Life insurance isn’t that expensive. The real reason is that it is weird.
It’s weird because it’s not what anyone else thinks of immortality. Freezing your corpse isn’t the same as being young forever in the present.
In reality, humans have been pondering immortality since time immemorial. A lot of our myths involve immortals and some humans becoming immortals. A lot of our religions assume there’s an immortal afterlife.
I wrote about this in Appendix A of this post.
------
One might look at the rough 50⁄50 chance at immortality given surviving AGI and think “Wow, I should really speed up AGI so I can make it in time!”. But the action space is more something like:
Work on AI safety (transfers probability mass from “die from AGI” to “survive AGI”)
The amount of probability transferred is probably at least a few microdooms per person.
Live healthy and don’t do dangerous things (transfers probability mass from “die before AGI” to “survive until AGI”)
Intuitively, I’m guessing one can transfer around 1 percentage point of probability by doing this.
Do nothing (leaves probability distribution the same)
Preserve your brain if you die before AGI (kind of transfers probability mass from “die before AGI” to “survive until AGI”)
This is a weird edge case in the model and it conditions on various beliefs about preservation technology and whether being “revived” is possible
Delay AGI (transfers probability from “die from AGI” to “survive AGI” and from “survive until AGI” to “die before AGI”)
Accelerate AGI (transfer probability mass from “survive AGI” to “die from AGI” and from “die before AGI” to “survive until AGI”)
I think working on AI safety and healthy living seem like a much better choice than accelerating AI. I’d guess this is true even for a vast majority of purely selfish people.
For altruistic people, working on AI safety clearly trumps any other action in this space as it has huge positive externalities. This is true for people who only care about current human lives (as one microdoom ≈ 8,000 current human lives saved), and it’s especially true for people who place value on future lives as well (as one microdoom = one millionth of the value of the entire long term future).
This is a very simplified view of what it means to accelerate or delay AGI, which ignores that there are different ways to shift AGI timelines that transfer probability mass differently. In this model I assume that as timelines get longer, our probability of surviving AGI increases monotonically, but there are various considerations that make this assumption shaky and not generally true for every possible way to shift timelines (such as overhangs, different actors being able to catch up to top labs, etc.)
For most people in their 20s or 30s, it is quite unlikely (around 10%) that they die before AGI. And if you basically place any value on the lives of people other than yourself, then the positive externalities of working on AI safety probably strongly outweigh anything else you could be doing.
Acceleration probably only makes sense for people who are (1) extremely selfish (value their life more than everything else combined) and (2) likely to die before AGI unless it’s accelerated.
“10% is overconfident”, given huge uncertainty over AGI takeoff (especially the geopolitical landscape of it), and especially given the probability that AGI development may be somehow slowed (https://twitter.com/jachaseyoung/status/1723325057056010680 )
Most longevity researchers will still be super-skeptical if you say AGI is going to solve LEV in our lifetimes (one could say—a la Structure of Scientific Revolutions logic—that most of them have a blindspot for recent AGI progress—but AGI=>LEV is still handwavy logic)
Last year’s developments were fast enough for me to be somewhat more relaxed on this issue… (however, there is still slowing core aging rate/neuroplasticity loss down, which acts on shorter timelines, and still important if you want to do your best work)
https://twitter.com/search?q=from%3A%40RokoMijic%20immortality&src=typed_query
I don’t know whether to believe, but it’s a reasonable take...
Two days ago I wrote a tweet with similar idea and it get unexpected boost—which means that it is important thing for people: ’Perhaps, we don’t need superintelligent AI to solve aging quickly. Some more mundane level of AI will be enough.” https://twitter.com/turchin/status/1719315694813130896
EY wrote against the statement, saying that why not all these people not signing for cryonics
M. Batin wrote in favor of this statement and even made a counter-appearance during AI Pause meeting in SF https://twitter.com/MikhailBatin/status/1719948531233140754
I also agree with the statement. I’m guessing most people who haven’t been sold on longtermism would too.
When people say things like “even a 1% chance of existential risk is unacceptable”, they are clearly valuing the long term future of humanity a lot more than they are valuing the individual people alive right now (assuming that the 99% in that scenario above is AGI going well & bringing huge benefits).
Related question: You can push a button that will, with probability P, cure aging and make all current humans immortal. But with probability 1-P, all humans die. How high does P have to be before you push? I suspect that answers to this question are highly correlated with AI caution/accelerationsim
If I choose P=1, then 1-P=0, so I am immortal and nobody dies
Well sure, but the interesting question is the minimum value of P at which you’d still push
I think the point of the statement is: wait until the probability of you dying before you next get an opportunity to push the button is > 1-P then push the button.
You have a very strange understanding of politics, wherein you have laymen who want to advance their interests at the expense of other peoples, who realize that would be unpopular if they stated exactly what they were doing, and then as a consequence lie on Twitter about needing to do it for a different reason. This is insufficiently cynical. People almost always genuinely think their political platform is the Right and Just one. It’s just that people also have strong instincts to align themselves with whatever tribe is saying they’re the Good Guys, and that giving the Good Guys money is going to be great for the economy, and that the Bad Guys deserve to get their money taken away from them, the greedy pricks.
So the tradeoff you mention in your problem statement is simply too abstract for these instincts to kick in, even if it were real. The instincts kick in for Yann Lecun or Marc Andreesen, because their hindbrains don’t have to spend any extra time to see the obvious point that they lose near term prestige if AI research is considered evil. But I doubt either of them are losing sleep over this because they’re making a calculation about how likely it is they’ll be immortal; that’s the sort of high-concept tradeoff that people simply don’t organize their entire politics around, and just end up finding cartoon-villanous.
You are right, there are three possible avenues of approaching this: (1) people have certain goals and lie about them to advance their interests, (2) people have certain goals, and they self-delude about their true content so that they advance their interests, (3) people don’t have any goals, they are simply executing certain heuristics that proved to be useful in-distribution (Reward is not an optimisation target approach), I omitted the last one from the post. But think that my observation about (2) having non-zero chance of explaining variance in opinions still stands true. And this is even more true for people engaged in AI safety, such as members of Pause AI, e/acc and (to a lesser extent) academics doing research on AI.
Even if (3) has more explanatory power, it doesn’t really defat the central point of the post, which is the ought question (which is a bit of a evasive answer, I admit).
My limited impression of “e/accs”, and you may think this is unfair, is that most of them seem not to have any gears-level model of the problem at all, and have instead claimed the mantle because they decided amongst each other it’s the attire of futurism and libertarianism. George Hotz will show up to the Eliezer/Leahy debates with a giant American flag in the background and blurt out stuff like “Somalia is my preferred country”, not because he’s actually going to live there, but because he thinks that sounds based and the point of the discussion for him is to wave a jersey in the air. I don’t think Hotz has made the expected value calculation you mention because I don’t think he’s even really gotten to the point of developing an inside view in the first place.
In other words, they are based-tuned stochastic parrots? Seems harsh, but the Hotz-Yudkowsky ‘debate’ can only be explained by something in the vicinity of this hypothesis AFAICT (haven’t seen others).
Based on personal experience, you are definitely not the only one thinking about that Statement.
The statement seems like it’s assuming:
we know roughly how to build AGI
we decide when to do that
we use the time between now and then to increase chance of successful alignment
if we succeed in alignment early enough, you and your loved ones won’t die
I don’t think any of these are necessarily true, and I think the ways they are false is asymmetric in a manner that favors caution
It’s also assuming:
We know roughly how to achieve immortality
We can do that exactly in the window of “the last possible moment” of AGI.
Efforts between immortality and AGI are fungible and exclusive, or at least related in some way.
Ok, yeah—we have to succeed on BOTH alignment and immortality to keep any of us from dying.
3 and 4 are, I think, the point of the post. To the extent that work on immortality rather than alignment, we narrow the window of #2, and risk getting neither.
Isn’t the assumption that once we successfully align AGI, it can do the work on immortality? So “we” don’t need to know how beyond that.
I think most humans agree with this statement in an “I emotionally want this” sort of way. The want has been sublimated via religion or other “immortality projects” (see The Denial of Death). The question is, why is it taboo, and it is taboo in the sense you say? (a signal of low status)
I think these elements are at play most in peoples mind, from lay people to rationalists:
It’s too weird to think about: Considering the possibility of a strange AI-powered world where either complete extinction or immortality are possible feels “unreal”. Our instinct that everything that happens in the world is within an order of magnitude of “normal” directly opposes being able to believe in this. As a result, x/s-risk discussions, either due to personal imagination or optics reasons, are limited to natural extrapolations of things that have occurred in history (eg. biological attacks, disinformation, weapons systems, etc.). It’s too bizarre to even reckon that there is a non-zero chance immortality via any conduit is possible. This also plays into the low status factor: weird, outlandish opinions on the future not validated by a high-status figure are almost always met with resistance
The fear of “missing out” leads to people not even wanting to think about it seriously at all: People don’t want give higher credence to hypotheticals that increase the scale of their losses. If we think death is the end for everyone, it doesn’t seem so bad to imagine. If we think that we will be the ones to maybe die and others won’t, or that recent or past loved ones are truly gone forever in a way not unique to humankind, it feels unfair/insulting by the universe.
Taking it seriously would massively change one’s priorities in life and upset the equilibrium of their current value structures: As in: one would do everything they could to minimize the risk of early death. if they believe immortality could be possible in 20 years or less, their need for long term planning is reduced, as immortality would also imply post-scarcity, so their assiduous saving, sacrifices for their children, future are worthless. That cognitive dissonance does not sit well in the mind and hinders one’s individual agentic efficiency.
I have expressed versions of this Statement, or at least parts of it, to those close to me. They are not in the EA or LW communities and so I have to rephrase a lot. Mostly along the lines of, “One way or another, the last generation to die of old age has most likely already been born. I’m not quite sure which one, but if we manage to survive the century as a species, then I expect it’s my own, and that my nieces and nephew will be able to live for millennia. I hope I’m wrong about which generation it is.”
For me, both AI and medicine are far outside my areas of expertise, so I’m focusing on other problems that I have a better chance of influencing, hence the passive framing. This is even more true for my family members.
Off-topic, but somewhat related. I want to know if there is anyone reading these words who is willing to admit that he or she is kind of hoping humanity will go extinct because humanity has been unfair to him or her or because (for some other reason) humanity is bad or unworthy.
Honestly, I haven’t seen much about individual biological immortality, or even significant life-extension, in the last few years.
I suspect progress on computational consciousness-like mechanisms has fully eclipsed the idea that biological brains in the current iteration are the way of the future. And there’s been roughly no progress on upload, so the topic of immortality for currently-existing humans has mostly fallen away.
Also, if/when AI is vastly more effective than biological intelligence, it takes a lot of the ego-drive away for the losers.
Who said anything about slowly and painfully?? I don’t think a fully reflective you would endorse that.
(FWIW I’m not whoever downvoted your comments, though these definitely aren’t to my taste and could be interpreted as threatening)
Me. And I don’t actually endorse that. That was my point.
And my comments are threatening? Saying I’d defend with my life my loved ones from those (like you) who are happy for them to die to achieve their goals? Sure, I guess, but I stand by that.
I’m very much of an egoist but I don’t really agree with this statement. Even if the last smidge of my altruism were deleted, I still wouldn’t rob banks or run a fraudulent asset exchange because I have a very cushy position in society as it is and cooperating with humanity in fact makes my life much better than not. I similarly wouldn’t do existentially crazy things to try to reach the singularity for me personally. Even if normal people wouldn’t understand how I’m burning the commons, you guys would.
I’m like way more likely to do norm-breaking stuff (like sell unlicensed securities) for altruistic reasons, though it is still usually a bad idea.
--
I really do feel it’s a bummer that I don’t get to be part of the Future. It really does suck. The way I cope is by trying to make the present a little bit more future (in a way that isn’t existentially threatening). I’ll get to see a few cool things this way at least.
I think a lot of people haven’t fully considered how amazing immortality is. Your 7th observation is probably very accurate too. Socially sophisticated people want others to believe they’re fighting for the greater good rather than their own self interest. That doesn’t mean they’re necessarily lying.
I fully agree with the Statement but also support Pause/Stop. The reason is that I’m highly confident that everyone dies if we achieve AGI in the near future, so since I’m still relatively young I prefer getting to live some more years instead.
Not much to add content wise but just to be one data point: I strongly agree and have talked about it with exactly one person. I have no plans to advance the position openly and proactively, largely for reasons stated, but would quietly and strongly support efforts to that end should I become aware of them.
As I’ve put it to my friend: the worst day for a transcendent AI to emerge is the week after I would die and the best day is the week before. We then argue over where “tomorrow” fits on that spectrum
I’m very glad that you’re raising this topic for discussion. I’ve recently been thinking about the same thing myself. If I could delay the advent of AGI/TAI by, say, 100 years, would I? There are at least two relevant considerations here: 1) the commonly held (but not, to me, obviously true) assumption that delaying AGI/TAI increases the probability of it being safely deployed and 2) the costs of delaying AGI/TAI.
(2) motivates thinking really hard about whether (1) is true. General concern about AI safety also motivates thinking hard about whether (1) is true, since if delaying AGI/TAI does not increase the probability of a safe deployment, then we should think about what would increase it.
You should consider cross-posting to EAF. Would be interesting to see the poll differences, but might be fiddly to set up.
If I take the statement face value as written, s-risks are very relevant to disagreement—a life lived longer without solving sources of suffering tends to result in accruing damage, and saving my life immediately before I would die is one of the more likely ways to force me to live with the maximum possible damage dealt by (what just nearly killed me, and cumulative health troubles in general).
I notice that “don’t die” is rarely a terminal value. Most people who say they want immortality are making some assumptions about the quality of life it would entail—both in an s-risk sense and in a “technically people who are permanently comatose on life support aren’t actually dead” sense as well.
I wouldn’t call “don’t die” a terminal value in the cryonics case either, because in that case you do die for awhile but then you get better. Eventual reincarnation after a delay seems meaningfully distinct from immortality, because immortality implies continuity of life.
To be naively utilitarian, this question is reducible to following expression:
shouldBeAMoronAndAccelerate = ( pda * (ud + gud / sf) + pia * ui + (1 - pda—pia) * ud > pds * (ud + gud / sf) + pis * ui + (1 - pds—pis) * ud )
where:
pda: Probability of doom if you ignore safety and accelerate
pds: Probability of doom if you pay attention to safety
pia: Probability of achieving immortality if you accelerate
pis: Probability of achieving immortality of you play safe
ud: “Selfish” utility gain if you die (very negative)
gud: “Common good” utility gain if everyone dies (even more negative)
ui: “Selfish” utility gain if you achieve immortality (positive)
sf: “Selfish factor”—how much “Common good” utility you’re willing to sacrifice per unit “Selfish” utility.
(you can also add terms for living happily ever after without immortality and stuff like that, but utility gain in this cases is negligible in comparison)
The main problem I see is that pia < pis.
In other words, no matter how high your “Selfish factor” is—accelerating is equivalent to shooting yourself in a leg. Also you die. Also everyone dies.