Harry was wondering if he could even get a Bayesian calculation out of this. Of course, the point of a subjective Bayesian calculation wasn’t that, after you made up a bunch of numbers, multiplying them out would give you an exactly right answer. The real point was that the process of making up numbers would force you to tally all the relevant facts and weigh all the relative probabilities. Like realizing, as soon as you actually thought about the probability of the Dark Mark not-fading if You-Know-Who was dead, that the probability wasn’t low enough for the observation to count as strong evidence.
− HPMOR Chapter 86: Multiple Hypothesis Testing
I’m 28 years old and have never had a drivers license. At some point earlier on in my life I decided that driving is something that has a bad expected value (EV) due to the risk of death and the massive value of life, but at the same time, the EV isn’t so bad where I will go out of my way to avoid it. Since deciding this, the belief has become cached. However, various things have recently prompted me to reconsider the belief.
I’m looking to buy a home and am leaning towards making it a requirement that the place is fully walkable (as opposed to mostly walkable but sometimes requiring a car).
The topic of MIRI relocating came up and the badness of cars is relevant to that.
I have a wedding coming up where I have to travel from Vegas to SF. I have the option of getting a ride from my girlfriend’s mom, or taking a flight. I’m tempted to go by car so I don’t have to pay the money for a flight, but I don’t actually think that is the right decision. (“No thanks Sandra. I think that computers are going to take over the world and make us all immortal. You have a slightly higher chance of dying in a car than a plane, so I’d rather pay for a flight.”)
Covid has prompted me to explore the EV of doing things. Eg. looking at the chances of catching covid and dying, trying to put a dollar amount on that, and then asking whether such an activity is worth it. In doing so for covid, it naturally leads to asking the question for activities such as driving as well.
Perhaps this is a good starting point. As I started to get into in my comment on the MIRI relocation post, in 2018, there were 11.18 deaths per 100k people in the US, or a 0.01118% chance of dying. If you value life at $10M, 0.0001118 * $10,000,000 = $1,118. Let’s ballpark it at $1,000/year.
But you’re a safer driver than average, right? I say that a little bit tongue in cheek because of the cognitive bias where 93% of people think they’re above average. But to be conservative in this analysis, let’s say you’re in the 95th percentile in driving ability/safety. How much does that reduce the risk?
This is a big unknown for me. I really hope it cuts it by a few orders of magnitude. On the one hand it seems plausible, because a lot of deaths happen due to things like driving drunk, driving drowsy, road rage, youth, elderliness, etc. Maybe if you drive really carefully under safe road conditions without any impairments, you can be 1000x safer than the baseline. On the other hand, after a cursory search, it looks like there’s roughly a 2.5-to-1 ratio of non-alcohol to alcohol related fatalities. But alcohol isn’t the only thing you’re avoiding by being in that 95th percentile. Maybe we can ballpark it and say that half of deaths are due to avoidable stuff. Being conservative, maybe we can add a little bit more buffer and say that you have 1⁄4 the risk of dying compared to baseline. Which brings us to $250 a year. And we’ll ignore the potential for injury, harming others directly, and harming others because people are devastated when people they love die or get hurt.
$250 sounds like a very reasonable price to pay for the convenience of being able to drive. But here’s the kicker: due to the potential for living eg. hundreds of thousands of years, perhaps life should be valued way more than the standard $10M. You can get to a valuation of $10M if you value a year at $200k and expect to live another 50 years. But what if you think there’s a 10% chance of living another 100k years? That means there’s an expectation of living roughly 10k years instead of 50 years. And those are hopefully going to be some pretty awesome years to be a part of.
I’m not an futurist or an AI researcher so I am not in the best position to estimate this. Fortunately for me, this community seems to have a lot of people who know about this stuff, so please let me know what you all think! In brief, here are my own thoughts.
The Wait But Why article on AI had some good commentary on what AI experts think. This is a long quote, but the length seems appropriate.
In 2013, Vincent C. Müller and Nick Bostrom conducted a survey that asked hundreds of AI experts at a series of conferences the following question: “For the purposes of this question, assume that human scientific activity continues without major negative disruption. By what year would you see a (10% / 50% / 90%) probability for such HLMI4 to exist?” It asked them to name an optimistic year (one in which they believe there’s a 10% chance we’ll have AGI), a realistic guess (a year they believe there’s a 50% chance of AGI—i.e. after that year they think it’s more likely than not that we’ll have AGI), and a safe guess (the earliest year by which they can say with 90% certainty we’ll have AGI). Gathered together as one data set, here were the results:2
Median optimistic year (10% likelihood): 2022
Median realistic year (50% likelihood): 2040
Median pessimistic year (90% likelihood): 2075
So the median participant thinks it’s more likely than not that we’ll have AGI 25 years from now. The 90% median answer of 2075 means that if you’re a teenager right now, the median respondent, along with over half of the group of AI experts, is almost certain AGI will happen within your lifetime.
A separate study, conducted recently by author James Barrat at Ben Goertzel’s annual AGI Conference, did away with percentages and simply asked when participants thought AGI would be achieved—by 2030, by 2050, by 2100, after 2100, or never. The results:3
By 2030: 42% of respondents
By 2050: 25%
By 2100: 20%
After 2100: 10%
Never: 2%
Pretty similar to Müller and Bostrom’s outcomes. In Barrat’s survey, over two thirds of participants believe AGI will be here by 2050 and a little less than half predict AGI within the next 15 years. Also striking is that only 2% of those surveyed don’t think AGI is part of our future.
But AGI isn’t the tripwire, ASI is. So when do the experts think we’ll reach ASI?
Müller and Bostrom also asked the experts how likely they think it is that we’ll reach ASI A) within two years of reaching AGI (i.e. an almost-immediate intelligence explosion), and B) within 30 years. The results:4
The median answer put a rapid (2 year) AGI → ASI transition at only a 10% likelihood, but a longer transition of 30 years or less at a 75% likelihood.
We don’t know from this data the length of this transition the median participant would have put at a 50% likelihood, but for ballpark purposes, based on the two answers above, let’s estimate that they’d have said 20 years. So the median opinion—the one right in the center of the world of AI experts—believes the most realistic guess for when we’ll hit the ASI tripwire is [the 2040 prediction for AGI + our estimated prediction of a 20-year transition from AGI to ASI] = 2060.
Of course, all of the above statistics are speculative, and they’re only representative of the center opinion of the AI expert community, but it tells us that a large portion of the people who know the most about this topic would agree that 2060 is a very reasonable estimate for the arrival of potentially world-altering ASI. Only 45 years from now.
Okay now how about the second part of the question above: When we hit the tripwire, which side of the beam will we fall to?
Superintelligence will yield tremendous power—the critical question for us is:
Who or what will be in control of that power, and what will their motivation be?
The answer to this will determine whether ASI is an unbelievably great development, an unfathomably terrible development, or something in between.
Of course, the expert community is again all over the board and in a heated debate about the answer to this question. Müller and Bostrom’s survey asked participants to assign a probability to the possible impacts AGI would have on humanity and found that the mean response was that there was a 52% chance that the outcome will be either good or extremely good and a 31% chance the outcome will be either bad or extremely bad. For a relatively neutral outcome, the mean probability was only 17%.
Let’s try to ballpark this. How much is a post-singularity year worth? We said $200k as a rough estimate for a 21st century year. The article said a 52% chance that the outcome is good, and a 31% chance it is bad. Suppose a good year is worth $500k and a bad year costs $500k, and otherwise it is $0. 0.52 * $500k + 0.31 * -$500k = $105k. Sounds fair enough to go with $100k. Although, my mind doesn’t want to go there, but we may be dealing with way higher magnitudes here. Eg. more along the lines of heaven and hell types of good/bad. In which case, if good and bad scale evenly, post-singularity life years become way more valuable, in expectation. But if badness scales faster than goodness, those post-singularity life years start to have a negative EV. But that is a hard attitude to adopt. “Humanity is doomed. I’m destined to die. May as well maximize the amount of fun I have now.” Actually, I suppose a lot of people have that attitude, but for a certain type of personality that I sense is common here, it seems like a hard thing to accept.
Anyway, darkness aside, let’s just go with $100k as the value of a post-singularity life year. As the article says:
a large portion of the people who know the most about this topic would agree that 2060 is a very reasonable estimate for the arrival of potentially world-altering ASI
This is another place where I could be wrong, but I would think that ASI basically implies that we would solve death. Right? I’m going to assume that. If I’m wrong, tell me in the comments.
The surveys seem pretty damn optimistic that ASI will happen at some point this century, but let’s be conservative and say that it only has a 10% chance of happening. That’s conservative, right?
How many years do you expect to live post-singularity? I would think it’d be a ton! Death is bad, ASI → we solved death, so an expectation of 100k years sounds totally plausible to me. A 10% chance of 100k years is an expectation of 10k years.10k years * $100k/year = $1B as the value of life. Which is 100x the $10M we used earlier, so the cost of $250/year becomes $25,000/year. And that probably crosses the line of “not worth it”.
However, there are a lot of places where my assumptions could be off by orders of magnitude. Maybe life expectancy post-singularity is only 1k years instead of 10k. That would bring the cost of driving way down back to “worth it” levels. On the other hand, I do feel like I’ve been pretty conservative in my assumptions, and it is plausible that there’s something like a 50% chance of me living to ASI, and life expectancy given ASI is something like 1M years. In that case, the value of life is something like 50% chance * 1M years * $100k/year = $50B, and thus the cost of driving for a given year is 0.001118 * $50B = $5,590,000. It sounds crazy to say that driving costs north of $5M a year in expectation, but these are crazy big/small numbers, and technological growth is exponential. Humans are known to have terrible intuitions for both of those things, so perhaps it isn’t worth putting too much weight into the common sensical idea that $5M a year is ludicrous.
Two things:
ONE: I love how “should I learn to drive for this trip right here?” cascades into this vast set of questions about possible future history, and AGI, and so on <3
Another great place for linking “right now practical” questions with “long term civilizational” questions is retirement. If you have no cached thoughts on retirement, you might profitably apply the same techniques used for car stuff to “being rich if or when the singularity happens” and see if either thought changes the other?
TWO: I used to think “I want to live this year”, “If I want to live in year Y then I will also want to live in year Y+1″. Then by induction: “I will want to live forever”.
However, then I noticed that this model wasn’t probabilistic, and was flinching from possible the deepest practical question in philosophy, which is “suicide”. Figuring out the causes and probabilities of people changing from “I do NOT want to kill myself in year Y” to “I DO want to kill myself in year Y+1” suggests a target for modeling? Which would end up probabilistic?
Occam (applied to modeling) says that the simplest possible model is univariate, so like maybe there is some value P which is the annual probability of “decaying into suicidalness that year”? I do mean decay here, sadly. Tragically, it looks to me like suicide goes up late in life… and also suicides might be hiding in “accidental car deaths” for insurance reasons? So maybe the right thing is not just a univariate model but a model where the probability goes up the older you get?
This approach, for me, put bounds on the value of my life (lowering the expected value of cryonics, for example) and caused me to be interested in authentic durable happiness, in general, in humans, and also a subject I invented for myself that I call “gerontopsychology” (then it turned out other people thought of the same coinage, but they aren’t focused on the generalizable causes of suicidal ideation among the elderly the way I am).
Ok three things...
THREE: I drive <3
Yeah, it is interesting isn’t it. Personally I actually would prefer if it were a more mundane decision, like whether or not I want to deal with the traffic or something :)
Wow, this is a very good point. Thank you! I thought about it briefly in the past, but it hadn’t hit me until now that similar logic of “small chance of moderate utility over an extremely long time horizon” might apply to things like “be rich when the singularity happens”. I always had imagined that singularity = amazingness forever for everyone, but I think my inner Professor Quirrell must have fell asleep. He’s awake now and is yelling at me.
Do you have any thoughts in particular on this? For me personally I plan on being rich anyway so that I can fund FAI research or something, but I still think it’s worth thinking about.
My thought here is that the post-singularity future will be a black swan, and that this natural decay you’re describing wouldn’t apply. It’ll be an amazing place to live where people won’t want to die. Hopefully.
I know there might be normal things about human psychology that would make people perhaps get bored and stop enjoying life at eg. their 10,000th birthday, but in a post-singularity world, it seems quite likely that we’d be able to overcome this. I guess that gets a little wire heading-y, but it’s not excessive to my taste.
Maybe I’m not thinking about this properly, but it seems to me like more of a yes-or-no sort of thing than a progressive decay sort of thing. Eg. it doesn’t get more likely that people will want to die as time goes on in the post-singularity world. It’s just a question of whether the post-singularity world has solved that problem or not.
I have a weird feeling you’d be interested in Michael Plant of the EA community’s work on “ordinary human unhappiness”. It’s always something I’ve thought about too as a worthwhile idea to pursue.
Haha, to each their own :)
Around half deaths from car accidents are pedestrians (may be less in US). By choosing not to drive, you increase the time of walking and your chances of being hit by other person’s car.
Other means of transport like cycling or buses are also risky.
Sitting home is even more dangerous as there are risks of depression and being overweight.
Finally, some cars are like two-three orders safer than others, if we look at the number of reported deaths per billion km driving. I saw once that Toyota Prius had 1 death for 1 billion km, but Kia Rio was only 1 for 10 millions. Also, there are special racing cars which are reinforced from inside and can roll safely
Wearing helmet inside a car is also useful.
Waiting few years for self-driving Tesla Cybertrack may be an option.
My impression is that if you don’t drive you wouldn’t be covering the same distances as you would in a car. Eg. if you walked 5 miles vs drove 5 miles, maybe the risk of being hit as a pedestrian starts to add up, but since in practice you’d only be walking a few blocks, the risk of death would be many times less. Similar point for biking.
Also, maybe this is overconfidence, but I feel like you have more control over your risk of death when walking or biking. Eg. you can be really careful about crossing the street, whereas on the road there’s only so much you can do. For example, if the crosswalk thing turns green but there is a car coming fast such that it’d have to stop in order to avoid hitting you as you cross the cross walk, I say you shouldn’t cross the crosswalk even though you’re “supposed” to. Maybe my memory is being overly dramatic, but I’m pretty sure I’d be dead right now if I didn’t adopt this policy.
That example in particular doesn’t control for Prius drivers potentially having a more cautious personality or something than Rio drivers. However, it is still quite telling. If cars really do differ this much in safety then that is a very important point. Do you or anyone else have more info about this? I feel some skepticism about it, but am ready to update!
In theory I think this is beyond the scope of the question, but in practice I think it’s a good point. It’s nice to know if driving might become something that is worth the risk in the future.
BTW, my personal choice is Uber Black.: I don’t have car and I delegate driving to special trained person. Every time I take Comfort, I regret, as I have near-miss accidents. It is relatively cheap in my area.
I have two-three people who I knew and who died in accidents: all of the were “reckless pedestrians”. It supports you point about the ability of pedestrians to manage risks.
Can’t find a link on statistic of accidents by car types
Thanks for the tip on Uber Black. Sorry to hear about the accidents.
I think making utility linear in years is a mistake. The remote possibility of finding a physics hack to control infinite matter in finite time does not curbstomp all other considerations, therefore the utility of that outcome is finite. I prefer 66% of BB(1000) years to 33% of BB(10000) years. I am uncertain about my preferences, but utility functions are not aggregated by taking the expectation.
The only authority on your preferences is yourself; but reasonable agents, when a hypothetical proves them dutch-bookable/incoherent, will become less certain about their preferences.
What this cashes out to is that you should calculate the value not of a year but of (an extra 1% chance of) making it to takeoff. (From what you would do in (perhaps physically impossible) hypotheticals.)
Descriptively I know that hyperbolic discounting is a thing. But prescriptively, it’s not clear to me that it should be. Do you mind elaborating on why you think it should be?
Also, I think this only really starts to change the conclusions we draw when the discounting is significant, eg. a post-singularity year going from $100k to $10k rather than $100k to $90k. It sounds like you are saying it should be significant. Is that true? Why is that?
Using numbers from this post: https://www.lesswrong.com/posts/ucjfY46L6qyXefvBT/quick-examination-of-miles-per-micromort-for-us-drivers-with
Assuming you’re in a passenger vehicle, without correcting for drunk drivers/unsafe drivers, you have 132 miles per micromort. Using the $10 million value for your life:
I think that means that driving 13.2 miles is equivalent to $1.
If you’re willing to categorize yourself in the least risky class in the linked post, that’s 54.8 miles for $1.
The other thing to consider is how much you care about other risks to your life. For instance, I’ve been living with (a limit of) 200 microcovid a week for the past year. I’m young and healthy, so let’s say that’s 2 micromorts. Given that amount, I should be comfortable driving (a maximum of) 264 miles each week, as they provide the same risk.
That makes enough sense, but the $10M valuation of life seems like it’s the cruxiest part of the question.
I’ve been thinking about this recently. I’ve been pretty risk averse when it comes to covid. There are probably things I do, like driving, that are riskier than the risks I avoid due to covid. You can say, “well you’re willing to eg. drive, so you should also be willing to take these covid associated risks”.
The issue with this logic is that revealed preferences aren’t real. Humans aren’t consistent and don’t always make good decisions. In your case, if the conclusion of this investigation into driving says “it’s a terrible idea!” and that conflicts with your microcovid budget, I think it’d make sense to re-examine your microcovid budget rather than rejecting the conclusion about driving.
Seems roughly on par with the estimate I used of 1⁄4 the baseline risk of death.
Splitting the expected outcome of a risk 2 ways, “life or death”, often leads to unsatisfying reasoning about risk taking. Sometimes splitting the expected outcome 3 ways: “health, life incapacitated and in agony, death” yields more satisfying explanations.
Covid is especially characteristic of this risk split in that while the expected risk of death from infection in a young and otherwise healthy person is relatively low, the risk of unknown and potentially extreme long-term health complications from infection is relatively high.
The incapacitation/agony possibility makes these calculations extremely subjective: some people might value a year of their own life spent bed-bound or in horrible pain as highly as a year in good health, while others might not. When calculating for others, things tend to go badly wrong if we assume that their relative values for incapacitation/agony versus health match our own (in either direction—opponents to life-saving treatments for the disabled and opponents to right-to-die laws are both prone to this projection), but when calculating for ourselves we can more safely use our own values.
I agree that pruning down the list of expected outcomes is a simplification. However, it seems like the badness is dominated by the outcome of dying enough where the other outcomes become negligible, and thus such a simplification is appropriate.
For example, suppose we use $100k as the value of a post-singularity year, 100k years as the expected post-singularity lifespan, and 10% as our chance of reaching the singularity. Those assumptions give us a value of life of $1B, which makes the death component of the EV calculation of driving -$25k. Now compare that to the injury component. For a lifelong injury, maybe it reduces the value of life from $200k to $100k, and you expect to live another 50 years. So the injury costs $5M versus death costing $1B. Getting permanently injured would have to be many, many times more likely than dying in a car accident for it to start becoming non-trivial next to the death component, with these assumptions.
I hold the impression that in car crashes, injuries are vastly more common than deaths. When I seek actual statistics on this, I’m surprised by how little reporting of non-fatal injury statistics is readily available online compared to fatal injury statistics.
Wikipedia claims that “In 2010, there were an estimated 5,419,000 crashes, 30,296 deadly, killing 32,999, and injuring 2,239,000.”, but the citation leads to this page, which doesn’t appear to offer injury statistics. So I’m not sure where they actually got their 2.2 million injuries per 33 thousand deaths numbers from.
injuryfacts.nsc.org claims that in 2019, there were 39,107 deaths in motor-vehicle crashes, and 4.5 million “medically consulted injuries”.
So if we trust either of those sources (which both claim to be derivations of NHTSA data), the rates are somewhere in the millions of injuries per tens of thousands of deaths kind of ballpark.
The other question here is whether injuries are permanent. I expect that any injury worth recording will have at least some long-term effect on the recipient’s quality of life, based primarily on having had a too-minor-to-report injury in a collision over a decade ago which still causes occasional pain that I wouldn’t experience without it, and also from conversations with others about the lasting effects of various “fully recovered” injuries. In trying to find any data about outcomes classified as disability, I see this and this making claims about some stats on injury-related disability, but sadly neither appears to cite any actual studies.
So, if we assume that most injuries severe enough to report cause some long-term change to quality of life, I would indeed call getting injured “many, many times more likely than dying” when it comes to modern car accidents.
Of course, all that the more reputable of this data shows is that injuries are probably something to take seriously. For those who’d value a year of life with occasional or constant pain significantly below a year without, injuries are a more relevant concern than for those who’d value the years similarly.
Good points, maybe you’re right.
Suppose we use the assumptions I made in my previous comment of death costing $1B and injury costing $5M. And suppose we ballpark it and say that injuries are 100x more likely than death. That makes it more like $1B to $500M. However, that is assuming all injuries cost $5M, and that seems high. It assumes that a) it is a permanent injury and b) it cuts the value of life fully in half. I’d ballpark it at more like maybe $1M on average. In which case it’s more like $1B to $100M.
But this is comparing injury costs to the $1B number for the value of life, which you could argue is high. If you disagree with the $1B number, then injuries seem like they might be important. For example, if we forget about the whole life extension thing and use the assumption of $10M for the value of life and $1M for the value of avoiding injury, with injury being 100x as likely as death, the injury component becomes the bigger factor.
But this extra injury stuff probably doesn’t change driving from “worth it” to “not worth it”. So the crux of the question still does seem to revolve around how much you value life.
I just looked into injuries some more. The CDC says that for every 1 person killed, 8 are hospitalized and 100 are treated and released from the emergency department. Call it 1:10:100.
In the treated and released case, let’s say the cost of the injury is $1k. And let’s normalize things with death costing $1B. Since this injury is 100x more likely than death, we can say it costs $100k. Even if it were 1000k more likely than death it’d only cost $1M next to the $1B, so it seems like we can call this negligible and forget about it.
In the hospitalized case, suppose it causes issues lasting 50 years and costs you $100k/year (in utility). That’s $5M. Because it’s 10x more likely than death, call it $50M. But that’s still a good amount smaller than the $1B. And it was with conservative assumption of every person who is hospitalized ends up with a lifelong injury that causes significant distress. In reality it might be closer to $1M than $50M. So relative to death it also seems negligible. And perhaps more importantly, it doesn’t seem like it’s enough to outweigh the convenience of driving.
The standard $10M figure is not about the value of a life. I don’t value my or any other human life at $X/yr,. I value them much more than that. What I don’t and can’t do (individually, or for everyone as a collective) is try to spend much more than that on preserving them, because the limiting factor isn’t intrinsic value it’s resource availability. I’m not going to pretend my life is anything close to an optimal arrangement of my resources for preserving said life, but even if it were, I don’t think I’d get as far as “never drive” by the time I ran out of money.
I don’t want to die, but I also want to live. The future is inherently uncertain, so if I were to take a decreased quality of life (no more driving) to better my chances of surviving to ASI, I had better have strong intuition that ASI would come.
I’m short on AI timelines (2035 median, right tailed distribution), but I also drive and take more risky activities on covid because I like being in person with my friends and family. The fact that I don’t know what a post singularity world would look like helps me feel comfortable taking these risks, it seems almost anything can happen post singularity good (hopefully) and weird.
My main worry about decreasing my quality of life by not driving / decreasing risk is the alignment problem for AI. I can image a counterfactual world where AI is created but it is not aligned, where if I were to take heavy precautions I would suffer decreased quality of life many years just to die to a rouge agent.
If I am to take this serious and hyperbolic discount my life I would need to be more assured AI is align-able, and I am too new on my journey into AI to feel comfortable talking about that yet. It may seem myopic to take these risks considering the odds, but everything is uncertain, and I could always die one day before ASI anyway.
I live in a more rural place however, if you live in a big city where walking everywhere is feasible, your calculations for quality of life change.
Yeah, I definitely hear ya. I have these feelings too. But at the same time, I think it’s in violation of Shut Up and Multiply.
I hear ya here too. It’s once of the main places that affects the conclusion, I think. My reasoning for expecting a post-singularity year to have positive utility is because it seems like a place where it’d make sense to adopt the opinion of the experts I quoted in the article. (And that it’s less depressing.)
With 0 years of experience, you are not in the top half of safe drivers.
Even if brain upload is available within your lifetime, there is a less than 100% chance that you, personally, get to do it. How rich and/or valuable are you (be honest and realistic)?
I’d you’re looking at mere “cure for aging” then years remaining is even less, and need for $ is greater than the brain upload scenario.