I had the same sense of “This is the kind of criticism where you say ‘we need two Stalins’” as one of the commenters. That doesn’t mean its correct, and I, like some others, particularly liked the phrase “pretending to actually try”. It also seems to me self-evident that this is a huge step forward and a huge improvement over merely pretending to try. Much of what is said here is correct, but none of it is the kind of criticism which would kill EA if it were correct. For that you would have to cross over into alleging things which are false.
From my perspective, by far the most obvious criticism of EA is to take the focus on global poverty at face value and then remark that from the pespective of 100,000,000 years later it is unlikely that the most critical point in this part of history will have been the distribution of enough malaria nets. Since our descendants will reliably think this was not the most utility-impactful intervention 100,000,000 years later, we should go ahead and update now, etc. And indeed I regard the non-x-risk parts of EA as being important only insofar as they raise visibility and eventually get more people involved in, as I would put it, the actual plot.
Excuse me, but this sounds to me like a terrible argument. If the far future goes right, our descendents will despise us as complete ignorant barbarians and won’t give a crap what we did or didn’t do. If it goes wrong (ie: rocks fall, everyone dies), then all those purported descendents aren’t a minus on our humane-ness ledger, they’re a zero: potential people don’t count (since they’re infinite in number and don’t exist, after all).
Besides, I damn well do care how people lived 5000 years ago, and I would certainly hope that my great-to-the-Nth-grandchildren will care how I live today. This should especially matter to someone whose idea of the right future involves being around to meet those descendents, in which case the preservation of lives ought to matter quite a lot.
God knows you have an x-risk fetish, but other than FAI (which carries actual benefits aside from averting highly improbable extinction events) you’ve never actually justified it. There has always been some small risk we could all be wiped out by a random disaster. The world has been overdue for certain natural disasters for millenia now, and we just don’t really have a way to prevent any of them. Space colonization would help, but there are vast and systematic reasons why we can’t do space colonization right now.
Except, of course, the artificial ones: nuclear winter, global warming, blah blah blah. Those, however, like all artificial problems, are deeply tied in with the human systems generating them, and they need much more systematic solutions than “donate to this anti-global-warming charity to meliorate the impact or reduce the risk of climate change killing everyone everywhere”. But rather like the Silicon Valley start-up community, there’s a nasty assumption that problems too large for 9 guys in a basement simply don’t exist.
You seem to suffer a bias where you simply say, “people are fools and the world is insane” and thus write off any notion of doing something about it, modulo your MIRI/CFAR work.
I think future humans are definitely worthy of consideration. Consider placing a time bomb in a childcare centre for 6 year old kids set to go off in 10 years. Even though the children who will be blown up don’t yet exist, this is still a bad thing to do, because it robs those kids of their future happiness and experience.
If you subscribe to the block model of the universe, then time is just another dimension, and future beings exist in the same way that someone in the room over who you can’t see also exists.
Even though the children who will be blown up don’t yet exist, this is still a bad thing to do, because it robs those kids of their future happiness and experience.
Well, it’s definitely a bad thing to do because it kills the children. I dunno if I’d follow that next inference ;-).
If you subscribe to the block model of the universe, then time is just another dimension,
Luckily, I don’t. It works well for general relativity at the large scale, but doesn’t yet some to integrate well with the smallest scales of possible causality at the quantum level. I think that a model which ontically elides the distinction between past, present, and future as “merely epistemic” is quite possibly mistaken and requires additional justification.
I realize this makes me a naive realist about time, but on the other hand, I just don’t see which predictions a “block model” actually makes about causality that account for both the success of general relativity and my very real ability to make interventions such as bombing or not bombing (personally, I’d prefer not bombing, there’s too many damn bombs lately) a day-care. You might say “you’ve already made the choice and carried out the bombing in the future”, but then you have to explain what the fundamental physical units of information are and how they integrate with relativity to form time as we know it in such a way that there can be no counterfactuals, even if only from some privileged informational reference frame.
In fact, the lack of privileged reference frames seems like an immediate issue: how can there be a “god’s eye view” where complete information about past, present, and future exist together without violating relativity by privileging some reference frame? Relativity seems configured to allow loosely-coupled causal systems to “run themselves”, so to speak, in parallel, without some universal simulator needing a global clock, so that synchronization only happens at the speed-of-light causality-propagation rate.
If I’m correctly understanding the subtext of that question (“if it doesn’t affect what you actually do besides talking, it’s meaningless to say you care about it”) then I respectfully disagree.
I am quite happy to say that A cares about B if, e.g., A’s happiness is greatly affected by B. If it happens that A is able to have substantial effect on B, then (1) we may actually be more interested in the question “what if anything does A do about B?”, which could also be expressed as “does A care about B?”, and (2) if the answer is that A doesn’t do anything about B, then we might well doubt A’s claims that her happiness is greatly affected by B. But in cases like this one—where, so far as we know, there is and could be nothing whatever that A can do to affect B—I suggest that “cares about” should be taken to mean something like “has her happiness affected by”, and that asking what A does about B is simply a wrong response.
(Note 1. I am aware that I may be quite wrong about the subtext of the question. If an answer along the lines of “It manifests itself as changes in my emotional state when I discover new things about the lives of people 5000 years ago or when I imagine different ways their lives might have been” would have satisfied you, then the above is aimed not at you but at a hypothetical version of you who meant something else by the question.)
(Note 2. You might say that caring about something you can’t influence is pointless and irrelevant. That might be correct, though I’m not entirely convinced, but in any case “how does that caring manifest itself?” seems like a strange thing to say to make that point.)
Judging by the overwhelmingly favorable response, it certainly came out as we-need-two-Stalins criticism, whether or not I “intended” it that way. (One of the less expected side effects of this post was to cause me to update towards devoting more time to things that, unlike writing, don’t give me a constant dribble of social reinforcement.)
I think my criticism includes yours, in the following sense: if we solve the “we fail to converge on truth because too much satisficing” problem, we will presumably stop saying things like “but global poverty could totally be the best thing for the far future!” (which has been argued) and start to find the things that are actually the best thing for the far future without privileging certain hypotheses.
This is my main problem with the idea that we should have a far-future focus. I just have no idea at all how to get a grip on far-future predictions, and so it seems absurdly unlikely that my predictions will be correct, making it therefore also absurdly unlikely that I (or even most people) will be able to make a difference except in a very few cases by pure luck.
It seems easier to evaluate “is trying to be relevant” than “has XYZ important long-term consequence”. For instance, investing in asteroid detection may not be the most important long-term thing, but it’s at least plausibly related to x-risk (and would be confusing for it to be actively harmful), whereas third-world health has confusing long-term repercussions, but is definitely not directly related to x-risk.
Even if third world health is important to x-risk through secondary effects, it still seems that any effect on x-risk it has will necessarily be mediated through some object-level x-risk intervention. It doesn’t matter what started the chain of events that leads to decreased asteroid risk, but it has to go through some relatively small family of interventions that deal with it on an object level.
Insofar as current society isn’t involved in object-level x-risk interventions, it seems weird to think that bringing third-world living standards closer to our own will lead to more involvement in x-risk intervention without there being some sort of wider-spread availability of object-level x-risk intervention.
(Not that I care particularly much about asteroids, but it’s a particularly easy example to think about.)
investing in asteroid detection may not be the most important long-term thing, but it’s at least plausibly related to x-risk (and would be confusing for it to be actively harmful), whereas third-world health has confusing long-term repercussions, but is definitely not directly related to x-risk.
I’m inclined to agree. A possible counterargument does come to mind, but I don’t know how seriously to take it:
Global pandemics are an existential risk. (Even if they don’t kill everyone, they might serve as civilizational defeaters that prevent us from escaping Earth or the solar system before something terminal obliterates humanity.)
Such a pandemic is much more likely to emerge and become a threat in less developed countries, because of worse general health and other conditions more conducive to disease transmission.
Funding health improvements in less developed countries would improve their level of general health and impede disease transmission.
From the above, investing in the health of less developed countries may well be related to x-risk.
Point 4 seems to follow from points 1-3. To me point 2 seems plausible; point 3 seems qualitatively correct, but I don’t know whether it’s quantitatively strong enough for the argument’s conclusion to follow; and point 1 feels a bit strained. (I don’t care so much about point 5 because you were just using asteroids as an easy example.)
Any given asteroid will either be detected and deflected in time, or not. There, to my understanding at least, no mediocre level of asteroid impact risk management which makes the situation worse, in the sense of outright increasing the chance of an extinction event. More resources could be invested for further marginal improvements, with no obvious upper bound.
Poverty and disease are more complicated problems. Incautious use of antibiotics leads to disease-resistant strains, or you give a man a fish and he spends the day figuring out how to ask you for another instead of repairing his net. Sufficient resources need to be committed to solve the problem completely, or it just becomes even more of a mess. Once it’s solved, it tends to stay solved, and then there are more resources available for everything else because the population of healthy, adequately-capitalized humans has increased.
In a situation like that, my preferred strategy is to focus on the end-in-sight problem first, and compare the various bottomless pits afterward.
I would have to disagree that there is no mediocre way to make asteroid risk worse through poor impact risk management, but perhaps it depends on what we mean by this. If we’re strictly talking about the risk of some unmitigated asteroid hitting Earth, there is indeed likely nothing we can do to increase this risk. However, a poorly construed detection, characterisation and deflection process could deflect an otherwise harmless asteroid into Earth. Further, developing deflection techniques could make it easier for people with malicious intent to deflect an otherwise harmless asteroid into Earth on purpose. Given how low the natural risk of a catastrophic asteroid impact is, I would argue that the chances of a man-made asteroid impact (either on purpose or by accident) is much higher than the chances of a natural one occurring in the next 100 years.
Yes, most x-risk reduction will have to come about through explicit work on x-risk reduction at some point.
It could still easily be the case that working on improving the living standards of the world’s poorest people is an effective route to x-risk reduction. In practice, scarcely anyone is going to work on x-risk as long as their own life is precarious, and scarcely anyone is going to do useful work on x-risk reduction if they are living somewhere that doesn’t have the resources to do serious scientific or engineering work. So interventions that aim, in the longish term, to bring the whole world up to something like current affluent-West living standards seem likely to produce a much larger population of people who might be interested in reducing x-risk and better conditions for them to do such work in.
See the point about why its weird to think that new affluent populations will work more on x-risk if current affluent populations don’t do so at a particularly high rate.
Also, it’s easier to move specific people to a country than it is to raise the standard of living of entire countries. If you’re doing raising-living-standards as an x-risk strategy, are you sure you shouldn’t be spending money on locating people interested in x-risk instead?
I quite agree that if all you care about is x-risk then trying to address that by raising everyone’s living standards is using a nuclear warhead to crack a nut. I was addressing the following thing you said:
it seems weird to think that bringing third-world living standards closer to our own will lead to more involvement in x-risk intervention without there being some sort of wider-spread availability of object-level x-risk intervention.
which I think is clearly wrong: bringing everyone’s living standards up will increase the pool of people who have the motive and opportunity to work on x-risk, and since the number of people working on x-risk isn’t zero that number will likely increase (say, by 2x) if the size of that pool increases (say, by 2x) as a result of making everyone better off.
I wasn’t claiming (because it would be nuts) that the way to get the most x-risk bang per buck is to reduce poverty and disease in the poorest parts of the world. It surely isn’t, by a large factor. But you seemed to be saying it would have zero x-risk impact (beyond effects like reducing pandemic risk by reducing overall disease levels). That’s all I was disagreeing with.
This logic suffers from an “infinity discontinuity” problem:
Consider a hypothetical paperclip maximizer. It has some resources, it has to choose between using them to make paperclips or using them to develop more efficient ways of gathering resources. A basic positive feedback calculation means the latter will lead to more paperclips in the long run. But if it keeps using that logic, it will keep developing more and more efficient ways of gathering resources and never actually get around to making paperclips.
Consider a hypothetical paperclip maximizer. It has some resources, it has to choose between using them to make paperclips or using them to develop more efficient ways of gathering resources. A basic positive feedback calculation means the latter will lead to more paperclips in the long run. But if it keeps using that logic, it will keep developing more and more efficient ways of gathering resources and never actually get around to making paperclips.
Can’t this be solved through exponential discounting? If paperclips made later are discounted more than paperclips made sooner, then we can settle on a stable strategy for when to optimize vs. when to execute, based on our estimations of optimization returns at each stage being exponential, super-exponential, or sub-exponential.
Clarifying anti-tldr edit time! If you got the above, no need to read on. (I wanted this to be an edit, but apparently I fail at clicking buttons)
The simple algorithm is the greedy decision-finding method “Choose that action which leads to one-time-tick-into-future self having the best possible range of outcomes available via further actions”, which you think could handle this problem if only the utility function employed exponential discounting (whether it actually could is irrelevant, since I adress another point).
But your utility function is part of the territory, and the utility function that you use for calculating your actions is part of the map; it is rather suspicious that you want to tweak your map towards a version that is more convenient to your calculations.
There are questions about why we should discount at all, or if we are going to, how to choose an appropriate rate.
But even setting those aside: this isn’t any more of a solution than the version without discounting. They’re similarly reliant on empirical facts about the world (the rate of resource growth); they just give differing answers about how fast that rate needs to be before you should wait rather than cash out.
Unless, or rather until, it hits diminishing returns on resource-gathering. Maybe an ocean, maybe a galaxy, maybe proton decay. With the accessible resources fully captured, it has to decide how much of that budget to convert directly into paperclips, how much to risk on an expedition across the potential barrier, and how much to burn gathering and analyzing information to make the decision. How many in-hand birds will you trade for a chance of capturing two birds currently in Andromeda?
from the perspective of 100,000,000 years later it is unlikely that the most critical point in this part of history will have been the distribution of enough malaria nets
I read this as presuming that generating/saving more humans is a worse use of smart/rich people’s attention and resources than developing future-good theory+technology (or maybe it’s only making more malaria-net-charity-recipients and their descendants that isn’t a good investment toward those future-good things, but that’s not likely to figure, since we can save quite a few lives at a very favorable ratio).
I wonder if you meant that it’s a worse use because we have more people alive now than is optimal for future good, or because we only want more smart people, or something else.
I had the same sense of “This is the kind of criticism where you say ‘we need two Stalins’” as one of the commenters. That doesn’t mean its correct, and I, like some others, particularly liked the phrase “pretending to actually try”. It also seems to me self-evident that this is a huge step forward and a huge improvement over merely pretending to try. Much of what is said here is correct, but none of it is the kind of criticism which would kill EA if it were correct. For that you would have to cross over into alleging things which are false.
From my perspective, by far the most obvious criticism of EA is to take the focus on global poverty at face value and then remark that from the pespective of 100,000,000 years later it is unlikely that the most critical point in this part of history will have been the distribution of enough malaria nets. Since our descendants will reliably think this was not the most utility-impactful intervention 100,000,000 years later, we should go ahead and update now, etc. And indeed I regard the non-x-risk parts of EA as being important only insofar as they raise visibility and eventually get more people involved in, as I would put it, the actual plot.
Excuse me, but this sounds to me like a terrible argument. If the far future goes right, our descendents will despise us as complete ignorant barbarians and won’t give a crap what we did or didn’t do. If it goes wrong (ie: rocks fall, everyone dies), then all those purported descendents aren’t a minus on our humane-ness ledger, they’re a zero: potential people don’t count (since they’re infinite in number and don’t exist, after all).
Besides, I damn well do care how people lived 5000 years ago, and I would certainly hope that my great-to-the-Nth-grandchildren will care how I live today. This should especially matter to someone whose idea of the right future involves being around to meet those descendents, in which case the preservation of lives ought to matter quite a lot.
God knows you have an x-risk fetish, but other than FAI (which carries actual benefits aside from averting highly improbable extinction events) you’ve never actually justified it. There has always been some small risk we could all be wiped out by a random disaster. The world has been overdue for certain natural disasters for millenia now, and we just don’t really have a way to prevent any of them. Space colonization would help, but there are vast and systematic reasons why we can’t do space colonization right now.
Except, of course, the artificial ones: nuclear winter, global warming, blah blah blah. Those, however, like all artificial problems, are deeply tied in with the human systems generating them, and they need much more systematic solutions than “donate to this anti-global-warming charity to meliorate the impact or reduce the risk of climate change killing everyone everywhere”. But rather like the Silicon Valley start-up community, there’s a nasty assumption that problems too large for 9 guys in a basement simply don’t exist.
You seem to suffer a bias where you simply say, “people are fools and the world is insane” and thus write off any notion of doing something about it, modulo your MIRI/CFAR work.
I think future humans are definitely worthy of consideration. Consider placing a time bomb in a childcare centre for 6 year old kids set to go off in 10 years. Even though the children who will be blown up don’t yet exist, this is still a bad thing to do, because it robs those kids of their future happiness and experience.
If you subscribe to the block model of the universe, then time is just another dimension, and future beings exist in the same way that someone in the room over who you can’t see also exists.
Well, it’s definitely a bad thing to do because it kills the children. I dunno if I’d follow that next inference ;-).
Luckily, I don’t. It works well for general relativity at the large scale, but doesn’t yet some to integrate well with the smallest scales of possible causality at the quantum level. I think that a model which ontically elides the distinction between past, present, and future as “merely epistemic” is quite possibly mistaken and requires additional justification.
I realize this makes me a naive realist about time, but on the other hand, I just don’t see which predictions a “block model” actually makes about causality that account for both the success of general relativity and my very real ability to make interventions such as bombing or not bombing (personally, I’d prefer not bombing, there’s too many damn bombs lately) a day-care. You might say “you’ve already made the choice and carried out the bombing in the future”, but then you have to explain what the fundamental physical units of information are and how they integrate with relativity to form time as we know it in such a way that there can be no counterfactuals, even if only from some privileged informational reference frame.
In fact, the lack of privileged reference frames seems like an immediate issue: how can there be a “god’s eye view” where complete information about past, present, and future exist together without violating relativity by privileging some reference frame? Relativity seems configured to allow loosely-coupled causal systems to “run themselves”, so to speak, in parallel, without some universal simulator needing a global clock, so that synchronization only happens at the speed-of-light causality-propagation rate.
Nick Bostrom has written some essays arguing for the prioritization of existential risk reduction over other causes, e.g. this one and this one.
I agree with your last paragraph.
Do you, now?
And how does that caring manifest itself?
Presumably by staying on the lookout for opportunities to get their hands on a time machine.
Hand me a time machine and you’ll find out!
Go look for blue Public Call Police Boxes :-P
If I’m correctly understanding the subtext of that question (“if it doesn’t affect what you actually do besides talking, it’s meaningless to say you care about it”) then I respectfully disagree.
I am quite happy to say that A cares about B if, e.g., A’s happiness is greatly affected by B. If it happens that A is able to have substantial effect on B, then (1) we may actually be more interested in the question “what if anything does A do about B?”, which could also be expressed as “does A care about B?”, and (2) if the answer is that A doesn’t do anything about B, then we might well doubt A’s claims that her happiness is greatly affected by B. But in cases like this one—where, so far as we know, there is and could be nothing whatever that A can do to affect B—I suggest that “cares about” should be taken to mean something like “has her happiness affected by”, and that asking what A does about B is simply a wrong response.
(Note 1. I am aware that I may be quite wrong about the subtext of the question. If an answer along the lines of “It manifests itself as changes in my emotional state when I discover new things about the lives of people 5000 years ago or when I imagine different ways their lives might have been” would have satisfied you, then the above is aimed not at you but at a hypothetical version of you who meant something else by the question.)
(Note 2. You might say that caring about something you can’t influence is pointless and irrelevant. That might be correct, though I’m not entirely convinced, but in any case “how does that caring manifest itself?” seems like a strange thing to say to make that point.)
I feel guilty for not living in ways that would be approved of by our ancestors.
Judging by the overwhelmingly favorable response, it certainly came out as we-need-two-Stalins criticism, whether or not I “intended” it that way. (One of the less expected side effects of this post was to cause me to update towards devoting more time to things that, unlike writing, don’t give me a constant dribble of social reinforcement.)
I think my criticism includes yours, in the following sense: if we solve the “we fail to converge on truth because too much satisficing” problem, we will presumably stop saying things like “but global poverty could totally be the best thing for the far future!” (which has been argued) and start to find the things that are actually the best thing for the far future without privileging certain hypotheses.
I have strong doubts about your (not personal but generic) ability to evaluate the far-future consequences of most anything.
This is my main problem with the idea that we should have a far-future focus. I just have no idea at all how to get a grip on far-future predictions, and so it seems absurdly unlikely that my predictions will be correct, making it therefore also absurdly unlikely that I (or even most people) will be able to make a difference except in a very few cases by pure luck.
It seems easier to evaluate “is trying to be relevant” than “has XYZ important long-term consequence”. For instance, investing in asteroid detection may not be the most important long-term thing, but it’s at least plausibly related to x-risk (and would be confusing for it to be actively harmful), whereas third-world health has confusing long-term repercussions, but is definitely not directly related to x-risk.
Even if third world health is important to x-risk through secondary effects, it still seems that any effect on x-risk it has will necessarily be mediated through some object-level x-risk intervention. It doesn’t matter what started the chain of events that leads to decreased asteroid risk, but it has to go through some relatively small family of interventions that deal with it on an object level.
Insofar as current society isn’t involved in object-level x-risk interventions, it seems weird to think that bringing third-world living standards closer to our own will lead to more involvement in x-risk intervention without there being some sort of wider-spread availability of object-level x-risk intervention.
(Not that I care particularly much about asteroids, but it’s a particularly easy example to think about.)
I’m inclined to agree. A possible counterargument does come to mind, but I don’t know how seriously to take it:
Global pandemics are an existential risk. (Even if they don’t kill everyone, they might serve as civilizational defeaters that prevent us from escaping Earth or the solar system before something terminal obliterates humanity.)
Such a pandemic is much more likely to emerge and become a threat in less developed countries, because of worse general health and other conditions more conducive to disease transmission.
Funding health improvements in less developed countries would improve their level of general health and impede disease transmission.
From the above, investing in the health of less developed countries may well be related to x-risk.
Optional: asteroid detection, meanwhile, is mostly a solved problem.
Point 4 seems to follow from points 1-3. To me point 2 seems plausible; point 3 seems qualitatively correct, but I don’t know whether it’s quantitatively strong enough for the argument’s conclusion to follow; and point 1 feels a bit strained. (I don’t care so much about point 5 because you were just using asteroids as an easy example.)
Any given asteroid will either be detected and deflected in time, or not. There, to my understanding at least, no mediocre level of asteroid impact risk management which makes the situation worse, in the sense of outright increasing the chance of an extinction event. More resources could be invested for further marginal improvements, with no obvious upper bound.
Poverty and disease are more complicated problems. Incautious use of antibiotics leads to disease-resistant strains, or you give a man a fish and he spends the day figuring out how to ask you for another instead of repairing his net. Sufficient resources need to be committed to solve the problem completely, or it just becomes even more of a mess. Once it’s solved, it tends to stay solved, and then there are more resources available for everything else because the population of healthy, adequately-capitalized humans has increased.
In a situation like that, my preferred strategy is to focus on the end-in-sight problem first, and compare the various bottomless pits afterward.
I would have to disagree that there is no mediocre way to make asteroid risk worse through poor impact risk management, but perhaps it depends on what we mean by this. If we’re strictly talking about the risk of some unmitigated asteroid hitting Earth, there is indeed likely nothing we can do to increase this risk. However, a poorly construed detection, characterisation and deflection process could deflect an otherwise harmless asteroid into Earth. Further, developing deflection techniques could make it easier for people with malicious intent to deflect an otherwise harmless asteroid into Earth on purpose. Given how low the natural risk of a catastrophic asteroid impact is, I would argue that the chances of a man-made asteroid impact (either on purpose or by accident) is much higher than the chances of a natural one occurring in the next 100 years.
Yes, most x-risk reduction will have to come about through explicit work on x-risk reduction at some point.
It could still easily be the case that working on improving the living standards of the world’s poorest people is an effective route to x-risk reduction. In practice, scarcely anyone is going to work on x-risk as long as their own life is precarious, and scarcely anyone is going to do useful work on x-risk reduction if they are living somewhere that doesn’t have the resources to do serious scientific or engineering work. So interventions that aim, in the longish term, to bring the whole world up to something like current affluent-West living standards seem likely to produce a much larger population of people who might be interested in reducing x-risk and better conditions for them to do such work in.
See the point about why its weird to think that new affluent populations will work more on x-risk if current affluent populations don’t do so at a particularly high rate.
Also, it’s easier to move specific people to a country than it is to raise the standard of living of entire countries. If you’re doing raising-living-standards as an x-risk strategy, are you sure you shouldn’t be spending money on locating people interested in x-risk instead?
I quite agree that if all you care about is x-risk then trying to address that by raising everyone’s living standards is using a nuclear warhead to crack a nut. I was addressing the following thing you said:
which I think is clearly wrong: bringing everyone’s living standards up will increase the pool of people who have the motive and opportunity to work on x-risk, and since the number of people working on x-risk isn’t zero that number will likely increase (say, by 2x) if the size of that pool increases (say, by 2x) as a result of making everyone better off.
I wasn’t claiming (because it would be nuts) that the way to get the most x-risk bang per buck is to reduce poverty and disease in the poorest parts of the world. It surely isn’t, by a large factor. But you seemed to be saying it would have zero x-risk impact (beyond effects like reducing pandemic risk by reducing overall disease levels). That’s all I was disagreeing with.
This logic suffers from an “infinity discontinuity” problem:
Consider a hypothetical paperclip maximizer. It has some resources, it has to choose between using them to make paperclips or using them to develop more efficient ways of gathering resources. A basic positive feedback calculation means the latter will lead to more paperclips in the long run. But if it keeps using that logic, it will keep developing more and more efficient ways of gathering resources and never actually get around to making paperclips.
In this situation, a maximizer can’t work anyway because there is no maximum.
Well, there are states that are better than others.
Can’t this be solved through exponential discounting? If paperclips made later are discounted more than paperclips made sooner, then we can settle on a stable strategy for when to optimize vs. when to execute, based on our estimations of optimization returns at each stage being exponential, super-exponential, or sub-exponential.
Finding a problem with the simple algorithm that usually gives you a good outcome doesn’t mean you get to choose a new utility function.
Clarifying anti-tldr edit time! If you got the above, no need to read on. (I wanted this to be an edit, but apparently I fail at clicking buttons)
The simple algorithm is the greedy decision-finding method “Choose that action which leads to one-time-tick-into-future self having the best possible range of outcomes available via further actions”, which you think could handle this problem if only the utility function employed exponential discounting (whether it actually could is irrelevant, since I adress another point).
But your utility function is part of the territory, and the utility function that you use for calculating your actions is part of the map; it is rather suspicious that you want to tweak your map towards a version that is more convenient to your calculations.
There are questions about why we should discount at all, or if we are going to, how to choose an appropriate rate.
But even setting those aside: this isn’t any more of a solution than the version without discounting. They’re similarly reliant on empirical facts about the world (the rate of resource growth); they just give differing answers about how fast that rate needs to be before you should wait rather than cash out.
Yes, but Eliezer doesn’t believe in discounting terminal values.
So, let’s be clear—are we talking about what works, or what we think Eliezer is dumb for believing?
Well, first I’m not a consequentialist.
However, the linked post has a point, why should we value future live less?
Unless, or rather until, it hits diminishing returns on resource-gathering. Maybe an ocean, maybe a galaxy, maybe proton decay. With the accessible resources fully captured, it has to decide how much of that budget to convert directly into paperclips, how much to risk on an expedition across the potential barrier, and how much to burn gathering and analyzing information to make the decision. How many in-hand birds will you trade for a chance of capturing two birds currently in Andromeda?
I read this as presuming that generating/saving more humans is a worse use of smart/rich people’s attention and resources than developing future-good theory+technology (or maybe it’s only making more malaria-net-charity-recipients and their descendants that isn’t a good investment toward those future-good things, but that’s not likely to figure, since we can save quite a few lives at a very favorable ratio).
I wonder if you meant that it’s a worse use because we have more people alive now than is optimal for future good, or because we only want more smart people, or something else.
I don’t think he’s saying that saving net-recipients is bad, or pointless. So I doubt either of those suggestions are correct.