One important dimension to consider is how hard it is to solve philosophical problems well enough to have a pretty good future (which includes avoiding bad futures). It could be the case that this is not so hard, but fully resolving questions so we could produce an optimal future is very hard or impossible. It feels like this argument implicitly relies on assuming that “solve philosophical problems well enough to have a pretty good future” is hard (ie. takes thousands of millions of years in scenario 4) - can you provide further clarification on whether/why you think that is the case?
I tried to make arguments in this direction in Beyond Astronomical Waste and Two Neglected Problems in Human-AI Safety. Did you read them and/or find them convincing? To be clear I do think there’s a significant chance that we could just get lucky and it turns out that solving philosophical problems well enough to have a pretty good future isn’t that hard. (For example maybe it turns out to be impossible or not worthwhile to influence bigger/richer universes so we don’t lose anything even if we never solve that problem.) But from the perspective of trying to minimize x-risk, it doesn’t seem like a good idea to rely on that.
thousands of millions
That was “thousands or millions”. I think it’s unlikely we’ll need billions of years. :) BTW I think I got the idea of thousands or millions of years of “the long reflection” from William MacAskill’s 80,000 Hours interview, but I’m not sure who was the first to suggest it. (I think it’s fairly likely that we’ll need at least a hundred years which doesn’t seem very different from thousands or millions from a strategic perspective. Not sure if that’s the part that you’re having an issue with.)
Thanks, this position makes more sense in light of Beyond Astronomical Waste (I guess I have some concept of “a pretty good future” that is fine with something like a bunch of human-descended beings living a happy lives that misses out on the sort of things mentioned in Beyond Astronomical Waste, and “optimal future” which includes those considerations). I buy this as an argument that “we should put more effort into making philosophy work to make the outcome of AI better, because we risk losing large amounts of value” rather than “our efforts to get a pretty good future are doomed unless we make tons of progress on this” or something like that.
I buy this as an argument that “we should put more effort into making philosophy work to make the outcome of AI better, because we risk losing large amounts of value” rather than “our efforts to get a pretty good future are doomed unless we make tons of progress on this” or something like that.
What about the other post I linked, Two Neglected Problems in Human-AI Safety? A lot more philosophical progress would be one way to solve those problems, and I don’t see many other options.
One important dimension to consider is how hard it is to solve philosophical problems well enough to have a pretty good future (which includes avoiding bad futures). It could be the case that this is not so hard, but fully resolving questions so we could produce an optimal future is very hard or impossible. It feels like this argument implicitly relies on assuming that “solve philosophical problems well enough to have a pretty good future” is hard (ie. takes thousands of millions of years in scenario 4) - can you provide further clarification on whether/why you think that is the case?
I tried to make arguments in this direction in Beyond Astronomical Waste and Two Neglected Problems in Human-AI Safety. Did you read them and/or find them convincing? To be clear I do think there’s a significant chance that we could just get lucky and it turns out that solving philosophical problems well enough to have a pretty good future isn’t that hard. (For example maybe it turns out to be impossible or not worthwhile to influence bigger/richer universes so we don’t lose anything even if we never solve that problem.) But from the perspective of trying to minimize x-risk, it doesn’t seem like a good idea to rely on that.
That was “thousands or millions”. I think it’s unlikely we’ll need billions of years. :) BTW I think I got the idea of thousands or millions of years of “the long reflection” from William MacAskill’s 80,000 Hours interview, but I’m not sure who was the first to suggest it. (I think it’s fairly likely that we’ll need at least a hundred years which doesn’t seem very different from thousands or millions from a strategic perspective. Not sure if that’s the part that you’re having an issue with.)
Thanks, this position makes more sense in light of Beyond Astronomical Waste (I guess I have some concept of “a pretty good future” that is fine with something like a bunch of human-descended beings living a happy lives that misses out on the sort of things mentioned in Beyond Astronomical Waste, and “optimal future” which includes those considerations). I buy this as an argument that “we should put more effort into making philosophy work to make the outcome of AI better, because we risk losing large amounts of value” rather than “our efforts to get a pretty good future are doomed unless we make tons of progress on this” or something like that.
“Thousands of millions” was a typo.
What about the other post I linked, Two Neglected Problems in Human-AI Safety? A lot more philosophical progress would be one way to solve those problems, and I don’t see many other options.