I’m not aware of a satisfying resolution to the problems of infinite ethics. It calls into question the underlying assumptions of classical utilitarianism, which is my justification for prioritizing AI-Xrisk above all else. I can imagine ways of resolving infinite ethics that convince me of a different ethical viewpoint which in turn changes my cause prioritization.
I think infinite ethics will most likely be solved in a way that leaves longtermism unharmed.
Yes, or it might just never be truly “solved”. I agree that complexity theory seems fairly likely to hold (something like) a solution.
Do you have specific candidate solutions in mind?
Not really. I don’t think about infinite ethics much, which is probably one of the reasons it seems likely to change my mind. I expect that if I spent more time thinking about it, I would just become increasingly convinced that it isn’t worth thinking about.
But it definitely troubles me that I haven’t taken the time to really understand it, since I feel like I am in a similar epistemic state to ML researchers who dismiss Xrisk concerns and won’t take the time to engage with them.
I guess there’s maybe a disanalogy there, though, in that it seems like people who *have* thought more about infinite ethics tend to not be going around trying to convince others that it really actually matters a lot and should change what they work on or which causes they prioritize.
-------------------
I guess the main way I can imagine changing my views by studying infinite ethics would be to start believing that I should actually just aim to increase the chances of generating infinite utility (to the extent this is actually a mathematically coherent thing to try to do), which doesn’t necessarily/obviously lead to prioritizing Xrisk, as far as I can see.
The possibility of such an update seems like it might make studying infinite ethics until I understand it better a higher priority than reducing AI-Xrisk.
I guess there’s maybe a disanalogy there, though, in that it seems like people who have thought more about infinite ethics tend to not be going around trying to convince others that it really actually matters a lot and should change what they work on or which causes they prioritize.
Yep, seems like a good reason to not be too worried about it...
I guess the main way I can imagine changing my views by studying infinite ethics would be to start believing that I should actually just aim to increase the chances of generating infinite utility (to the extent this is actually a mathematically coherent thing to try to do), which doesn’t necessarily/obviously lead to prioritizing Xrisk, as far as I can see.
If infinite utility is actually possible, then not maximizing the chances of generating infinite utility would count as an x-risk, wouldn’t it? And it seems like the best way to prevent that would be to build a superintelligent AI that would do a good job of maximizing the chances of generating infinite utility, in case that was possible. Metaphilosophical AI seems to be an obvious approach to this.
And it seems like the best way to prevent that would be to build a superintelligent AI that would do a good job of maximizing the chances of generating infinite utility, in case that was possible.
I haven’t thought about it enough to say… it certainly seems plausible, but it seems plausible that spending a good chunk of time thinking about it *might* lead to different conclusions. *shrug
I think it makes sense to at least spend some time reading up on papers and posts about infinite ethics then. It doesn’t take very long to catch up to the state of the art and then you’ll probably have a much better idea if infinite ethics would be a better field to spend more time in, and it’s probably a good idea for an AI alignment researcher to have some background in it anyway. I’d say the same thing about moral uncertainty, if your reasoning about that is similar.
I have spent *some* time on it (on the order of 10-15hrs maybe? counting discussions, reading, etc.), and I have a vague intention to do so again, in the future. At the moment, though, I’m very focused on getting my PhD and trying to land a good professorship ~ASAP.
The genesis of this list is basically me repeatedly noticing that there are crucial considerations I’m ignoring (/more like procrastinating on :P) that I don’t feel like I have a good justification for ignoring, and being bothered by that.
It seemed important enough to at least *flag* these things.
If you think most AI alignment researchers should have some level of familiarity with these topics, it seems like it would be valuable for someone to put together a summary for us. I might be interested in such a project at some point in the next few years.
The genesis of this list is basically me repeatedly noticing that there are crucial considerations I’m ignoring (/more like procrastinating on :P) that I don’t feel like I have a good justification for ignoring, and being bothered by that.
It seemed important enough to at least flag these things.
That makes sense. Suggest putting this kind of background info in your future posts to give people more context.
If you think most AI alignment researchers should have some level of familiarity with these topics, it seems like it would be valuable for someone to put together a summary for us.
Hmm, I guess I think that more for moral uncertainty than for infinite ethics. For infinite ethics, it’s more that I think at least some people in AI alignment should have some level of familiarity, and it makes sense for whoever is most interested the topic (or otherwise motivated to learn it) to learn about it. Others could just have some sense of “this is a philosophical problem that may be relevant, I’ll look into it more in the future if I need to.”
I’m often prioritizing posting over polishing posts, for better or worse.
I’m also sometimes somewhat deliberately underspecific in my statements because I think it can lead to more interesting / diverse / “outside-the-box” kinds of responses that I think are very valuable from an “idea/perspective generation/exposure” point-of-view (and that’s something I find very valuable in general).
Some of the same human moral heuristics that care about the cosmic endowment also diverge when contemplating an infinite environment. Therefore, someone who finds that the environment is infinite might exclude such heuristics from their aggregate and come to care less about what happens regarding AI than, say, their friends and family.
Why does this reduce the priority of working on AI risk? Is it just that it makes the problem harder and hence less tractable?
I’ve written down some thoughts on how the possibility of being in simulations affects my motivation to work on AI risk: Beyond Astronomical Waste, Comment 1, Comment 2.
I’m not aware of a satisfying resolution to the problems of infinite ethics. It calls into question the underlying assumptions of classical utilitarianism, which is my justification for prioritizing AI-Xrisk above all else. I can imagine ways of resolving infinite ethics that convince me of a different ethical viewpoint which in turn changes my cause prioritization.
I think infinite ethics will most likely be solved in a way that leaves longtermism unharmed. See my recent comment to William MacAskill on this topic.
Do you have specific candidate solutions in mind?
Yes, or it might just never be truly “solved”. I agree that complexity theory seems fairly likely to hold (something like) a solution.
Not really. I don’t think about infinite ethics much, which is probably one of the reasons it seems likely to change my mind. I expect that if I spent more time thinking about it, I would just become increasingly convinced that it isn’t worth thinking about.
But it definitely troubles me that I haven’t taken the time to really understand it, since I feel like I am in a similar epistemic state to ML researchers who dismiss Xrisk concerns and won’t take the time to engage with them.
I guess there’s maybe a disanalogy there, though, in that it seems like people who *have* thought more about infinite ethics tend to not be going around trying to convince others that it really actually matters a lot and should change what they work on or which causes they prioritize.
-------------------
I guess the main way I can imagine changing my views by studying infinite ethics would be to start believing that I should actually just aim to increase the chances of generating infinite utility (to the extent this is actually a mathematically coherent thing to try to do), which doesn’t necessarily/obviously lead to prioritizing Xrisk, as far as I can see.
The possibility of such an update seems like it might make studying infinite ethics until I understand it better a higher priority than reducing AI-Xrisk.
Yep, seems like a good reason to not be too worried about it...
If infinite utility is actually possible, then not maximizing the chances of generating infinite utility would count as an x-risk, wouldn’t it? And it seems like the best way to prevent that would be to build a superintelligent AI that would do a good job of maximizing the chances of generating infinite utility, in case that was possible. Metaphilosophical AI seems to be an obvious approach to this.
I haven’t thought about it enough to say… it certainly seems plausible, but it seems plausible that spending a good chunk of time thinking about it *might* lead to different conclusions. *shrug
I think it makes sense to at least spend some time reading up on papers and posts about infinite ethics then. It doesn’t take very long to catch up to the state of the art and then you’ll probably have a much better idea if infinite ethics would be a better field to spend more time in, and it’s probably a good idea for an AI alignment researcher to have some background in it anyway. I’d say the same thing about moral uncertainty, if your reasoning about that is similar.
I have spent *some* time on it (on the order of 10-15hrs maybe? counting discussions, reading, etc.), and I have a vague intention to do so again, in the future. At the moment, though, I’m very focused on getting my PhD and trying to land a good professorship ~ASAP.
The genesis of this list is basically me repeatedly noticing that there are crucial considerations I’m ignoring (/more like procrastinating on :P) that I don’t feel like I have a good justification for ignoring, and being bothered by that.
It seemed important enough to at least *flag* these things.
If you think most AI alignment researchers should have some level of familiarity with these topics, it seems like it would be valuable for someone to put together a summary for us. I might be interested in such a project at some point in the next few years.
That makes sense. Suggest putting this kind of background info in your future posts to give people more context.
Hmm, I guess I think that more for moral uncertainty than for infinite ethics. For infinite ethics, it’s more that I think at least some people in AI alignment should have some level of familiarity, and it makes sense for whoever is most interested the topic (or otherwise motivated to learn it) to learn about it. Others could just have some sense of “this is a philosophical problem that may be relevant, I’ll look into it more in the future if I need to.”
I’m often prioritizing posting over polishing posts, for better or worse.
I’m also sometimes somewhat deliberately underspecific in my statements because I think it can lead to more interesting / diverse / “outside-the-box” kinds of responses that I think are very valuable from an “idea/perspective generation/exposure” point-of-view (and that’s something I find very valuable in general).
Some of the same human moral heuristics that care about the cosmic endowment also diverge when contemplating an infinite environment. Therefore, someone who finds that the environment is infinite might exclude such heuristics from their aggregate and come to care less about what happens regarding AI than, say, their friends and family.