It’s not a distribution over agents in the universe, it’s a distribution over possible agents in possible universes. The possible universes can be given usual credence-based weightings based on conditional probability given the moral agent’s observations and models, because what else are they going to base anything on?
If your actions make 1000 people unhappy, and presumably some margin “less satisfied” in some hypothetical post-mortem universe rating, the idea seems to be that you first estimate how much less satisfied they would be. Then the novel (to me) part of this idea is that you multiply this by the estimated fraction of all agents, in all possible universes weighted by credence, who would be in your position. Being a fraction, there is no unboundedness involved. The fraction may be extremely small, but should always be nonzero.
As I see it the exact fraction you estimate doesn’t actually matter, because all of your options have the same multiplier and you’re evaluating them relative to each other. However this multiplier is what gives ethical decisions nonzero effect even in an infinite universe, because there will only be finitely many ethical scenarios of any given complexity.
So it’s not just “make 1000 happy people unhappy”, it’s “the 1 in N people with similar incentives as me in a similar situation would each make 1000 happy people unhappy”, resulting in a net loss of 1000/N of universal satisfaction. N may be extremely large, but it’s not infinite.
How is it a distribution over possible agents in possible universes (plural) when the idea is to give a way of assessing the merit of one possible universe?
I do agree that an ideal consequentialist deciding between actions should consider for each action the whole distribution of possible universes after they do it. But unless I’m badly misreading the OP, I don’t see where it proposes anything like what you describe. It says—emphasis in all cases mine, to clarify what bits I think indicate that a single universe is in question—”… but you still knew you would be born into this universe”, and “Imagine hypothetically telling an agent everything significant about the universe”, and “a prior over situations in the universe you could be born into”, and “my ethical system provides a function mapping from possible worlds to their moral value”, and “maximize the expected value of your life satisfaction given you are in this universe”, and “The appeal of aggregate consequentialism is that its defines some measure of “goodness” of a universe”, and “the moral value of the world”, and plenty more.
Even if somehow this is what OP meant, though—or if OP decides to embrace it as an improvement—I don’t see that it helps at all with the problem I described; in typical cases I expect picking a random agent in a credence-weighted random universe-after-I-do-X to pose all the same difficulties as picking a random agent in a single universe-after-I-do-X. Am I missing some reason why the former would be easier?
(Assuming you’re read my other response you this comment):
I think it might help if I give a more general explanation of how my moral system can be used to determine what to do. This is mostly taken from the article, but it’s important enough that I think it should be restated.
Suppose you’re considering taking some action that would benefit our world or future life cone. You want to see what my ethical system recommends.
Well, for almost possible circumstances an agent could end up in in this universe, I think your action would have effectively no causal or acausal effect on them. There’s nothing you can do about them, so don’t worry about them in your moral deliberation.
Instead, consider agents of the form, “some agent in an Earth-like world (or in the future light-cone of one) with someone just like <insert detailed description of yourself and circumstances>”. These are agents you can potentially (acausally) affect. If you take an action to make the world a better place, that means the other people in the universe who are very similar to you and in very similar circumstances would also take that action.
So if you take that action, then you’d improve the world, so the expected value of life satisfaction of an agent in the above circumstances would be higher. Such circumstances are of finite complexity and not ruled out by evidence, so the probability of an agent ending up in such a situation, conditioning only on being in this universe, in non-zero. Thus, taking that action would increase the moral value of the universe and my ethical system would thus be liable to recommend taking that action.
To see it another way, moral deliberation with my ethical system works as follows:
I’m trying to make the universe a better place. Most agents are in situations in which I can’t do anything to affect them, whether causally or acausally. But there are some agents in situations that that I can (acausally) affect. So I’m going to focus on making the universe as satisfying as possible for those agents, using some impartial weighting over those possible circumstances.
Your comments are focusing on (so to speak) the decision-theoretic portion of your theory, the bit that would be different if you were using CDT or EDT rather than something FDT-like. That isn’t the part I’m whingeing about :-). (There surely are difficulties in formalizing any sort of FDT, but they are not my concern; I don’t think they have much to do with infinite ethics as such.)
My whingeing is about the part of your theory that seems specifically relevant to questions of infinite ethics, the part where you attempt to average over all experience-subjects. I think that one way or another this part runs into the usual average-of-things-that-don’t-have-an-average sort of problem which afflicts other attempts at infinite ethics.
As I describe in another comment, the approach I think you’re taking can move where that problem arises but not (so far as I can currently see) make it actually go away.
How is it a distribution over possible agents in possible universes (plural) when the idea is to give a way of assessing the merit of one possible universe?
I do think JBlack understands the idea of my ethical system and is using it appropriately.
my system provides a method of evaluating the moral value of a specific universe. The point of moral agents to to try to make the universe one that scores highlly on this moral valuation. But we don’t know exactly what universe we’re in, so to make decisions, we need to consider all universes we could be in, and then take the action that maximizes the expected moral value of the universe we’re actually in.
For example, suppose I’m considering pressing a button that will either make everyone very slightly happier, or make everyone extremely unhappy. I don’t actually know which universe I’m in, but I’m 60% sure I’m in the one that would make everyone happy. Then if I press the button, there’s a 40% chance that the universe would end up with very low moral value. That means pressing the button would not in expectation decrease the moral value of the universe, so my morally system would recommend not pressing it.
Even if somehow this is what OP meant, though—or if OP decides to embrace it as an improvement—I don’t see that it helps at all with the problem I described; in typical cases I expect picking a random agent in a credence-weighted random universe-after-I-do-X to pose all the same difficulties as picking a random agent in a single universe-after-I-do-X. Am I missing some reason why the former would be easier?
I think to some extent you may be over-thinking things. I agree that it’s not completely clear how to compute P(“I’m satisfied” | “I’m in this universe”). But to use my moral system, I don’t need a perfect, rigorous solution to this, nor am I trying to propose one.
I think the ethical system provides reasonably straightforward moral recommendations in the situations we could actually be in. I’ll give an example of such a situation that I hope is illuminating. It’s paraphrased from the article.
Suppose you can have the ability to create safe AI and are considering whether my moral system recommends doing so. And suppose if you create safe AI everyone in your world will be happy, and if you don’t then the world will be destroyed by evil rogue AI.
Consider an agent that knows it will be in this universe, but nothing else. Well, consider the circumstances, “I’m an agent in an Earth-like world that contains someone who is just like gjm and in a very similar situation who has the ability to create safe AI”. That above description has finite description length, and the AI has no evidence ruling it out. So it must have some non-zero probability of ending up in such a situation, conditioning on being somewhere in this universe.
All the gjms have the same knowledge and value and are in pretty much the same circumstances. So their actions are logically constrained to be the same as yours. Thus, if you decide to create the AI, you are acausally determining the outcome of arbitrary agents in the above circumstances, by making such an agent end up satisfied when they otherwise wouldn’t have been. Since an agent in this universe has non-zero probability of ending up in those circumstances, by choosing to make the safe AI you are increasing the moral value of the universe.
As I said to JBlack, so far as I can tell none of the problems I think I see with your proposal become any easier to solve if we switch from “evaluate one possible universe” to “evaluate all possible universes, weighted by credence”.
to use my moral system, I don’t need a perfect, rigorous solution to this
Why not?
Of course you can make moral decisions without going through such calculations. We all do that all the time. But the whole issue with infinite ethics—the thing that a purported system for handling infinite ethics needs to deal with—is that the usual ways of formalizing moral decision processes produce ill-defined results in many imaginable infinite universes. So when you propose a system of infinite ethics and I say “look, it produces ill-defined results in many imaginable infinite universes”, you don’t get to just say “bah, who cares about the details?” If you don’t deal with the details you aren’t addressing the problems of infinite ethics at all!
It’s nice that your system gives the expected result in a situation where the choices available are literally “make everyone in the world happy” and “destroy the world”. (Though I have to confess I don’t think I entirely understand your account of how your system actually produces that output.) We don’t really need a system of ethics to get to that conclusion!
What I would want to know is how your system performs in more difficult cases.
We’re concerned about infinitarian paralysis, where we somehow fail to deliver a definite answer because we’re trying to balance an infinite amount of good against an infinite amount of bad. So far as I can see, your system still has this problem. E.g., if I know there are infinitely many people with various degrees of (un)happiness, and I am wondering whether to torture 1000 of them, your system is trying to calculate the average utility in an infinite population, and that simply isn’t defined.
So, I think this is what you have in mind; my apologies if it was supposed to be obvious from the outset.
We are doing something like Solomonoff induction. The usual process there is that your prior says that your observations are generated by a computer program selected at random, using some sort of prefix-free code and generating a random program by generating a random bit-string. Then every observation updates your distribution over programs via Bayes, and once you’ve been observing for a while your predictions are made by looking at what all those programs would do, with probabilities given by your posterior. So far so good (aside from the fact that this is uncomputable).
But what you actually want (I think) isn’t quite a probability distribution over universes; you want a distribution over experiences-in-universes, and not your experiences but those of hypothetical other beings in the same universe as you. So now think of the programs you’re working with as describing not your experiences necessarily but those of some being in the universe, so that each update is weighted not by Pr(I have experience X | my experiences are generated by program P) but by Pr(some subject-of-experience has experience X | my experiences are generated by program P), with the constraint that it’s meant to be the same subject-of-experience for each update. Or maybe by Pr(a randomly chosen subject-of-experience has experience X | my experiences are generated by program P) with the same constraint.
So now after all your updates what you have is a probability distribution over generators of experience-streams for subjects in your universe.
When you consider a possible action, you want to condition on that in some suitable fashion, and exactly how you do that will depend on what sort of decision theory you’re using; I shall assume all the details of that handwaved away, though again I think they may be rather difficult. So now you have a revised probability distribution over experience-generating programs.
And now, if everything up to this point has worked, you can compute (well, you can’t because everything here is uncomputable, but never mind) an expected utility because each of our programs yields a being’s stream of experiences, and modulo some handwaving you can convert that into a utility, and you have a perfectly good probability distribution over the programs.
And (I think) I agree that here if we consider either “torture 1000 people” or “don’t torture 1000 people” it is reasonable to expect that the latter will genuinely come out with a higher expected utility.
OK, so in this picture of things, what happens to my objections? They apply now to the process by which you are supposedly doing your Bayesian updates on experience. Because (I think) now you are doing one of two things, neither of which need make sense in a world with infinitely many beings in it.
If you take the “Pr(some subject-of-experience has experience X)” branch: here the problem is that in a universe with infinitely many beings, these probabilities are likely all 1 and therefore you never actually learn anything when you do your updating.
If you take the “Pr(a randomly chosen subject-of-experience has experience X)” branch: here the problem is that there’s no such thing as a randomly chosen subject-of-experience. (More precisely, there are any number of ways to choose one at random, and I see no grounds for preferring one over another, and in particular neither a uniform nor a maximum entropy distribution exists.)
The latter is basically the same problem as I’ve been complaining about before (well, it’s sort of dual to it, because now we’re looking at things from the perspective of some possibly-other experiencer in the universe, and you are the randomly chosen one). The former is a different problem but seems just as difficult to deal with.
Of course you can make moral decisions without going through such calculations. We all do that all the time. But the whole issue with infinite ethics—the thing that a purported system for handling infinite ethics needs to deal with—is that the usual ways of formalizing moral decision processes produce ill-defined results in many imaginable infinite universes. So when you propose a system of infinite ethics and I say “look, it produces ill-defined results in many imaginable infinite universes”, you don’t get to just say “bah, who cares about the details?” If you don’t deal with the details you aren’t addressing the problems of infinite ethics at all!
Well, I can’t say I exactly disagree with you here.
However, I want to note that this isn’t a problem specific to my ethical system. It’s true that in order to use my ethical system to make precise moral verdicts, you need to more fully formalize probability theory. However, the same is also true with effectively every other ethical theory.
For example, consider someone learning about classical utilitarianism and its applications in a finite world. Then they could argue:
Okay, I see your ethical system says to make the balance of happiness to unhappiness as high as possible. But how am I supposed to know what the world is actually like and what the effects of my actions are? Do other animals feel happiness and unhappiness? Is there actually a heaven and Hell that would influence moral choices? This ethical system doesn’t answer any of this. You can’t just handwave this away! If you don’t deal with the details you aren’t addressing the problems of ethics at all!
Also, I just want to note that my system as described seems to be unique among the infinite ethical systems I’ve seen in that it doesn’t make obviously ridiculous moral verdicts. Every other one I know of makes some recommendations that seem really silly. So, despite not providing a rigorous formalization of probability theory, I think my ethical system has value.
But what you actually want (I think) isn’t quite a probability distribution over universes; you want a distribution over experiences-in-universes, and not your experiences but those of hypothetical other beings in the same universe as you. So now think of the programs you’re working with as describing not your experiences necessarily but those of some being in the universe, so that each update is weighted not by Pr(I have experience X | my experiences are generated by program P) but by Pr(some subject-of-experience has experience X | my experiences are generated by program P), with the constraint that it’s meant to be the same subject-of-experience for each update. Or maybe by Pr(a randomly chosen subject-of-experience has experience X | my experiences are generated by program P) with the same constraint.
Actually, no, I really do want a probability distribution over what I would experience, or more generally, the situations I’d end up being in. The alternatives you mentioned,
Pr(some subject-of-experience has experience X | my experiences are generated by program P) and Pr(a randomly chosen subject-of-experience has experience X | my experiences are generated by program P), both lead to problems for the reasons you’ve already described.
I’m not sure what made you think I didn’t mean, P(I have experience x | …). Could you explain?
We’re concerned about infinitarian paralysis, where we somehow fail to deliver a definite answer because we’re trying to balance an infinite amount of good against an infinite amount of bad. So far as I can see, your system still has this problem. E.g., if I know there are infinitely many people with various degrees of (un)happiness, and I am wondering whether to torture 1000 of them, your system is trying to calculate the average utility in an infinite population, and that simply isn’t defined.
My system doesn’t compute the average utility of anything. Instead, it tries to compute the expected value of utility (or life satisfaction). I’m sorry if this was somehow unclear. I didn’t think I ever mentioned I was dealing with averages anywhere, though. I’m trying to get better at writing clearly, so if you remember what made you think this, I’d appreciate hearing.
I’ll begin at the end: What is “the expected value of utility” if it isn’t an average of utilities?
You originally wrote:
suppose you had no idea which agent in the universe it would be, what circumstances you would be in, or what your values would be, but you still knew you would be born into this universe. Consider having a bounded quantitative measure of your general satisfaction with life, for example, a utility function. Then try to make the universe such that the expected value of your life satisfaction is as high as possible if you conditioned on you being an agent in this universe, but didn’t condition on anything else.
What is “the expected value of your life satisfaction [] conditioned on you being an agent in this universe but [not] on anything else” if it is not the average of the life satisfactions (utilities) over the agents in this universe?
(The slightly complicated business with conditional probabilities that apparently weren’t what you had in mind were my attempt at figuring out what else you might mean. Rather than trying to figure it out, I’m just asking you.)
I’ll begin at the end: What is “the expected value of utility” if it isn’t an average of utilities?
I’m just using the regular notion of expected value. That is, let P(u) be the probability density you get utility u. Then, the expected value of utility is ∫[a,b]uP(u)du, where ∫ uses Lebesgue integration for greater generality. Above, I take utility to be in [a,b].
Also note that my system cares about a measure of satisfaction, rather than specifically utility. In this case, just replace P(u) to be that measure of life satisfaction instead of a utility.
Also, of course, P(u) is calculated conditioning on being an agent in this universe, and nothing else.
And how do you calculate P(u) given the above? Well, one way is to first start with some disjoint prior probability distribution over universes and situations you could be in, where the situations are concrete enough to determine your eventual life satisfaction. Then just do a Bayes update on “is an agent in this universe and get utility u” by setting the probabilities of hypothesis in which the agent isn’t in this universe or doesn’t have preferences. Then just renormalize the probabilities so they sum to 1. After that, you can just use this probability distribution of possible worlds W to calculate P(u) in a straightforward manner. E.g. ∫WP(utility=U|W)dP(w).
(I know I pretty much mentioned the above calculation before, but I thought rephrasing it might help.)
If you are just using the regular notion of expected value then it is an average of utilities. (Weighted by probabilities.)
I understand that your measure of satisfaction need not be a utility as such, but “utility” is shorter than “measure of satisfaction which may or may not strictly speaking be utility”.
Oh, I’m sorry; I misunderstood you. When you said the average of utilities, I thought you meant the utility averaged among all the different agents in the world. Instead, it’s just, roughly, an average among probability density function of utility. I say roughly because I guess integration isn’t exactly an average.
It’s not a distribution over agents in the universe, it’s a distribution over possible agents in possible universes. The possible universes can be given usual credence-based weightings based on conditional probability given the moral agent’s observations and models, because what else are they going to base anything on?
If your actions make 1000 people unhappy, and presumably some margin “less satisfied” in some hypothetical post-mortem universe rating, the idea seems to be that you first estimate how much less satisfied they would be. Then the novel (to me) part of this idea is that you multiply this by the estimated fraction of all agents, in all possible universes weighted by credence, who would be in your position. Being a fraction, there is no unboundedness involved. The fraction may be extremely small, but should always be nonzero.
As I see it the exact fraction you estimate doesn’t actually matter, because all of your options have the same multiplier and you’re evaluating them relative to each other. However this multiplier is what gives ethical decisions nonzero effect even in an infinite universe, because there will only be finitely many ethical scenarios of any given complexity.
So it’s not just “make 1000 happy people unhappy”, it’s “the 1 in N people with similar incentives as me in a similar situation would each make 1000 happy people unhappy”, resulting in a net loss of 1000/N of universal satisfaction. N may be extremely large, but it’s not infinite.
How is it a distribution over possible agents in possible universes (plural) when the idea is to give a way of assessing the merit of one possible universe?
I do agree that an ideal consequentialist deciding between actions should consider for each action the whole distribution of possible universes after they do it. But unless I’m badly misreading the OP, I don’t see where it proposes anything like what you describe. It says—emphasis in all cases mine, to clarify what bits I think indicate that a single universe is in question—”… but you still knew you would be born into this universe”, and “Imagine hypothetically telling an agent everything significant about the universe”, and “a prior over situations in the universe you could be born into”, and “my ethical system provides a function mapping from possible worlds to their moral value”, and “maximize the expected value of your life satisfaction given you are in this universe”, and “The appeal of aggregate consequentialism is that its defines some measure of “goodness” of a universe”, and “the moral value of the world”, and plenty more.
Even if somehow this is what OP meant, though—or if OP decides to embrace it as an improvement—I don’t see that it helps at all with the problem I described; in typical cases I expect picking a random agent in a credence-weighted random universe-after-I-do-X to pose all the same difficulties as picking a random agent in a single universe-after-I-do-X. Am I missing some reason why the former would be easier?
(Assuming you’re read my other response you this comment):
I think it might help if I give a more general explanation of how my moral system can be used to determine what to do. This is mostly taken from the article, but it’s important enough that I think it should be restated.
Suppose you’re considering taking some action that would benefit our world or future life cone. You want to see what my ethical system recommends.
Well, for almost possible circumstances an agent could end up in in this universe, I think your action would have effectively no causal or acausal effect on them. There’s nothing you can do about them, so don’t worry about them in your moral deliberation.
Instead, consider agents of the form, “some agent in an Earth-like world (or in the future light-cone of one) with someone just like <insert detailed description of yourself and circumstances>”. These are agents you can potentially (acausally) affect. If you take an action to make the world a better place, that means the other people in the universe who are very similar to you and in very similar circumstances would also take that action.
So if you take that action, then you’d improve the world, so the expected value of life satisfaction of an agent in the above circumstances would be higher. Such circumstances are of finite complexity and not ruled out by evidence, so the probability of an agent ending up in such a situation, conditioning only on being in this universe, in non-zero. Thus, taking that action would increase the moral value of the universe and my ethical system would thus be liable to recommend taking that action.
To see it another way, moral deliberation with my ethical system works as follows:
Your comments are focusing on (so to speak) the decision-theoretic portion of your theory, the bit that would be different if you were using CDT or EDT rather than something FDT-like. That isn’t the part I’m whingeing about :-). (There surely are difficulties in formalizing any sort of FDT, but they are not my concern; I don’t think they have much to do with infinite ethics as such.)
My whingeing is about the part of your theory that seems specifically relevant to questions of infinite ethics, the part where you attempt to average over all experience-subjects. I think that one way or another this part runs into the usual average-of-things-that-don’t-have-an-average sort of problem which afflicts other attempts at infinite ethics.
As I describe in another comment, the approach I think you’re taking can move where that problem arises but not (so far as I can currently see) make it actually go away.
I do think JBlack understands the idea of my ethical system and is using it appropriately.
my system provides a method of evaluating the moral value of a specific universe. The point of moral agents to to try to make the universe one that scores highlly on this moral valuation. But we don’t know exactly what universe we’re in, so to make decisions, we need to consider all universes we could be in, and then take the action that maximizes the expected moral value of the universe we’re actually in.
For example, suppose I’m considering pressing a button that will either make everyone very slightly happier, or make everyone extremely unhappy. I don’t actually know which universe I’m in, but I’m 60% sure I’m in the one that would make everyone happy. Then if I press the button, there’s a 40% chance that the universe would end up with very low moral value. That means pressing the button would not in expectation decrease the moral value of the universe, so my morally system would recommend not pressing it.
I think to some extent you may be over-thinking things. I agree that it’s not completely clear how to compute P(“I’m satisfied” | “I’m in this universe”). But to use my moral system, I don’t need a perfect, rigorous solution to this, nor am I trying to propose one.
I think the ethical system provides reasonably straightforward moral recommendations in the situations we could actually be in. I’ll give an example of such a situation that I hope is illuminating. It’s paraphrased from the article.
Suppose you can have the ability to create safe AI and are considering whether my moral system recommends doing so. And suppose if you create safe AI everyone in your world will be happy, and if you don’t then the world will be destroyed by evil rogue AI.
Consider an agent that knows it will be in this universe, but nothing else. Well, consider the circumstances, “I’m an agent in an Earth-like world that contains someone who is just like gjm and in a very similar situation who has the ability to create safe AI”. That above description has finite description length, and the AI has no evidence ruling it out. So it must have some non-zero probability of ending up in such a situation, conditioning on being somewhere in this universe.
All the gjms have the same knowledge and value and are in pretty much the same circumstances. So their actions are logically constrained to be the same as yours. Thus, if you decide to create the AI, you are acausally determining the outcome of arbitrary agents in the above circumstances, by making such an agent end up satisfied when they otherwise wouldn’t have been. Since an agent in this universe has non-zero probability of ending up in those circumstances, by choosing to make the safe AI you are increasing the moral value of the universe.
As I said to JBlack, so far as I can tell none of the problems I think I see with your proposal become any easier to solve if we switch from “evaluate one possible universe” to “evaluate all possible universes, weighted by credence”.
Why not?
Of course you can make moral decisions without going through such calculations. We all do that all the time. But the whole issue with infinite ethics—the thing that a purported system for handling infinite ethics needs to deal with—is that the usual ways of formalizing moral decision processes produce ill-defined results in many imaginable infinite universes. So when you propose a system of infinite ethics and I say “look, it produces ill-defined results in many imaginable infinite universes”, you don’t get to just say “bah, who cares about the details?” If you don’t deal with the details you aren’t addressing the problems of infinite ethics at all!
It’s nice that your system gives the expected result in a situation where the choices available are literally “make everyone in the world happy” and “destroy the world”. (Though I have to confess I don’t think I entirely understand your account of how your system actually produces that output.) We don’t really need a system of ethics to get to that conclusion!
What I would want to know is how your system performs in more difficult cases.
We’re concerned about infinitarian paralysis, where we somehow fail to deliver a definite answer because we’re trying to balance an infinite amount of good against an infinite amount of bad. So far as I can see, your system still has this problem. E.g., if I know there are infinitely many people with various degrees of (un)happiness, and I am wondering whether to torture 1000 of them, your system is trying to calculate the average utility in an infinite population, and that simply isn’t defined.
So, I think this is what you have in mind; my apologies if it was supposed to be obvious from the outset.
We are doing something like Solomonoff induction. The usual process there is that your prior says that your observations are generated by a computer program selected at random, using some sort of prefix-free code and generating a random program by generating a random bit-string. Then every observation updates your distribution over programs via Bayes, and once you’ve been observing for a while your predictions are made by looking at what all those programs would do, with probabilities given by your posterior. So far so good (aside from the fact that this is uncomputable).
But what you actually want (I think) isn’t quite a probability distribution over universes; you want a distribution over experiences-in-universes, and not your experiences but those of hypothetical other beings in the same universe as you. So now think of the programs you’re working with as describing not your experiences necessarily but those of some being in the universe, so that each update is weighted not by Pr(I have experience X | my experiences are generated by program P) but by Pr(some subject-of-experience has experience X | my experiences are generated by program P), with the constraint that it’s meant to be the same subject-of-experience for each update. Or maybe by Pr(a randomly chosen subject-of-experience has experience X | my experiences are generated by program P) with the same constraint.
So now after all your updates what you have is a probability distribution over generators of experience-streams for subjects in your universe.
When you consider a possible action, you want to condition on that in some suitable fashion, and exactly how you do that will depend on what sort of decision theory you’re using; I shall assume all the details of that handwaved away, though again I think they may be rather difficult. So now you have a revised probability distribution over experience-generating programs.
And now, if everything up to this point has worked, you can compute (well, you can’t because everything here is uncomputable, but never mind) an expected utility because each of our programs yields a being’s stream of experiences, and modulo some handwaving you can convert that into a utility, and you have a perfectly good probability distribution over the programs.
And (I think) I agree that here if we consider either “torture 1000 people” or “don’t torture 1000 people” it is reasonable to expect that the latter will genuinely come out with a higher expected utility.
OK, so in this picture of things, what happens to my objections? They apply now to the process by which you are supposedly doing your Bayesian updates on experience. Because (I think) now you are doing one of two things, neither of which need make sense in a world with infinitely many beings in it.
If you take the “Pr(some subject-of-experience has experience X)” branch: here the problem is that in a universe with infinitely many beings, these probabilities are likely all 1 and therefore you never actually learn anything when you do your updating.
If you take the “Pr(a randomly chosen subject-of-experience has experience X)” branch: here the problem is that there’s no such thing as a randomly chosen subject-of-experience. (More precisely, there are any number of ways to choose one at random, and I see no grounds for preferring one over another, and in particular neither a uniform nor a maximum entropy distribution exists.)
The latter is basically the same problem as I’ve been complaining about before (well, it’s sort of dual to it, because now we’re looking at things from the perspective of some possibly-other experiencer in the universe, and you are the randomly chosen one). The former is a different problem but seems just as difficult to deal with.
Well, I can’t say I exactly disagree with you here.
However, I want to note that this isn’t a problem specific to my ethical system. It’s true that in order to use my ethical system to make precise moral verdicts, you need to more fully formalize probability theory. However, the same is also true with effectively every other ethical theory.
For example, consider someone learning about classical utilitarianism and its applications in a finite world. Then they could argue:
Also, I just want to note that my system as described seems to be unique among the infinite ethical systems I’ve seen in that it doesn’t make obviously ridiculous moral verdicts. Every other one I know of makes some recommendations that seem really silly. So, despite not providing a rigorous formalization of probability theory, I think my ethical system has value.
Actually, no, I really do want a probability distribution over what I would experience, or more generally, the situations I’d end up being in. The alternatives you mentioned, Pr(some subject-of-experience has experience X | my experiences are generated by program P) and Pr(a randomly chosen subject-of-experience has experience X | my experiences are generated by program P), both lead to problems for the reasons you’ve already described.
I’m not sure what made you think I didn’t mean, P(I have experience x | …). Could you explain?
My system doesn’t compute the average utility of anything. Instead, it tries to compute the expected value of utility (or life satisfaction). I’m sorry if this was somehow unclear. I didn’t think I ever mentioned I was dealing with averages anywhere, though. I’m trying to get better at writing clearly, so if you remember what made you think this, I’d appreciate hearing.
I’ll begin at the end: What is “the expected value of utility” if it isn’t an average of utilities?
You originally wrote:
What is “the expected value of your life satisfaction [] conditioned on you being an agent in this universe but [not] on anything else” if it is not the average of the life satisfactions (utilities) over the agents in this universe?
(The slightly complicated business with conditional probabilities that apparently weren’t what you had in mind were my attempt at figuring out what else you might mean. Rather than trying to figure it out, I’m just asking you.)
I’m just using the regular notion of expected value. That is, let P(u) be the probability density you get utility u. Then, the expected value of utility is ∫[a,b]uP(u)du, where ∫ uses Lebesgue integration for greater generality. Above, I take utility to be in [a,b].
Also note that my system cares about a measure of satisfaction, rather than specifically utility. In this case, just replace P(u) to be that measure of life satisfaction instead of a utility.
Also, of course, P(u) is calculated conditioning on being an agent in this universe, and nothing else.
And how do you calculate P(u) given the above? Well, one way is to first start with some disjoint prior probability distribution over universes and situations you could be in, where the situations are concrete enough to determine your eventual life satisfaction. Then just do a Bayes update on “is an agent in this universe and get utility u” by setting the probabilities of hypothesis in which the agent isn’t in this universe or doesn’t have preferences. Then just renormalize the probabilities so they sum to 1. After that, you can just use this probability distribution of possible worlds W to calculate P(u) in a straightforward manner. E.g. ∫WP(utility=U|W)dP(w).
(I know I pretty much mentioned the above calculation before, but I thought rephrasing it might help.)
If you are just using the regular notion of expected value then it is an average of utilities. (Weighted by probabilities.)
I understand that your measure of satisfaction need not be a utility as such, but “utility” is shorter than “measure of satisfaction which may or may not strictly speaking be utility”.
Oh, I’m sorry; I misunderstood you. When you said the average of utilities, I thought you meant the utility averaged among all the different agents in the world. Instead, it’s just, roughly, an average among probability density function of utility. I say roughly because I guess integration isn’t exactly an average.