Further writings at https://thewiderangle.substack.com/
Jonah Wilberg
Karma: 121
Thanks for these thoughts. Yes this is broadly the direction I am headed, and planning to explore further in future posts. I agree that both repeated (IPD-style) interactions and kin-interactions are ways of developing the evolutionary prisoner’s dilemma so as to ‘integrate Moloch and the Goddess into a single model’ as I put it at the end of my post. Another approach is to introduce spatial structure.
There are a number of different specific models to explore here, and I prefer to look at the simplest and clearest versions. Your presentation is very dense, and I think introduces some complications that are not necessary to make the point. For example I’m not sure why you introduce the assumption about two descendants and their probabilities to cooperate. You can just construct the model using a variant of Hamilton’s rule where r is the probability of interacting with kin rather than a random member of the population, this will result in cooperation growing if r is high enough relative to the payoffs.
On the contrary, the advantage of evolutionary game theory is that you do not need to assume that individuals in the model are rational agents. As I said in the previous post, evolutionary game theory stems from biology, where you’re dealing with animals that are of course not acting rationally—instead, the payoffs are what is selected for. In EPD individuals don’t really make choices at all, they just have strategies (C or D).
My focus here was on solutions that work for generic collective action problems (usual concept of Moloch), but don’t work for evolutionary prisoner’s dilemma (my concept of Moloch). It’s true I didn’t go through all possible solutions to collective action problems—instead I focused on those that don’t involve making structural changes (as opposed to changing parameters) of the model.
I don’t think your examples work as direct solutions to EPD:
Bottom-up control: any form of ‘control’ seems like ‘changing the payoffs’ - doesn’t matter whether it’s top-down or bottom-up.
Irrational agents: we’ve already covered this—the model assumes non-rational agents.
Illegibility: only helps if you assume rational agents who want legibility.
Decentralisation: you actually get basically the same overall results as EPD in a fairly small, finite population—this is called a ‘Moran process’.
Though to be clear, I do think that bottom-up control, decentralisation and social preferences are partial solutions to Moloch, just not directly—they require a switch to a different model, e.g. the Goddess model.
Thanks for the comment and for sharing your drafts—feel free to DM regarding possible collaboration. I agree: there are number of ways in which the Molochian regime can be avoided in evolutionary models, including memory of past games (iterated prisoner’s dilemma) and introducing spatial structure. And one of the benefits of seeing Moloch in these terms is that these models of the ‘evolution of cooperation’ become applicable to issues associated with Moloch like AI governance. I was planning to cover this in later posts in the sequence :)
Thanks for the comment—the reason I focus on cosmic unfairness here is because I addressed local unfairness in a previous post in the sequence—apologies this wasn’t clear, I’ve now added a hyperlink to clarify.
I don’t agree that the challenges of Dostoevsky etc are only about local unfairness though: as I say I think it’s typically a mixture of local and cosmic unfairness that are not clearly distinguished.
The ‘problem from the lack of intervention’ that you mention is much discussed by people in this context, so presumably they think it is relevant to the challenges they are considering, even if there is no easy solution.
Yes, very much agree with those points. Virtue ethics is another angle to come at the same point that there’s a process whereby you internalise system 2 beliefs into system 1. Virtues need to be practised and learned, not just appreciated theoretically. That’s why stoicism has been thought of (e.g. by Pierre Hadot) as promoting ‘spiritual exercises’ rather than systematic philosophy—I draw some further connections to stoicism in the next post in the sequence.
Thanks, yes good to see people independently arriving at a similar conclusion.
OK ‘impossible’ is too strong, I should have said ‘extremely difficult’. That was my point in footnote 3 of the post. Most people would take the fact that it has implications like needing to “maximize splits of good experiences” (I assume you mean maximise the number of splits) as a reductio ad absurdum, due to the fact that this is massively different from our normal intuitions about what we should do. But some people have tried to take that approach, like in the article I mentioned in the footnote. If you or someone else can come up with a consistent and convincing decision approach that involves branch counting I would genuinely love to see it!
I’m not at all saying the experiences of a person in a low-weight world are less valuable than a person in a high-weight world. Just that when you are considering possible futures in a decision-theoretic framework you need to apply the weights (because weight is equivalent to probability).
Wallace’s useful achievement in this context is to show that there exists a set of axioms that makes this work, and this includes branch-indifference.
This is useful because makes clear the way in which the branch-counting approach you’re suggesting is in conflict with decision theory. So I don’t disagree that you can care about the number of your thin instances, but what I’m saying is in that case you need to accept that this makes decision theory and probably consequentialist ethics impossible in your framework.
First of all, macroscopical indistinguishability is not fundamental physical property—branching indifference is additional assumption, so I don’t see how it’s not as arbitrary as branch counting.
You’re right it’s not a fundamental physical property—the overall philosophical framework here is that things can be real—as emergent entities—without being fundamental physical properties. Things like lions, and chairs are other examples.
But more importantly, branching indifference assumption is not the same as informal “not caring about macroscopically indistinguishable differences”!
This is how Wallace defines it (he in turn defines macroscopically indistinguishable in terms of providing the same rewards). It’s his term in the axiomatic system he uses to get decision theory to work. There’s not much to argue about here?
As Wallace showed, branching indifference implies the Born rule implies you almost shouldn’t care about you in a branch with a measure of 0.000001 even though it may involve drastic macroscopic difference for you in that branch. You being macroscopic doesn’t imply you shouldn’t care about your low-measure instances.
Yes this is true. Not caring about low-measure instances is a very different proposition from not caring about macroscopically indistinguishable differences. We should care about low-measure instances in proportion to the measure, just as in classical decision theory we care about low-probability instances in proportion to the probability.
OK but your original comment reads like you’re offering things not mattering cosmically as a reason for thinking MWI doesn’t change anything (if that’s not a reason, then you haven’t given any reason, you’ve just stated your view). And I think that’s a good argument—if you have general reasons that are independent of specific physics to think nothing matters (cosmically), then it will follow that nothing matters in MWI as well. I was responding to that argument.
I don’t get why you would say that the preferences are fine-grained, it kinda seems obvious to me that they are not fine-grained. You don’t care about whether worlds that are macroscopically indistinguishable are distinguishable at the quantum level, because you are yourself macroscopic. That’s why branching indifference is not arbitrary. Quantum immortality is a whole other controversial story.
You’re right that you can just take whatever approximation you make at the macroscopic level (‘sunny’) and convert that into a metric for counting worlds. But the point is that everyone will acknowledge that the counting part is arbitrary from the perspective of fundamental physics—but you can remove the arbitrariness that derives from fine-graining, by focusing on the weight. (That is kind of the whole point of a mathematical measure.)
OK I think I see where you’re coming from—but I do think the unimaginable bigness of the universe has more ‘irrelevance’ implications for a consequentialist view which tries to consider valuable states of the universe than for a virtue approach which considers valuable states of yourself. Also if you think the implication of physics is that everything is irrelevant, that seems like an important implication in it’s own right, and different from ‘normality’ (the normal way most people think about ethics, which assumes that some things actually are relevant).
Thanks for the interesting comments.
You’re right, I didn’t discuss the possibility of infinite numbers of branches, though as you suggest this leads to essentially the same conclusion as I reach in the case of finite branches, which is that it causes problems for consequentialist ethics (Joe Carlsmith’s Infinite Ethics is good on this). If what you mean by ‘normalize everything’ is to only consider the quantum weights (which are finite as mathematical measures) and not the number of worlds, then that seems more a case of ignoring those problems rather than addressing them.
I hope it was clear that I was suggesting a third approach (the number of worlds is neither finite, nor infinite, but indefinite) which does I think address the ethical problems better, since if there is no definite number of worlds then we have a reason to ignore the number of worlds and focus on the weights.
This third approach is based on the idea that ‘worlds’ are macroscopic, emergent phenomena created through decoherence (Wallace’s book contains a full mathematical treatment of this). This supports both the claim that the number of worlds is indefinite (since it depends on ultimately arbitrary mappings of macroscopic to microscopic states) and the claim that worlds are created through quantum processes (since they are macroscopically indistinguishable before decoherence occurs). My point in the post was that these two claims in combination can avoid the repugnant conclusion via the approach of focusing on the weights.
(And when it comes to the virtue-theoretic implications, I’ve again tried to follow a weight-based approach, and not make assumptions about whether worlds are created or revealed.)
Thanks for the Egan suggestion, yea I love his work though need to read Quarantine more fully. It seems like the most philosophically relevant bit might be the ending which of course is the source of Egan’s Law (it all adds up to normality). I also need to read his short story ‘Singleton’, which I gather is very relevant too.
Very useful post, thanks. While the ‘talking past each other’ is frustrating, the ‘not necessarily disagreeing’ suggests the possibility of establishing surprising areas of consensus. And it might be interesting to explore further what exactly that consensus is. For example:
Yann suggested that there was no existential risk because we will solve it
I’m sure the air of paradox here (because you can’t solve a problem that doesn’t exist) is intentional, but if we drill down, should we conclude that Yann actually agrees that there is an existential risk (just that the probabilities are lower than other estimates, and less worth worrying about)? Yann sometimes compares the situation to the risk of designing a car without brakes, but if the car is big enough that crashing it would destroy civilization that still kinda sounds like an existential risk.
I’m also not sure the fire department analogy helps here—as you note later in the paper Yann thinks he knows in outline how to solve the problem and ‘put out the fires’, so it’s not an exogenous view. It seems like the difference between the fire chief who thinks their job is easy vs the one who thinks it’s hard, though everyone agrees fires spreading would be a big problem.
I’d be interested to know more about the make-up of the audience e.g. whether they were AI researchers or interested general public. Having followed recent mainstream coverage of the existential risk from AI, my sense is that the pro-X-risk arguments have been spelled out more clearly and in more detail (within the constraints of mainstream media) than the anti-X-risk ones (which makes sense for an audience who may not have previously been aware of detailed pro- arguments, and also makes sense as doomscroll clickbait). I’ve seen lot of mainstream articles where the anti-X-risk case is basically ‘this distracts from near-term risks from AI’ and that’s it.
So if the audience is mainly people who have been following mainstream coverage it might make sense for them to update very slightly towards the anti- case on hearing many more detailed anti- arguments (not saying they are good arguments, just that they would be new to such an audience) even if the pro-X-risk arguments presented by Tegmark and Bengio were strong.
It maybe better first to force person to undergo a course with psychologist and psychiatrist and only after that allow a suicide..
Something along these lines seems essential. It may be better to talk of the right to informed suicide. Arguably being informed is what makes it truly voluntary.
Forcing X to live will be morally better than forcing Y to die in circumstances where X’s desire for suicide is ill-informed (let’s assume Y’s desire to live is not). X’s life could in fact be worth living—perhaps because of a potential for future happiness, rather than current experiences—but X might be unable to recognise that fact.
It may be that a significant portion of humanity has been psychologically ‘forced’ to live by an instinct of self-preservation, since if they stopped to reflect they would not immediately find their lives to be worth living. That would be a very good thing if their lives were in fact worth living in ways that they could not intellectually recognise.
Thanks—and that’s a completely fair point. My focus here has been on systems of pairwise interactions (prisoner’s dilemma) as that seems closer to the original Moloch framing. But you’re right that another very relevant angle on this is to consider the move to multiplayer or ‘public goods’ games. I think you could argue that network structure is still relevant to those systems too (see e.g. this article) - as far as I can make out the consensus seems to be that the effects of the kinds of things mentioned in this post are weaker in such systems, but not non-existent. And the fact that they matter at all is important to avoiding the pure pessimism involved in the classic Moloch framing.