I write and talk about game theory, moral philosophy, ethical economics and artificial intelligence—focused on non-zero-sum games and their importance in solving the world’s problems.
I have admitted I am wrong at least 10 times on the internet.
James Stephen Brown
Karma: 415
What About Us?
The Whole Kitten-Cavoodle
An Argument for Analogies—Polymaths 1/3
There is a version of the Trolley Problem, which extends to a hospital scenario, where instead of switching tracks from 5 people to kill one, you’re killing one healthy person to save the lives of 5 people who need transplants.
Two points:
1. To my mind this is not even a problem, if doctors killed people with no fatal illnesses when they went to the hospital… no one would go to the hospital, and many more people would die. Trust is required for an ordered society.Now, we can play the game where we stipulate that this is entirely isolated, and then keep taking real-world variables away from a situation until it’s identical to the other… sure… if you take away society… (even though we’re a social species and have moral responsibilities precisely because we live in a society) then I guess, if I mention all the nurses, doctors, surgeons, anaesthetists etc involved in these extensive surgeries, and the recipients who receive the organs who now have to live their lives with this dark secret, we’ll just magic away that with magic surgery, and memory wiping… sure, if you keep eliminating all the factors that make a hospital a hospital… then sure, you can literally recreate the trolley scenario, and at some point in this transformation the moral calculus will return to the switching tracks answer.
Moral concerns take on multiple factors. These “paradoxes” assume if you can’t reduce an equation to a single factor, you can’t get a valid answer. But that’s not how equations work.
Like the answer to bad science is better science, the answer to problematically framed utilitarianism is a more complex and accurate utilitarian framing.
2. Reflecting on this question, I tried plugging this into my Shapley Value Calculator, which measures marginal contributions to a group. Taking each coalition’s total years lived as the utilitarian goal, the “donor” (in this case Meredith) scores higher than all the others combined. Unfortunately I couldn’t do it with one donor and 5 patients because I was limited to 5 people total in the calculator (my bad).Bob: 23.00
Allie: 23.00
Tom: 23.00
Helen: 23.00
Meredith: 108.00So Shapley agrees that Meredith’s life is worth more… ethically. This might make no sense to you, but Lloyd Shapley did win a Nobel Prize for his work in game theory so I’m happy to extend a little authority to him.
Having said this, Meredith’s “worth” could be seen as a justification to sacrifice her, precisely because her marginal contribution is greater than the total of all the others, but I don’t think that would be reading the marginal gain correctly—it is usually used to justify the share of the “profits” or the “decision making power” meaning she would have a more than 50% stake in the decision, and should be allowed to opt out of the deal.
This is all purely theoretical of course… but it was an interesting experiment and could have gone either way.
Right, nice catch. I’ll add that as a note, as I think it’s still fair to say that the term came from the world of video games, as that’s the route it took into common parlance, even if it’s not the origin.
The NPC → MC Spectrum
It is quite possible to hyperoptimize for that one particular yumminess, then burn out and later realize that one values other things too—as many a parent learns when the midlife crisis hits.
So true, this reminds me of Jung’s emphasis on “the shadow“—it’s important to acknowledge (and not discount) “values” you hold that are selfish or otherwise not ostensibly pro-social.… your actual Values long term (which usually involves other people)
This is also important to note. We are often torn between selfish wants and the wants and needs of others. This can be framed as selfishness = bad, concern for others = good. But I think it’s better interpreted as you say, that “goodness” is usually aligned with our own long-term interests which are often also aligned with the interests of others. So your values need not be a zero-sum contest between your interests and the interests of others.
Dr Evil & Realpolitik
I’ve been running into something I think of as “The Narrow Band Dilemma” where a moderate ethical position is fragile because it is in a battle on two fronts between a popular more pure ethical position and a popular unethical or a-moral position.
The first example is ethically produced meat / free-range farming, where a slightly more expensive product tries to find a market that loses patrons on both sides, either people who don’t care about where their meat comes from, or care more about the expense than the ethics on the one hand, and the people who forgo meat altogether (vegans / vegetarians) on the other. As messaging successfully advertises the benefits of ethical meat production, it gains market share from one end of the spectrum (the “I care about ethics but can’t justify the additional expense”) but loses them on the other because the heightened awareness of animal cruelty drives ethical meat eaters away from eating meat altogether.
I noticed another example today in moderate Christianity, where it is flanked by fundamentalism and atheism. The more that Christians want to align their beliefs with modern secular ethics (moving away from fundamentalism), the more they are likely to leave the faith altogether.
Now, I’m not actually saying this is a problem, just a phenomenon I’ve noticed, I’m an atheist, so don’t mind if more people come to think as I do, but I’d also like more Christians to be more moderate, but I can see it’s difficult to build critical mass when they exist in a narrow band. I am in the narrow band when it comes to ethical meat, and I have seen how long it has taken for ethical meat products to become ubiquitous and accessible, and truly ethical real meat (lab grown) is still a way off. I imagine, if a much larger market had emerged (without the pressures of the narrow band), it’s possible this could be available already.
Rights & Liberties—are opposites
Saidi, My Friend—what do we owe to each other?
Emergence Is Beautiful—beauty and meaning in an entropic universe
You’re right, but the term teme covers much more than that, for instance it’s also relevant to the development of AI agents, and AI self-editing / self-improvement. Although, identifying these systems as virus-like (because of the replication mechanism) might be instructive (as a red-flag).
Replicators—Pandora’s dangerous children
On The Shoulders of Substrates—how one phenomenon lays the foundation for the next
I agree, it’s not coming across at all well at present, needs a rewrite, give me a couple of weeks :)
I take your point, I think it needs a rewrite, I have not been nearly clear enough, and your notes are helpful in pointing me to areas I need to clarify. I have replies to your points here, but I should get my ducks-in-a-row before making them, so I don’t end up contradicting myself. Thanks for your comment.
Thanks Adele,
I appreciate your comment, and will take some time to process it and read the links. This is definitely not an area I have any expertise in and I’m not meaning to propose that this is how gravity actually works in reality—it’s more an illustration that something gravity-like, and elements that are like atoms or systems etc can arise out of very simple and random rules without the need for fine-tuning, and that constants (or regularities) can be arrived at by means of natural equilibria rather than being lucked upon, or designed.
But I probably haven’t made this clear. It was something I actually wrote I while ago and have only recently published here, so it may require a re-write, clarifying my intention and incorporating the points you’ve raised. You’re the first to provide a rigorous rebuttal for it so far, so I appreciate you lending your expertise in this respect.
Hey Seth, this was fascinating, a really beautifully thought out piece which I learned a lot from, and which also fired up a lot of associations. I hope you don’t mind but I wrote the thoughts that came to mind while listening and where it crosses over with ideas I’ve explored (less rigorously than you). They’re not arguments, just different ways I’ve thought about similar things, often in a way that ends up aligning with you. You’ve mentioned that you haven’t written for lay-people yet, but I found this quite accessible, and I’m sort of a layperson.
One short isolated association: your idea about infinite cognitive ability allowing us to believe true things but profess convenient falsehoods reminds me of Thrasymachus in Plato’s Republic, who proposes that we would do best to profess moral selflessness while acting selfishly in secret—Plato unfortunately doesn’t clock “massive cognitive dissonance” as a problematic factor, lol.
When it comes to human idiosyncracies like bias, I’m always asking how this bug might be a feature—I’ve done this with cogitive bias while simulating political alignment. I often find myself arguing in a way that validates the status quo, because I think often there are hidden or taken-for-granted elements or practices involved in the status quo that give it a sort of logic it doesn’t appear to have on a superficial level (and while I’m not dogmatically attached to the status quo, I find it’s counterintuitively underrepresented in arguments).
When a study in isolation finds a bias towards one view (like the studies you mention early in the piece) I ask “have they taken into account the path to that bias?”. Perhaps the subject got to that “bias” through reason and, having done that work already, are loathed to do it again (so it’s really an efficiency bias). If someone has already found 100 arguments against their position wanting, it makes sense to reduce the weight of new arguments. This is a sort of approximation of Bayesian reasoning (you acknowledge this around 24 minutes calling it a “inference machine” but only in a narrow domain, and then you mention something similar at 30 minutes—sorry, I’m listening obviously) and nature has cleverly done this to avoid us flip-flopping constantly whenever we are faced with a seemingly deductive argument (you mention later that memory isn’t relevant when weighing evidence, but it sort of is, if you’ve remembered evidence you’ve pre-processed). The weight of experience and our bias towards cognitive coherence protects against being duped on the reg.
I really felt like many of the issues I thought of, you addressed soon after they occurred to me (a sign of good writing). When thinking about bias toward the weight of personal experience, I was thinking, if this is valid then it’s also rational to take into account the quality and quantity of your interlocutor’s experience, and to weight that accordingly. You address this relationship when talking about the bias inherent in deferring to experts (which assumes the process I just mentioned).
The outsourcing to experts idea reminded me of an idea for digital democracy my mate proposed to me a couple of decades ago that, rather than politicians, we could have a range of political issues on a sort of perpetual referendum, but to avoid the overwhelm of having to constantly vote on every issue, we would nominate experts (who share some common moral compass) to vote on our behalf in relation to topics that are within their domain of expertise. Which I thought was a pretty clever idea—when the technology gets up to the task.
While I think deferring to experts is rational, I see your point with double-counting. This can be seen where one study, like that famous autism study, gets distributed widely before it is debunked and then the truth spends the next decades chasing the falsehood.
Where my intuitions about feature / bug break down (and they must, because problematic cognitive bias is a bug in the world, for sure) is this first-mover advantage when it comes to chaotic systems, leading me on a path of adjacent possibles, that are only available due to initial accidents of exposure (because chaotic systems are characteristically sensitive to “initial conditions”). I’m not sure how to protect against this, and don’t trust first principles thinking is going to help all that much, as it’s pretty prone to bias itself. I think perhaps a sort of “comparative religion” approach where you step back and look at the field of possibilities (the “raw distribution of beliefs” you mention) from time to time (I see you go on to suggest something very similar—naming the views). The scout mindset you mention later also adds a layer of redundancy to initial conditions in a similar way.
I think, bearing all this in mind your assertion that epistemic humility leads to clustering makes a lot of sense—and is an entirely new idea in my head, thanks. Come to think if it, it’s the sort of dynamic you’d expect to see with the digital democracy idea above, which might be an argument against that, I guess.
When you mentioned the “most irritating arguments” in relation to the “strongest arguments” I couldn’t help think… um, for me, those are almost always the same! But then I realised they are subtly different. Strong arguments might actually change my mind, so I actually see value in them (though they obviously make me feel uneasy, I’m only human). Irritating arguments, on the other hand, are, to me, those that I can see will be convincing to others who don’t have the weight of my experience telling me they are obviously incorrect—giving me the obligation to unpack that vague weight of experience and put it into words that will immunise those (gullible) people, without that experience, against the irritating argument. Such arguments include the ontological argument, and arguments for libertarianism.
Even though you’ve used estimates, I love that you’ve done actual Bayesian calculations in an accessible way, the chart tells the story nicely.
On the steelmanning point, an extension of this is the reverse argument, where you and your opponent, after arguing for a bit, switch roles. I’ve done this in my misspent youth arguing with religious apologists, and the reverse argument was the only thing that ever influenced the other person to change (and it upgraded my reasoning ability in the space too)—a friendly opponent I’d been arguing with for months, on my suggestion, switched roles, we went back and forth for a week then it trailed off. Two weeks later he informed me he was now an agnostic atheist. I didn’t ask why, but I think a week making arguments for the position (after months of exposure to pretty strong arguments for the position, in a polite and friendly exchange) had something to do with it.
I also like the idea of “identity defense” as a mindset to avoid.
Again, it was really nicely written and clear, I liked that it extended outside a strictly AI alignment realm into more general applications.