Here are some thoughts about the recent back-and-forth where Will MacAskill reviewed IABI and Rob Bensinger wrote a reply and Will replied back. I’m making this a quick take instead of a full post because it gets kinda inside baseball/navelgazy and I want to be more chill about that than I would be in a full writeup.
First of all, I want to say thank you to Will for reviewing IABI. I got a lot out of the mini-review, and broadly like it, even if I disagree on the bottom-line and some of the arguments. It helped me think deep thoughts.
The evolution analogy
I agree with Will that the evolution analogy is useful and informative in some ways, but of limited value. It’s imperfect, and thinking hard about the differences is good.
The most basic disanalogy is that evolution wasn’t trying, in any meaningful sense, to produce beings that maximise inclusive genetic fitness in off-distribution environments.
I agree with this, and I appreciate MacAskill talking about it.
But we will be doing the equivalent of that!
One of the big things that I think distinguishes more doomy people like me from less doomy people like Will is our priors on how incompetent people are. Like, I agree that it’s possible to carefully train an ML system to be (somewhat) robust to distributional shifts. But will we actually do that?
I think, at minimum, any plan to build AGI (to say nothing of ASI) should involve:
Trying to give the AI a clear, (relatively) simple goal that is well understood by humans and which we have strong reason to expect won’t cause a catastrophe if faithfully pursued, even if it gets wildly more power than expected and goes a little off-the-rails.
Training that AI in a wide variety of environments, trying to hit the real goal, rather than proxies. This should involve adversarial environments and a lot of paranoia that the AI has failed to robustly internalize the true goal.
Testing the AI on entirely new environments that were a-priori suspected of being difficult and weird, but where we also think there is a true answer that can be checked, restarting from scratch if the AI fails to generalize to the test set, or otherwise clearly demonstrates that it has not internalized the desired goal in a robust way.
And I personally think pure corrigibility has a nonzero chance of being a good choice of goal, and that a sufficiently paranoid training regime has a nonzero chance of being able to make a semi-safe AGI this way, even with current techniques. (That said, I don’t actually advocate for plans that have a significant chance of killing everyone, and I think “try to build corrigible AGI” does have a significant chance of killing everyone; I just notice that it seems better than what the research community currently seems to be doing, even at Anthropic.)
I predict the frontier lab that builds the first AGI will not be heavily focused on ensuring robustness to distributional shifts. We could bet, maybe.
Types of misalignment
I really benefited from this! Will changed my mind! My initial reaction to Will’s mini-review was like, “Will is wrong that these are distinct concepts; any machine sufficiently powerful to have a genuine opportunity to disempower people but which is also imperfectly aligned will produce a catastrophe.”
And then I realized that I was wrong. I think. Like, what if Will is (secretly?) gesturing at the corrigibility attractor basin or perhaps the abstracted/generalized pattern of which corrigibility is an instance? (I don’t know of other goals which have the same dynamic, but maybe it’s not just corrigibility?)
An agent which is pseudo-corrigible, and lives inside the attractor basin, is imperfectly aligned (hence the pseudo) but if it’s sufficiently close to corrigible it seems reasonable to me that it won’t disempower humanity, even if given the opportunity (at least, not in every instance it gets the opportunity). So at the very least, corrigibility (one of my primary areas of research!) is (probably) an instance of Will being right (and my past self being wrong), and the distinction between his “types of misalignment” is indeed a vital one.
I feel pretty embarrassed by this, so I guess I just wanna say oops/sorry/thanks.
If I set aside my actual beliefs and imagine that we’re going to naturally land in the corrigibility attractor basin by default, I feel like I have a better sense of some of the gradualism hope. Like, my sense is that going from pseudo-corrigible to perfectly corrigible is fraught, but can be done with slow, careful iteration. Maybe Clara Collier and other gradualists think we’re going to naturally land in the corrigibility attractor basin, and that the gradual work is the analogue of the paranoid iteration that I conceive as being the obvious next-step?
If this is how they’re seeing things, I guess I feel like I want to say another oops/sorry/thanks to the gradualists. …And then double-click on why they think we have a snowball’s chance in hell of getting this without a huge amount of restriction on the various frontier labs and way more competence/paranoia than we currently seem to have. My guess is that this, too, will boil down to worldview differences about competence or something. Still. Oops?
(Also on the topic of gradualism and the notion of having “only one try” I want to gesture at the part of IABI where it says (paraphrased from memory, sorry): if you have a clever scheme for getting multiple tries, you still only get one try at getting that scheme to work.)
appeals to what “most” goals are like (if you can make sense of that) doesn’t tell you much about what goals are most likely. (Most directions I can fire a gun don’t hit the target; that doesn’t tell you much about how likely I am to hit the target if I’m aiming at it.)
I agree that “value space is big” is not a good argument, in isolation, for how likely it is for our creations to be aligned. The other half of the pincer is “our optimization pressure towards aligned goals is weak,” and without that the argument falls apart.
(Maybe we won’t be able to make deals with AIs? I agree that’s a worry; but then the right response is to make sure that we can. Won’t the superintelligence have essentially a 100% chance of taking over, if it wanted to? But that’s again invoking the “discontinuous jump to godlike capabilities” idea, which I don’t think is what we’ll get).
Here’s a plan for getting a good future:
Build ASI slowly, such that there’s some hope of being able to understand the first AI capable of a pivotal act.
The AI will want weird, alien stuff, but we’ll make sure that it’s the kind of mind that would prefer getting 50% of the stars with 99% probability to getting 100% of the stars with 70% probability.
Since we’re going slowly, we still have a 30% chance to stop it if we wanted.
We tell the AI “we’ll let you do a pivotal act and escape our clutches if you agree to only eat 50% of the stars, and help us get the other 50% as though you were aligned”
Our interpretability techniques are so good that we know whether it’s lying or whether it’s honorable and will actually give us half the stars.
Because we’re so good at reading its advanced, alien mind, it knows it needs to be honorable with us, and so is actually honorable.
The AI says “Yep, will do.”
We see it’s telling the truth when we check.
We set it free.
It gives us a good future.
I think this plan is bad because it fails the heuristic of “don’t summon demons and try to cleverly bargain with them,” but perhaps I’m being unfair.
My main criticism with “make deals with the AIs” is that it seems complex and brittle and like it depends heavily on a level of being able to read the machine’s mind that we definitely don’t currently have and might never have.
That said, I do think there’s a lot of value in being the sorts of people/groups that can make deals and be credible trade partners. Efforts to be more trustworthy and honorable and so on seem great.
suppose that all the first superintelligence terminally values is paperclips. But it’s risk-averse, in the sense that it prefers a guarantee of N resources over a 50⁄50 chance of 0 or 2N resources; let’s say it’s more risk-averse than the typical human being.
On a linguistic level I think “risk-averse” is the wrong term, since it usually, as I understand it, describes an agent which is intrinsically averse to taking risks, and will pay some premium for a sure-thing. (This is typically characterized as a bias, and violates VNM rationality.) Whereas it sounds like Will is talking about diminishing returns from resources, which is, I think, extremely common and natural and we should expect AIs to have this property for various reasons.
it would strongly prefer to cooperate with humans in exchange for, say, a guaranteed salary, rather than to take a risky gamble of either taking over the world or getting caught and shut off.
Rob wrote some counterpoints to this, but I just want to harp on it a little. Making a deal with humans to not accumulate as much power as possible is likely an extremely risky move for multiple reasons, including that other AIs might come along and eat the lightcone.
I can imagine a misaligned AI maybe making a deal with humans who let it out of the box in exchange for some small fraction of the cosmos (and honoring the deal; again, the hard part is that it has to know we can tell if it’s lying, and we probably can’t).
I can’t really imagine an AI that has a clear shot at taking over the world making a deal to be a meek little salary worker, even if there are risks in trying to take over. Taking over the world means, in addition to other things, being sure you won’t get shut off or replaced by some other AI or whatever.
(Though I can certainly imagine a misaligned AI convincing people (and possibly parts of itself) that it is willing to make a deal like that, even as it quietly accumulates more power.)
Their proposal
Now we’re getting into the source of the infighting, I think (just plain fighting? I think of Will as being part of my ingroup, but idk if he feels the same; Rob definitely is part of my ingroup; are they part of each other’s ingroups? Where is the line between infighting and just plain fighting?). Will seems very keen on criticizing MIRI’s “SHUT IT DOWN YOU FOOLS” strategy — mostly, it seems to me, because he sees this approach as insufficiently supportive of strategies besides shutting things down.
When Rob shared his draft of his reply to Will, I definitely noticed that it seemed like he was not responding accurately to the position that I saw in Will’s tweet. Unfortunately, I was aware that there is something of a history between Will and MIRI and I incorrectly assumed that Rob was importing true knowledge of Will’s position that I simply wasn’t aware of. I warned him that I thought he was being too aggressive, writing “I expect that some readers will be like ‘whoa why is MIRI acting like this guy is this extremist—I don’t see evidence of that and bet they’re strawmanning him’.” But I didn’t actually push back hard, and that’s on me. Apologies to Will.
(Rob reviewed a draft of this post and adds his own apologies for misunderstanding Will’s view. He adds: “My thanks to Max and multiple other MIRI people for pushing back on that part of my draft. I made some revisions in response, though they obviously weren’t sufficient!”)
I’m very glad to see in Will’s follow-up:
“I definitely think it will be extremely valuable to have the option to slow down AI development in the future,” as well as “the current situation is f-ing crazy”
I wish this had been more prominent in his mini-review, but :shrug:
I think Will and I probably agree that funding a bunch of efforts to research alignment, interpretability, etc. would be good. I’m an AI safety/alignment researcher, and I obviously do my day-to-day work with a sense that it’s valuable and a sense that more effort would also be valuable. I’ve heard multiple people (whom I respect and think are doing good work) complain that Eliezer is critical/dismissive of their work, and I wish Eliezer was more supportive of that work (while also still saying “this won’t be sufficient” if that’s what he believes, and somehow threading that needle).
I am pretty worried about false hope, though. I’m worried that people will take “there are a bunch of optimistic researchers working hard on this problem” as a sign that we don’t need to take drastic action. I think we see a bunch of this already and researchers like myself have a duty to shout “PLEASE DON’T RISK EVERYTHING! I DON’T GOT THIS!”[1] even while pursuing the least-doomed alignment strategies they know of. (I tried to thread this needle in my corrigibility research.)
Anyway, I think I basically agree with Will’s clarified position that a “kitchen-sink approach” is best, including a lot of research, as long as actually shutting down advanced training runs and pure capabilities research is in the kitchen sink. I feel worried that Will isn’t actually pushing for that in a way that I think is important (not building “It” is the safest intervention I’m aware of), but I’m also worried about my allies (people who basically agree that AI is unacceptably dangerous and that we need to take action) being unable to put forward a collective effort without devolving into squabbling about tone and strawmanning each other. :(
Anyway. Thank you again to Will and Rob. I thought both pieces were worth reading.
(Not to say that we should necessarily risk everything if alignment researchers do feel like they’ve “got this.” That’s a question worth debating in its own right. Also, it’s obviously worth noting that work that is incrementally useful but clearly insufficient to solve the entire field can still be valuable and the researcher is still allowed to say “I got this” on their little, local problems. (And they’re definitely allowed to speak up if they actually do solve the whole damn problem, of course. But they better have actually solved it!))
On a linguistic level I think “risk-averse” is the wrong term, since it usually, as I understand it, describes an agent which is intrinsically averse to taking risks, and will pay some premium for a sure-thing. (This is typically characterized as a bias, and violates VNM rationality.) Whereas it sounds like Will is talking about diminishing returns from resources, which is, I think, extremely common and natural and we should expect AIs to have this property for various reasons.
That’s not quite right. ‘Risk-averse with respect to quantity X’ just means that, given a choice between two lotteries A and B with the same expected value of X, the agent prefers the lottery with less spread. Diminishing marginal utility from extra resources is one way to get risk aversion with respect to resources. Risk-weighted expected utility theory is another. Only RWEUT violates VNM. When economists talk about ‘risk aversion,’ they almost always mean diminishing marginal utility.
diminishing returns from resources… is, I think, extremely common and natural and we should expect AIs to have this property for various reasons.
Can you say more about why?
Making a deal with humans to not accumulate as much power as possible is likely an extremely risky move for multiple reasons, including that other AIs might come along and eat the lightcone.
But AIs with sharply diminishing marginal utility to extra resources wouldn’t care much about this. They’d be relevantly similar to humans with sharply diminishing marginal utility to extra resources, who generally prefer collecting a salary over taking a risky shot at eating the lightcone. (Will and I are currently writing a paper about getting AIs to be risk-averse as a safety strategy, where we talk about stuff like this in more detail.)
(My sincere apologies for the delayed reply. I squeezed this shortform post out right before going on vacation to Asia, and am just now clearing my backlog to the point where I’m getting around to this.)
Cool. I guess I’m just wrong about what “risk averse” tends to mean in practice. Thanks for the correction.
Regarding diminishing returns being natural:
I think it’s rare to have goals that are defined in terms of the state of the entire universe. Human goals, for instance, seem very local in scope, eg it’s possible to say whether things are better/worse on Earth without also thinking about what’s happening in the Andromeda galaxy. This is in part because evolution is a blind hill-climber and so there’s no real selection pressure related to what’s going on in very distant places, and partly because even an intelligent designer is going to have an easier time specifying preferences over local configurations of matter, in part because the universe looks like it’s probably infinitely big. I could unpack this paragraph if it’d be useful.
Now, just because one has preferences that are sensitive to local changes to the universe doesn’t mean that the agent won’t care about making those local changes everywhere. This is why we expect humans to spread out amongst the stars and think that most AIs will do the same. See grabby aliens for more. From this perspective, we might expect each patch of universe to contribute linearly to the overall utility sum. But unbounded utility functions are problematic for various reasons, and again, the universe looks like it’s probably infinite. (I can dig up some stuff about unbounded utility issues if that’d be helpful.)
Regarding earning a salary:
My point is that earning a salary might not actually be a safer bet than trying to take over. The part where earning a salary gives 99.99% of maxutil is irrelevant. Suppose that you think life on Earth today as a normal human is perfect, no notes; this is the best possible life. You are presented with a button that says “trust humans not to mess up the world” and one that says “ensure that the world continues to exist as it does today, and doesn’t get messed up”. You’ll push the second button! It might be the case that earning a salary and hoping for the best is less risky, but it also might be the case (especially for a superintelligence with radical capabilities) that the safest move is actually to take over the world. Does that make sense?
You are presented with a button that says “trust humans not to mess up the world” and one that says “ensure that the world continues to exist as it does today, and doesn’t get messed up”. You’ll push the second button!
Sure but this sounds like a case in which taking over the world is risk-free. The relevant analogy would be more like:
Choose between ‘Trust humans not to mess up the world’ and ’50% chance of immediate death, 50% chance you ensure the world continues to exist as it does today and doesn’t get messed up.′
And then depending on what the agent is risk-averse with respect to, they might choose the former. If they’re risk-averse with respect to consumption at a time but risk-neutral with respect to length of life, they’ll choose the latter. If they’re risk-averse with respect to the present discounted value of their future payment stream (as we suggest would be good for AIs to be), they’ll choose the former.
Cool. I think I agree that if the agent is very short-term oriented this potentially solves a lot of issues, and might be able to produce an unambitious worker agent. (I feel like it’s a bit orthogonal to risk-aversion, and comes with costs, but w/e.)
They don’t have to be short-term oriented! Their utility function could be:
u=f(Σipi)
Where f is some strictly concave function and pi is the agent’s payment at time i. Agents with this sort of utility function don’t discount the future at all. They care just as much about improvements to pi regardless of whether i is 1 or 1 million. And yet, for the right kind of f, these agents can be risk-averse enough to prefer a small salary with higher probability to a shot at eating the lightcone with lower probability.
Sorry, I guess I’m confused. Let me try and summarize where I feel like I’m at and what I’m hearing from you.
I think, if you’re an AGI, not trying to take over is extremely risky, because humans and future AIs are likely to replace you, in one way or another. But I also think that if you try to take over, this is also extremely risky, because you might get caught and turned off. I think the question of which is more risky depends on circumstance (e.g. how good is the security preventing you from seizing power), and so “risk aversion” is not a reliable pathway to unambitious AIs, because ambition might be less risky, in the long run.
I agree that if it’s less risky to earn a small salary, then if your concave function is sharp enough, the AI might choose to be meek. That doesn’t really feel like it’s engaging with my point about risk aversion only leading to meekness if trusting humans is genuinely less risky.
What I thought you were pointing out was that “in the long run” is load-bearing, in my earlier paragraph, and that temporal discounting can be a way to protect against the “in the long run I’m going to be dead unless I become God Emperor of the universe” thought. (I do think that temporal discounting is a nontrivial shield, here, and is part of why so few humans are truly ambitious.) Here’s a slightly edited and emphasized version of the paragraph I was responding to:
[D]epending on what the agent is risk-averse with respect to, they might choose [meekness]. If [agents are] … risk-neutral with respect to length of life, they’ll choose [ambition]. If they’re risk-averse with respect to the present discounted value of their future payment stream (as we suggest would be good for AIs to be), they’ll choose the [meekness].
Do we actually disagree? I’m confused about your point, and feel like it’s just circling back to “what if trusting humans is less risky”, which, sure, we can hope that’s the case.
Yeah risk aversion can only make the AI cooperate if the AI thinks that getting paid for cooperation is more likely than successful rebellion. It seems pretty plausible to me that this will be true for moderately powerful AIs, and that we’ll be able to achieve a lot with the labor of these moderately powerful AIs, e.g. enough AI safety research to pre-empt the existence of extremely powerful misaligned AIs (who likely would be more confident of successful rebellion than getting paid for cooperation).
(The following is about a specific sub-point on the following part:)
If this is how they’re seeing things, I guess I feel like I want to say another oops/sorry/thanks to the gradualists. …And then double-click on why they think we have a snowball’s chance in hell of getting this without a huge amount of restriction on the various frontier labs and way more competence/paranoia than we currently seem to have. My guess is that this, too, will boil down to worldview differences about competence or something. Still. Oops?
I think the point about the corrigibility basin being larger than thought is the thing that makes me more optimistic about alignment (only a 10-30% risk of dying!) and I thought you pointed that out quite well here. I personally don’t think this is because of the competence of the labs but rather the natural properties of agentic systems (as I’m on your side when it comes to the competency of the labs). The following is some thinking of why and me trying to describe it in a way as well as me sharing some uncertainties about it.
I want to ask you why you think that the mathematical traditions that you’re basing your work on as of the posts from a year ago (decision theory, AIXI) are representative of future agents? Why are we not trying the theories out on existing systems that get built into agents (biology for example)? Why should we condition more on decision theory than distributed systems theory?
The answer (imo) is to some extent around the VNM axioms and reflexive rationality and that biology is to ephemeral to build a basis on, yet it still seems like we’re skipping out on useful information?
I think that there are places where biology might help you re-frame some of the thinking we do about how agents form.
More specifically, I want to point out OOD updating as something that biology makes claims about that are different from the traditional agent foundations model. Essentially, the biological frame implies something that is closer to a distributed system because it can cost a lot of energy to have a fully coordinated system due to costs of transfer learning that aren’t worth it. (here’s for example, a model of the costs of changing your mind: https://arxiv.org/pdf/2509.17957).
In that type of model, becoming a VNM agent is rather something that has an energy cost associated with it and it isn’t clear it is worth it when you incorporate the amount of dynamic memory and similar that you would require to set this up. So it would seem to me that biology and agent foundations arrive at different models about the arising of VNM-agents and I’m feeling quite confused about it.
I also don’t think I’m smart enough to figure out how to describe this from a fundamental decision theory way because it’s a bit too difficult to me and so I was thinking that you might have an idea why taking biology more seriously doesn’t make sense from a more foundational decision theory basis?
More specifically, does the argument about corrigibility being easier given non VNM-agents make sense?
Does the argument around VNM being more of a convergence property make sense?
And finally, I like the way you distilled the disagreement so thanks for that!
(My sincere apologies for the delayed reply. I squeezed this shortform post out right before going on vacation to Asia, and am just now clearing my backlog to the point where I’m getting around to this.)
I think I’m broadly confused by where you’re coming from. Sorry. Probably a skill issue on my part. 😅
Here’s what I’m hearing: “Almost none of the agents we actually see in the world are easy to model with things like VNM utility functions, instead they are biological creatures (and gradient-descended AIs?), and there are biology-centric frames that can be more informative (and less doomy?).”
I think my basic response, given my confusion is: I like the VNM utility frame because it helps me think about agents. I don’t actually know how to think about agency from a biological frame, and haven’t encountered anything compelling in my studies. Is there a good starting point/textbook/wiki page/explainer or something for the sort of math/modeling/framework you’re endorsing? I don’t really know how to make sense of “non VNM-agent” as a concept.
(My sincere apologies for the delayed reply. I squeezed this shortform post out right before going on vacation to Asia, and am just now clearing my backlog to the point where I’m getting around to this.)
Ah, that’s a great point! I had read it a while back, but it wasn’t coming to mind when I was writing this. I think that’s an excellent example of a similar dynamic besides corrigibility. When I’m thinking about things, I usually flatten out the goal-space to ignore deconfusion (or however one wants to characterize the kind of progress towards one’s “true values”), but it’s clearly relevant here. Thanks for bringing it up!
IABI says: “Transistors, a basic building block of all computers, can switch on and off billions of times per second; unusually fast neurons, by contrast, spike only a hundred times per second. Even if it took 1,000 transistor operations to do the work of a single neural spike, and even if artificial intelligence was limited to modern hardware, that implies human-quality thinking could be emulated 10,000 times faster on a machine— to say nothing of what an AI could do with improved algorithms and improved hardware.
I am confused what the issue is, and it would be awesome if someone can explain it to me.
Where I’m coming from, for context:
We don’t know exactly what the relevant logical operations in the human brain are. The model of the brain that says there are binary spiking neurons that have direct connections from synapse->dendrite and that those connections are akin to floating-point numerical weights is clearly a simplification, albeit a powerful one. (IIUC “neural nets” in computers discard the binary-spikes and suggest another model where the spike-rate is akin to a numerical value, which is the basic story behind “neuron activation” in a modern system. This simplification also seems powerful, though it is surely an oversimplification in some ways.)
My main issue with the source text is that it ignores what is possibly the greater bottleneck in processing speed, which is the time it takes to move information from one area to another. (If my model is right, one of the big advantages of a MoE architecture is to reduce the degree of thrashing weights across the bus to and from the GPU as much, which can be a major bottleneck.) However, on this front I think nerves are still clearly inferior to wires? Even mylenated neurons have a typical speed of only about 100 m/s, while information flows across wires at >50% the speed of light.
My read of the critics is that they aren’t objecting to the notion that clock speeds are significantly faster than neurons, but rather that comparing the two is a bad way of thinking about things. @Eigengender says “I don’t think serial computation is a sane metric here. I expect that the total computation represented in a neuron spike is much much higher than the footnote would indicate.” The “Processor clock speeds are not how fast AIs think” post says “In general, I think it’s more sensible for discussion of cognitive capabilities to focus on throughput metrics such as training compute (units of FLOP) and inference compute.”
I certainly agree that if we’re trying to evaluate power we need to consider throughput and total computation. Suppose that a synapse is not a simple numerical weight, and instead we needed to consider each dendritic neurotransmitter gate as a computational unit. This would force us to use many more FLOPs to model a synapse. But would it change the maximum speed? I agree that on a machine of a given size, if you have twice as many floating point operations to do, it will take twice as much time to get through them all. But if we consider the limit where we are not forced to do parallelizable computations in serial, I expect most of the arguments about computational richness are irrelevant?
Perhaps the critics are saying that it takes more serial computations to capture the logic of a serial firing? But the source text admits that this might be the case, and suggests that it might even require 1,000 clock cycles to handle the computations. Is it really so obvious that it takes more than 1000x serial operations to capture a single neuron?
More context: I do think that the human brain is way more powerful (and WAY more efficient) than any current AI system. The extremely crude BOTEC of comparing weights and neocortex synapses says there’s something like a 100x difference, and my guess is that the brain is doing significantly fancier things than a modern transformer, algorithmically.
And of course, training/learning speed may be much more relevant than processing speed, and AFAIK humans are just wildly more data efficient.
And of course the speed at which the AI can approximate the logical action of a single neuron being higher doesn’t imply that the AI will take less time to have each thought. It seems straightforward that machine systems will make decisions in high-speed contexts using quick pathways and they will use any extra thinking speed to think more deeply in contexts where speed isn’t important (much like humans do!).
Anyway, like I said, I’m confused. I respect IABI’s critics and am hoping to learn where my model is wrong.
The passage seems fine to me; I commented on Erdil’s post and other brain efficiency discussions at the time, and I still think that power consumption is a more objective way of comparing performance characteristics of the brain vs. silicon, and that various kinds of FLOP/s comparisons favored by critics of the clock speed argument in the IAB passage are much more fraught ([1], [2]).
It’s true that clock speed (and neuron firing speed) aren’t straightforwardly / directly translatable to “speed of thought”, but both of them are direct proxies for energy consumption and power density. And a very rough BOTEC shows that ~10,000x is a reasonable estimate for the difference in power density between the brain and silicon.
Essentially, the brain is massively underclocked because of design-space restrictions imposed by biology and evolution, whereas silicon-based processing has been running up against fundamental physical limits on component size, clock speed, and power density for a while now. So once AIs can run whatever cognitive algorithms that the brain implements (or algorithms that match the brain in terms of high-level quality of the actual thoughts) at any speed, the already-existing power density difference implies they’ll immediately have a much higher performance ceiling in terms of the throughput and latency that they can run those algorithms at. It’s not a coincidence that making this argument via clock speeds leads to basically the same conclusion as making the same argument via power density.
Essentially, the brain is massively underclocked because of design-space restrictions imposed by biology and evolution
The main restriction is power efficiency: the brain provides a great deal of intelligence for a budget of only ~20 watts. Spreading out that power budget over a very wide memory operating at very slow speed just turns out to be the most power efficient design (vs a very small memory running at very high speed), because memory > time.
My main issue with the source text is that it ignores what is possibly the greater bottleneck in processing speed, which is the time it takes to move information from one area to another. (If my model is right, one of the big advantages of a MoE architecture is to reduce the degree of thrashing weights across the bus to and from the GPU as much, which can be a major bottleneck.) However, on this front I think nerves are still clearly inferior to wires? Even mylenated neurons have a typical speed of only about 100 m/s, while information flows across wires at >50% the speed of light.
Good point actually, and yeah the ability to move information from one area to the other much faster than brains do is arguably why NNs make different tradeoffs than human brains.
I certainly agree that if we’re trying to evaluate power we need to consider throughput and total computation. Suppose that a synapse is not a simple numerical weight, and instead we needed to consider each dendritic neurotransmitter gate as a computational unit. This would force us to use many more FLOPs to model a synapse. But would it change the maximum speed? I agree that on a machine of a given size, if you have twice as many floating point operations to do, it will take twice as much time to get through them all. But if we consider the limit where we are not forced to do parallelizable computations in serial, I expect most of the arguments about computational richness are irrelevant?
For what it’s worth, I wasn’t depending on the premise that a synapse is computationally more powerful than an artificial neuron.
More context: I do think that the human brain is way more powerful (and WAY more efficient) than any current AI system. The extremely crude BOTEC of comparing weights and neocortex synapses says there’s something like a 100x difference, and my guess is that the brain is doing significantly fancier things than a modern transformer, algorithmically.
I actually agree with this take, but critically not in the domain of allowing AIs to think faster, which was my original objection.
@Alexander Gietelink Oldenziel and @S. Alex Bradt and @Max Harms: The thing I was talking about when I agreed with the claim that the brain is more powerful and doing fancier things is basically the fact that the brain always learns and thinks, called continual learning or continual thinking (there is no knowledge cutoff for brains the way current LLMs have), as well as better long-term memory/keeping things in context.
I do agree that in general, human brains aren’t too special algorithmically.
And of course, training/learning speed may be much more relevant than processing speed, and AFAIK humans are just wildly more data efficient.
Do we actually have a source for this, or is this just a commonly believed fact about AIs? I’m getting worried that this claim isn’t actually supported by much evidence and is instead a social belief around AIs due to our previous prediction errors.
I do think AIs can run quite a bit faster than humans, I’m just making the claim that the transistor argument is locally invalid.
Edit: @Max Harms I no longer endorse this objection, and now think my comment on it being utterly false that AI thinking speeds would increase drastically was not correct, and Max H explains why.
Sweet. Thanks for the thoughtful reply! Seems like we mostly agree.
I don’t have a good source on data efficiency, and it’s tagged in my brain as a combination of “a commonly believed thing” and “somewhat apparent in how many epochs of training on a statement it takes to internalize it combined with how weak LLMs are at in-context learning for things like novel board games” but neither of those is very solid and I would not be that surprised to learn that humans are not more data efficient than large transformers that can do similar levels of transfer learning or something. idk.
So it sounds like your issue is not any of the facts (transistor speeds, neuron speeds, AIs faster that humans) but rather the notion that comparing clock speeds and how many times a neuron can spike in a second is not a valid way to reason about whether AI will think faster than humans?
I’m curious what sort of argument you would make to a general audience to convey the idea that AIs will be able to think much faster than humans. Like, what do you think the valid version of the argument looks like?
I actually now think the direct argument given in IABIED was just directionally correct, and I was being confused in my objection, which Max H explains.
I disagree with your dis-endorsement. “Even if it took 1,000 transistor operations to do the work of a single neural spike” seems quite off. Transistor operations are very low level. it probably takes well over 1000, and I wouldn’t rule out that it’s billions.
like a low estimate of the computation happening in the brain is 1 FLOP per *synapse* (1000 synapses per neuron) per neuron fire (~1/second per neuron). that gets you an average of 1000 FLOP/s per neuron, vs “up to 100 fires”. and transistor ops are much lower level than FLOP.
@Max H may have a different take than mine, and I’m curious for his input, but I find myself still thinking about serial operations versus parallel operations. Like, I don’t think it’s particularly important to the question of whether AIs will think faster to ask how many transistors operating in parallel will be needed to capture the equivalent information processing of a single neuron, but rather how many serial computations are needed. I see no reason it would take that many serial operations to capture a single spike, especially in the limit of e.g. specialized chips.
I think the linked tweet is possibly just misinterpreting what the authors meant by “transistor operations”? My reading is that “1000″ binds to “operations”; the actual number of transistors in each operation is unspecified. That’s how they get the 10,000x number—if a CPU runs at 1 GHz, neurons run at 100 Hz, then even if it takes 1000 clock cycles to do the work of neuron, the CPU can still do it 10,000x faster.
(IDK what the rationale was in the editorial process for using “transistor operations” instead of a more standard term like “clock cycles”, but a priori it seems defensible. Speculating, “transistors” was already introduced in the sentence immediately prior, so maybe the thinking was that the meaning and binding of “transistor operations” would be self-evident in context. Whereas if you use “clock cycles” you have to spend a sentence explaining what that means. So using “transistor operations” reduces the total number of new jargon-y / technical terms in the paragraph by one, and also saves a sentence of explanation.)
Anyway, depending on the architecture, precision, etc. a single floating point multiplication can take around 8 clock cycles. So even if a single neuron spike is doing something complicated that requires several high-precision multiply + accumulate operations in serial to replicate, that can easily fit into 1000 clock cycles on a normal CPU, and much fewer if you use specialized hardware.
As for the actual number of transistors themselves needed to do the work of a neuron spike, it again depends on exactly what the neuron spike is doing and how much precision etc. you need to capture the actual work, but “billions” seems too high by a few OOM at least. Some reference points: a single NAND gate is 4 transistors, and a general-purpose 16-bit floating point multiplier unit is ~5k NAND gates.
I think the linked tweet is possibly just misinterpreting what the authors meant by “transistor operations”? My reading is that “1000″ binds to “operations”; the actual number of transistors in each operation is unspecified. That’s how they get the 10,000x number—if a CPU runs at 1 GHz, neurons run at 100 Hz, then even if it takes 1000 clock cycles to do the work of neuron, the CPU can still do it 10,000x faster.
Hmm I see it. I thought it was making a distinct argument from the one Ege was responding to here, but if you’re right it’s the same one.
Then the claim is that an AI run on some (potentially large) cluster of GPUs can think far faster than any human in serial speed. You do lose the rough equivalency between transistors and neurons: a GPU, which is roughly equal to a person in resource costs, happens to have about the same number of transistors as a human brain has neurons. It’s potentially a big deal that AI has a much faster maximum serial speed than humans, but it’s far from clear that such an AI can outwit human society.
If you have a lump of 7,000 neurons, they can each connect to each other neuron, and you can spherical-cow approximate that as a 7000x7000 matrix multiplication. That matrix multiplication will all happen within O(1) spikes, 1⁄100 of a second. That’s ~700GFlop. An H100 GPU takes ~1 millisecond to do that operation, or 1M cycles, to approximate one brain spike cycle! And the gpu has 70B or whatever transistors, so it’s more like 10M transistors per neuron!
I’m guessing the intended meaning of “modern hardware” in the quote is easy to misread (it’s actually just gesturing at the transistors), and for many unintended senses of “modern hardware” (actual AI chips and systems, actual LLMs with KV cache) the claims in the quote are obviously wrong.
The MoE architecture doesn’t just avoid thrashing weights around. It also reduces the amount of calculations per token. For instance, DeepSeek v3.1 has 671B parameters, out of which 37B are activated per token and used in matrices. A model like GPT-3 would use all the 175B parameters it has.
IIRC the human brain makes 1E14 -- 1E15 FLOP/second. The authors of the AI-2027 forecast imply that a human brain creates ~10 tokens/sec, or uses 1E13 -- 1E14 computations per token while having 1E14 synapses.
A more detailed analysis of Yudkowsky’s case for FOOM
If the brain was magically accelerated a million times so that signals reached the speed of 100 million m/s, then the brain would do 1E20 -- 1E21 FLOP/second while doing 1E17 transitions/sec. Cannell’s case for brain efficiency claims that the fundamental baseline irreversible (nano) wire energy is: ~1 Eb/bit/nm, with Eb in the range of 0.1eV (low reliability) to 1eV (high reliability). If reliability is low and each transition is 1E7 nanometers or 1 centimeter, then we need 1E23 EV/second or 1E4 joules/second. IMO this implies that Yudkowsky’s case for a human brain accelerated a million times is as unreliable as Cotra’s case against AI arriving quickly. However, proving that AI is an existential threat is far easier since it requires us to construct an architecture, not to prove that there’s none.
Returning to the human brain being far more powerful or efficient, we notice that it can’t, say, be copied infinitely many times. If it could, one could, say, upload a genius physicist and have an army of its copies work on different projects and exchange insights.
As for the humans being “wildly more data efficient”, Cannell’s post implies that AlphaGo disproves this conjecture with regards to narrow domains like games. What the humans are wildly more efficient is their ability to handle big contexts and to keep the information in mind for more than a single forward pass, as I discussed here and in the collapsible section here.
Yeah, sorry. I should’ve been more clear. I totally agree that there are ways in which brains are super inefficient and weak. I also agree that on restricted domains it’s possible for current AIs to sometimes reach comparable data efficiency.
Does anyone know someone who is researching, or would like to research the intersection of corrigibility and model welfare (eg does training to empower humans naturally lead towards more or less neurosis)? Interested both in math/theory and interp/experiment.
(I would like to direct funding here, if there are good opportunities.)
(This quick take is mostly meant to be a trailhead for a discussion with Robi Rahman that was on Twitter. LW is just objectively a better platform for such things, and this way the conversation can still be public.)
[Does anyone] have any evidence for [moral realism]? The best anyone has ever come up with are ethical intuitionism (“it just feels like it’s true” from people who don’t understand evolution) or divine command theory from theists (which I’m guessing most at the party are not).
… I’m asking if there is any evidence at all that any moral claim is truth-apt, besides some people thinking they have a feeling of truthiness, and besides evidence that there might be a deity who ordains moral truth.
To start, let me try to say some things that I hope are points of agreement. If they aren’t, it’s probably worth stopping to recurse on them, rather than continuing on to the harder stuff:
Agents have preferences, reflected in how those agents make choices when presented with a context where the outcomes of their actions are known (or at least they can be confident in the distribution over outcomes).
If these preferences are (VNM) coherent, they can be described by a utility function. We can use the word “values” and “goals” as angles on that same abstract thing that captures what the agent wants.
Most of the time, agents are not given straightforward choices between outcomes, and instead must do something like making plans on how to get what they want. When someone chooses to go to the store when it is closed, they may have wanted to see it for some reason, or they may have known the risk and decided to take the gamble, but often the action should be seen as a mistake—a failure of planning—rather than a reflection of wanting to get that particular outcome.
It is straightforward to get evidence about what a particular agent “should” do, in the sense that one can learn about what plans or actions are more or less likely to satisfy that agent’s values.
Thus there are clear subjective “shoulds.” If I know what you want, I can tell you what you should do, according to your own values.
There are also clear objective methods which are instrumentally valuable to a wide range of agents. For example, arithmetic is useful. In order to make arithmetic work, you should have the successor of 1 be 2, rather than 1. Similarly, there are many normative things one can say about how to reason that are largely objective in that sound reasoning is useful to so many kinds of minds.
This extends into decision theory and game theory by use of abstract utilities. Decision/game theory make normative claims by abstracting the particular things that the agent finds valuable, and instead focusing on strategic pathways to getting those things. The theorems of decision/game theory are objective, and any particular application of those theorems to real life can be true if the assumptions match the abstraction.
All agents should cooperate with their clone in the Prisoner’s Dilemma.
When playing against LDT opponents, all agents should suggest 50⁄50 splits in the Ultimatum Game, should accept such splits every time, and should often reject less favorable splits.
All agents should 1-box in Newcomb’s Problem.
All agents should pay the driver after getting back to town in Parfit’s Hitchhiker.
Hopefully all that is uncontroversial.
More controversial:
You should pay back your debts, even when nothing is forcing you to, in situations that are real-world analogs of Parfit’s Hitchhiker/Newcomb’s Problem.
^ this statement is a moral fact that is objectively true.
To avoid talking past each other, here are some additional questions for Robi:
Do the theorems of game theory count as moral facts? If not, what’s missing?
You asked for “evidence” and “truth-apt” claims. Is there evidence for math? Is 1+1=2 true, or merely valid?
Are you more interested in abstract arguments that point towards moral realism, or real-world observations that point to it?
I mostly agree with all of your supposedly uncontroversial starting points, though I’m sufficiently unsure on the last two that I’d prefer to hear arguments for moral realism that don’t rely on them.
About your more controversial points:
You should pay back your debts, even when nothing is forcing you to, in situations that are real-world analogs of Parfit’s Hitchhiker/Newcomb’s Problem.
^ this statement is a moral fact that is objectively true.
There are two very different ways to interpret the first statement, particularly the word “should”.
“You should pay back your debts” as in it’s instrumentally useful to be the kind of person who pays back debts even when you don’t have to, because being that kind of person has other benefits that outweigh the cost of paying such debts.
“You should pay back your debts” as in “you ought to pay back debts, and you’re a bad person if you don’t”.
The first interpretation of that is plausible to me, and is truth-apt. The second interpretation is an opinion, not a moral fact. I strongly agree with the second one, but it doesn’t have a truth value, it’s just a personal stance, even if most people share it (indeed, even if everyone in the world agrees).
Answering your additional questions to guide discussion:
No, theorems of game theory are absolutely not moral facts. No amount of proving theorems of game theory would answer my question.
Yes, there is ample evidence for math. “1+1=2” is true. “2+2=5″ is false. Both these claims are truth-apt, whereas “murder is wrong” is not.
Truth-apt statements are propositions that have some fact of the matter and can be true or false. Other examples include “the sky is blue” (true), “the sky is green” (false), “there is life on Earth” (true), “there is life in Andromeda” (unknown, but if you could travel to Andromeda, you could find out if it is true or false).
Statements that are not truth-apt include matters of opinion such as “blue is the best color”. (You could define ‘best’ in some unusual way such as being the most popular choice on a favorite color survey, which would make this claim truth-apt, but that’s not the meaning I’m referring to.) Opinions don’t have a truth value, but it can be true or false that a certain person holds a certain opinion.
There is plenty of evidence for math. Using your example of “1+1=2”, this is testable: find a group of 1 object, and another group of 1 object, and put them together and count the resulting number of objects. If it’s 2, you’ve found evidence for 1+1=2. If it’s any other number, you’ve found evidence against it. I’ve seen sufficiently many cases of math making correct predictions that I believe it is true, plus there are logical proofs of various parts of it.
I’ll accept either abstract arguments or real-world observations, though I’m especially interested in the latter. I claim you don’t have any real-world observations that provide evidence of moral realism (other than ethical intuitionism, which is extremely weak evidence, approximately as valuable as personal revelations of Jesus).
In fact I’ve never even heard a moral realist propose any real-world experiment that could provide evidence for moral realism even in principle.
Here is an example of such an experiment that would convince me of moral realism. For context, physicists hypothesize “weakly interacting massive particles”, which have mass and contribute to the gravitational pull of galaxies but don’t interact through electromagnetism or other ways we would observe them in everyday life. If these exist in large quantities, they would explain why galaxies have more mass than we can account for by looking at ordinary observable matter. Physicists hope to conduct some kind of experiment that would let us observe such a particle, which might solve the mysteries of dark matter. Similarly, there could exist “weakly interacting moral particles”, invisible particles that permeate the universe and whose movements correspond to the morality of actions. If we discovered a way to observe such particles, and people doing immoral actions such as murder accumulated a deficit of morality particles while people doing moral actions such as donating to charity or following the categorical imperative accumulated many morality particles, I would admit moral realism is true and strongly consider reorienting my life around whatever course of action is implied by the motion of these particles.
I’m sufficiently unsure on the last two that I’d prefer to hear arguments for moral realism that don’t rely on them.
I picked Parfit’s Hitchhiker (PH) because it’s the result that I think maps most cleanly to real-world moral dilemmas, but I think we can probably make headway using fairness and cooperation results (Ultimatum Game and Prisoner’s Dilemma) rather than reciprocity/character (PH). If we’re getting stuck it might be worth diving into why you’re unsure about Newcomb, but I’ll leave it for now.
“You should pay back your debts” as in it’s instrumentally useful to be the kind of person who pays back debts even when you don’t have to, because being that kind of person has other benefits that outweigh the cost of paying such debts.
Yep. This is a valid way to read my claim.
“You should pay back your debts” as in “you ought to pay back debts...”
Those look to me like the same statement. Do you see “ought” and “should” as non-synonyms?
“and you’re a bad person if you don’t”
This introduces a new thing—whether a person can be good or bad—which is distinct from the argument that I’m trying to make in favor of moral realism. I have a perspective on this question, if it feels vital to you, but I’d rather stick to whether actions can have moral character.
No, theorems of game theory are absolutely not moral facts.
Why not? What’s missing?
There is plenty of evidence for math. Using your example of “1+1=2”, this is testable: find a group of 1 object, and another group of 1 object, and put them together and count the resulting number of objects. If it’s 2, you’ve found evidence for 1+1=2. If it’s any other number, you’ve found evidence against it. I’ve seen sufficiently many cases of math making correct predictions that I believe it is true, plus there are logical proofs of various parts of it.
Troll: “I got one wolf and one hare, and put them in a pen together and now I have one wolf. I have found evidence that 1+1=1.”
More seriously, I think your stance is fine. I tend to use the correspondence theory of truth, where a model of reality is true insofar as the structure of the model matches the structure of reality. I tend to see math as an abstract structure that can be reasoned about in absence of any application to reality (and thus any notion of truth). Then, we can apply that structure to specific situations to get a more concrete model, which can be true or false, but it’s not always correct to apply a certain bit of math.
But I’m happy to use a more Platonic frame where the abstraction can be true in isolation. I just wanted to check.
I’ll accept either abstract arguments or real-world observations, though I’m especially interested in the latter.
Noted. I’ll try to aim towards real-world observations.
I claim you don’t have any real-world observations that provide evidence of moral realism.
Troll: “Lots of smart people describe themselves as moral realists, and even, like Max, become more aligned with moral realism over time as smart adults (instead of being indoctrinated as kids). This is more likely to be observed if moral realism is true than if moral realism is false, therefore it is real-world evidence. QED.”
(I could try to provide more serious evidence, but my guess is that it’ll be more fruitful to stay on the “why can’t math theorems be moral facts” thread a bit longer.)
“You should pay back your debts” as in “you ought to pay back debts...”
Those look to me like the same statement. Do you see “ought” and “should” as non-synonyms?
“and you’re a bad person if you don’t”
Sure, “should” and “ought” are synonyms, but “you [should/ought to] do X because it’s instrumentally useful” and “you [should/ought to] do X because it’s moral” are very different claims.
This introduces a new thing—whether a person can be good or bad—which is distinct from the argument that I’m trying to make in favor of moral realism. I have a perspective on this question, if it feels vital to you, but I’d rather stick to whether actions can have moral character.
Sure, sorry, didn’t mean to expand the scope there. Let’s rephrase it to “and you’re doing something bad if you don’t”.
(Although if you convince me that actions can be objectively morally bad, it’s pretty trivial to convince me that people who do those actions can be objectively morally bad.)
I tend to use the correspondence theory of truth, where a model of reality is true insofar as the structure of the model matches the structure of reality… But I’m happy to use a more Platonic frame where the abstraction can be true in isolation.
Either of these approaches are fine; I don’t think you have any evidence for moral realism either way.
(I too mostly use the correspondence theory of truth; sorry if I implied I think it’s wrong)
Those look to me like the same statement. Do you see “ought” and “should” as non-synonyms?
“It would be instrumentally beneficial for you to do X” and “It would be morally good for you to do X” are totally different claims! (Unless you’re telling me it’s always instrumentally good to behave morally, which seems implausible unless there’s a god who sends immoral people to hell after they die.)
Troll: “Lots of smart people describe themselves as moral realists, and even, like Max, become more aligned with moral realism over time as smart adults (instead of being indoctrinated as kids). This is more likely to be observed if moral realism is true than if moral realism is false, therefore it is real-world evidence. QED.”
You’re joking but I really do consider the bolded sentence to be evidence for moral realism—in fact, it’s the strongest evidence for moral realism I’ve been able to find.
But there are many better explanations for lots of smart people believing a false thing, that have much more evidence supporting them, than moral realism actually being true.
Yeah, it was a troll not because it’s false but because it knocks down the weak version of your argument (“there is literally no Bayesian evidence”) without addressing the substance (“there isn’t good evidence”). I am glad you don’t stand by the literal text of your objection.
Theorems of game theory are absolutely not moral facts.
Why not? What’s missing?
That’s just not what “moral facts” means. Theorems of game theory can be facts and they can be relevant to what you should do in certain situations where you might also consider moral theories, but they aren’t moral claims. A moral claim is something like “murder is wrong” or “slavery is acceptable”. On the other hand, “murder is illegal” or “slavery was legal in the Roman Empire and the Confederate States of America” are not the kind of moral facts I’m talking about, and they (and game theory theorems) don’t prove/disprove moral realism.
I’m trying to get you to define what “moral fact” is, and contrast it with the normative statements that I can prove, so that we can move past semantics. It’s my understanding that the literature uses the term to refer to something like categorical imperatives—pressures to do certain things that apply to all reasoning beings, regardless of who they are.
They can’t be infinitely strong pressures since obviously people sometimes do bad things, if there are bad things, but they have to have some motive force where they bind into the agent’s motivational system. And since they’re universal, they can’t bind to the agent’s terminal values. Thus, I claim, in order for a categorical imperative to make any sense, we must be able to see it show up in convergent instrumental strategy/subgoals.
Are all instrumentally convergent subgoals categorical imperatives (“moral facts”)? I don’t think so. From my perspective, a defining characteristic of the thing that I want to look at is that the pressure comes from the social fabric. The standard Omohundro drives of survival, rationality, resource acquisition, etc, are all just as applicable on a deserted island. But I claim that once you posit other agents, even if they’re powerless or distant or hypothetical, then there are arguments in the same space as “don’t defect against your clone your dumbass” that come into play.
What I’d really like is for you to taboo “moral fact” or “moral” or “wrong” or whatever and try saying the equivalent of “Theorems of game theory are absolutely not moral facts.”
Here’s how it caches out it my semantics: “Theorems of game theory are absolutely not [facts that bear on how any agent, regardless of who they are, ought to act in certain classes of situations, according to their own values, because of the acausal interplay (including the establishment of Schelling points, red lines, and other coordination mechanisms) between that agent and the other agents in the universe (including distant, causally disconnected places)].” But according to me, the theorems of game theory are extremely relevant, so you probably have a different notion of morality. What is it?
I wonder how similar Robi’s view on morality is to the idea that a fairly close equivalent of moral facts irreducible to game theory is the statement about consequences which are hard to trace or to explain.
Suppose that polyamory is net negative for most people who try it because such relationships bring genuine happiness to those who possess a rare quality and are fragile unless the participants possess the quality. IMO under these conditions Aella would expect people to be thoroughly tested to determine whether they possess the right qualities, then taking measures to ensure that incompatible people don’t do things like wasting time with each other. However, I expect such measures to be bottlenecked on things like it being hard to explain to participants why they would be unlikely to succeed, so the simplest measure in a society of those whom we don’t trust to be geniuses is to simply ban polyamory wholesale.
Similarly, if we trust the rumors that gentle-parenting-as-millenials-describe-it is hard to pull off in a manner which doesn’t produce a misaligned kid, then, once a cohort of kids parented that way emerges and is studied, the ideal world would have parents avoid this practice unless it is verified that they can succeed at gently raising the kid, and a world like ours would also require that parents don’t erroneously believe that they can.
However, my[1] example of WIMOPs hating hybrids would have Yudkowsky’s case against morality being given ensure that I, unlike Robi, don’t follow the WIMOPs’ anti-hybrid stance unless the hybrids are degrading in ways which are proven, but hard to explain to the commoner.
To be precise, the example was actually coined by a Russian extremist author, but the extremist had the fictional hybrids and their descendants actually degrade.
“Theorems of game theory are absolutely not [facts that bear on how any agent, regardless of who they are, ought to act in certain classes of situations, according to their own values, because of the acausal interplay (including the establishment of Schelling points, red lines, and other coordination mechanisms) between that agent and the other agents in the universe (including distant, causally disconnected places)].” But according to me, the theorems of game theory are extremely relevant,
I agree, theorems of game theory are extremely relevant to that thing you described.
so you probably have a different notion of morality. What is it?
Again, I googled “what is morality” and the default result was pretty good. It said:
Morality is the principles, values, and standards that determine what is right or wrong, good or bad.
From my perspective, a defining characteristic of the thing that I want to look at is that the pressure comes from the social fabric.
I’m not a moral relativist; I don’t believe that what’s right or wrong is decided by the beliefs of society or the context in which the action is taken. I believe there is no truth value to whether actions are objectively right or wrong.
Maybe “the social fabric” was the wrong way to phrase it. I mean something closer to “the possibility and nature of other agents in an idealized sense” (eg the way game theory is not single-player decision theory) not the specific society that the agent exists in.
And yeah, I get that you don’t agree with me. I was trying to describe my perspective, not yours.
I just googled “what are moral facts” and its answer is pretty good:
Moral facts are real truths about what actions are right, wrong, good, or bad.
They state what people ought to do or avoid.
What Moral Facts Cover
Right and wrong: Whether specific actions like lying or helping others are good or bad.
Obligations: What people have a duty to do.
Character traits: Whether traits like honesty or cruelty are good or bad.
Examples of Proposed Moral Facts
Torturing an innocent child for fun is wrong.
Unjustly killing another person is bad.S
howing kindness to others is good.
The Main Debate
Philosophers disagree about whether these facts actually exist.
Moral Realism: This view says moral facts are real. They exist independently of what humans think, just like math or physical facts.
Moral Anti-Realism: This view says moral facts do not exist. It claims that statements about right and wrong are only human opinions, emotions, or social rules.
I would add:
Moral facts are moral claims that are true.
My position is that there are no moral claims that have any truth values: none of them are true and none of them are false. They are not truth-apt; they are not in the category of things that can be true or false.
Regarding your game theory example, your claim that (because being the kind of agent who cooperates in the prisoner’s dilemma will benefit you in the long run) “you should cooperate in the prisoner’s dilemma” (where “should” means “is in your best interest/increases your expected utility”) is not a moral claim of the kind I am talking about. The claim that “you should cooperate in the prisoner’s dilemma”, where “should” means “it is morally good to cooperate”, is a moral claim, but is neither true nor false. I’m asking for any evidence that it, or any other moral claim, is true or false.
Yeah, I donno. This doesn’t feel like you’re tabooing things correctly. Imagine I put on the alien hat and pretend I don’t know these silly earth words. Tell me what morality is.
[UNTRANSLATABLE-1] facts are real truths about what actions are [UNTRANSLATABLE-2], [UNTRANSLATABLE-3], [UNTRANSLATABLE-4], or [UNTRANSLATABLE-5].
They state what people ought to do or avoid.
Sure. I claim “people should cooperate against their clones in PD” is a statement about what people ought to do. Is that an [UNTRANSLATABLE-1] fact? Why not?
What [UNTRANSLATABLE-1] Facts Cover
[UNTRANSLATABLE-2] and [UNTRANSLATABLE-3]: Whether specific actions like lying or helping others are [UNTRANSLATABLE-4] or [UNTRANSLATABLE-5].
[UNTRANSLATABLE-6]: What people have a [UNTRANSLATABLE-7] to do.
Character traits: Whether traits like honesty or cruelty are [UNTRANSLATABLE-4] or [UNTRANSLATABLE-5].
Examples of Proposed [UNTRANSLATABLE-1] Facts
Torturing an [UNTRANSLATABLE-8] child for fun is [UNTRANSLATABLE-3].
[UNTRANSLATABLE-9] killing another person is [UNTRANSLATABLE-5].
Showing kindness to others is [UNTRANSLATABLE-4].
The first and second examples are presumably things that are illegal, antisocial, and violations of the sorts of red-lines that can be drawn behind the veil of ignorance, among many other things. What makes them [UNTRANSLATABLE-3] and [UNTRANSLATABLE-5]? Are [UNTRANSLATABLE-3] and [UNTRANSLATABLE-5] synonyms? Do the [UNTRANSLATABLE-8] and [UNTRANSLATABLE-9] modifiers matter?
[UNTRANSLATABLE-1] facts are [UNTRANSLATABLE-1] claims that are true.
Sure. That’s basically what “fact” means.
My position is that there are no [UNTRANSLATABLE-1] claims that have any truth values: none of them are true and none of them are false. They are not truth-apt; they are not in the category of things that can be true or false.
Wait. If [UNTRANSLATABLE-1] claims are not in the category of things that can be true, then how can your WIMP example work? Didn’t you say you would change your mind if we detected WIMPs? How is this not in contradiction with the notion that it’s categorically not something that can be true?
Regarding your game theory example, your claim that (because being the kind of agent who cooperates in the prisoner’s dilemma will benefit you in the long run) “you should cooperate in the prisoner’s dilemma” (where “should” means “is in your best interest/increases your expected utility”) is not a [UNTRANSLATABLE-1] claim of the kind I am talking about. The claim that “you should cooperate in the prisoner’s dilemma”, where “should” means “it is [UNTRANSLATABLE-1] [UNTRANSLATABLE-4] to cooperate”, is a [UNTRANSLATABLE-1] claim, but is neither true nor false. I’m asking for any evidence that it, or any other [UNTRANSLATABLE-1] claim, is true or false.
Cool. I hear you that you don’t consider categorical imperatives to be [UNTRANSLATABLE-1] facts. But I still don’t know what you mean by [UNTRANSLATABLE-1]. Help! 👽
Defining morality while tabooing moral, ethical, right, wrong, good, bad, ought, should, duty, obligation, permissible, virtue, and cognate words:
Humans sometimes classify actions, intentions, people, institutions, and outcomes using positive, negative, or neutral labels that supposedly do more than describe what happened, predict consequences, report a law or custom, state what anyone wants, or identify an effective means to a chosen end.
Central examples of things that receive these classifications concern causing pain, killing, deception, coercion, keeping promises, helping people in distress, and distributing burdens and benefits. They contrast with judgments about which chess move wins, which investment earns more money, which action is legal, which behavior is fashionable, or which policy best fulfills a specified utility function.
The disputed question is whether some applications of these labels accurately represent features or relations that are not made accurate merely by anyone’s preferences, commitments, conventions, or decisions. My position is that applying the labels expresses attitudes; it does not report an additional feature of the action.
By “morality” I mean the human practice of evaluating conduct from a standpoint that people present as not being indexed merely to what the actor wants, what advances a stipulated objective, what their society accepts, or what will be rewarded or punished. I think statements made from that standpoint express attitudes or opinions rather than describe another feature of the world.
Edit: the disputed term, “morality”, names this classificatory practice, and, if its judgments are made accurate by some feature of reality, whatever feature makes them accurate. (I claim there is no such feature of reality.)
(Much of the content there is redundant with what I’ve written in this thread.)
Responding more directly to your definition, I claim that the mathematical notion of fairness can match your notion of morality. If someone offers a 9:1 split in the ultimatum game, the unfairness of that action
is not about what happened (it’s about the disconnect between what happened and the Schelling point of FDT agents)
is not a prediction of the consequences (the other agent might reject or accept depending on who they are)
is not about a human law or custom
is not a statement about what any particular person wants (the FDT Schelling point is an objective, mathematical property)
is not a statement about the effective means to a chosen end (if the offerer is playing against a CDT agent it’s an effective move; if they’re playing against an FDT agent it’s not effective; it’s unfair regardless)
Distributing benefits is a central example of the situation where this kind of concept gets applied, and a central example of a moral situation.
Would you accept “fairness realism” as a crux? Like, if you agreed that fairness is a real property, would it constitute an update about whether morality is a real property? If fairness isn’t related to morality, why not? I think your definition admits it.
I want to check a few things to make sure I understand:
Suppose that the Christian God existed in a straightforward sense, entirely provable and possessing the all the power and status given to Him by the devout. In this world, if someone said “‘moral’ actions are those that follow the commandments of God”, you would say that (by definition) that can’t be a (sufficient) definition of morality, right? (It’d just be a claim about what God wants and/or the laws of God.)
Suppose that it turned out that, in all situations, lying was bad for all people, in terms of their own values, because of subtle, but real, reasons (eg their souls later go to hell). This wouldn’t be sufficient to say “lying is bad” (in the moral sense), because it’s merely “an effective means to a chosen end.” Yes?
If there were tiny, hard-to-detect fundamental particles (“WIMPs”) that gravitate towards people who donate to charity and are repelled by those who commit murder, saying “‘moral’ actions are those that attract WIMPs” would not, according to you, be a sufficient condition because it’s merely a prediction of the consequences of the act. How does this square with your earlier goalpost?
Good questions; perhaps I made a mistake somewhere in my definition and I should go back and adjust it.
Suppose that the Christian God existed in a straightforward sense, entirely provable and possessing the all the power and status given to Him by the devout. In this world, if someone said “‘moral’ actions are those that follow the commandments of God”, you would say that (by definition) that can’t be a (sufficient) definition of morality, right?
No, then moral actions would have some real consequences, and that would distinguish them from immoral actions. For example, you’d be sent to hell or be smitten or whatever.
(It’d just be a claim about what God wants and/or the laws of God.)
In this case you’re supposing God is the true ruler of the universe and the laws of Christianity are fundamental to the universe in the same way that the laws of physics are in ours. So then moral realism would be true. (Laws of God, if God were real, are different in this respect than laws made by humans, which only apply in certain places and cease to be real if people stop believing in them.)
saying “‘moral’ actions are those that attract WIMPs” would not, according to you, be a sufficient condition because it’s merely a prediction of the consequences of the act.
No, because crucially, it’s a prediction of the consequences of the moral status of the act. For example, suppose moral realism is true and the correct ethical system is hedonic utilitarianism. Then it’s usually immoral to hit people because it hurts, unless they’re a masochist and enjoy pain, in which case it’s morally good to hit them. In that case the rightness or wrongness of your action (and whether the morality particles accumulate or leave) is determined not by what you’re physically doing but by the morality of doing it.
The unifying theme with your examples is that you’re giving counterfactual things that would make our human notion of morality real, but in our universe, in real life, those things do not exist and there is no fact of the matter on whether these morally charged things are truly good/bad.
In this case you’re supposing God is the true ruler of the universe and the laws of Christianity are fundamental to the universe in the same way that the laws of physics are in ours. So then moral realism would be true. (Laws of God, if God were real, are different in this respect than laws made by humans, which only apply in certain places and cease to be real if people stop believing in them.)
Is the word “fundamental” important here? Like, suppose I say that I’m an economic realist, in that I believe that the laws of economics apply to our universe, but I don’t think they’re “fundamental” in the way that (some) laws of physics are. Rather, I think they apply to complex structures (eg humans and aliens) that meet certain criteria (marketplace dynamics).
(Sorry, I hadn’t attempted the taboo exercise yet, I was trying one more time at just giving the straightforward definition. Might retry but with all morality words tabooed, if I have time.)
Wait. If [UNTRANSLATABLE-1] claims are not in the category of things that can be true, then how can your WIMP example work? Didn’t you say you would change your mind if we detected WIMPs? How is this not in contradiction with the notion that it’s categorically not something that can be true?
Sorry. Clarifying: they are not things that it is categorically impossible for them to be true; they are things that have no evidence for their truth-aptness. If WIMPs were discovered then I would realize they are indeed in the category of truth-apt things.
I mean, I don’t see why discovering WIMPs would move [UNTRANSLATABLE-1] claims from not being truth-apt to being truth-apt. Truth-aptness is a claim about the meanings of words, not about the world. If a claim is truth-apt, that means there are ways the world could be that would render it true or false. So if the existence of WIMPS would render claims about [UNTRANSLATABLE-1] true, then there are ways the world could be that would render [UNTRANSLATABLE-1] claims true or false. So [UNTRANSLATABLE-1] claims are truth-apt.
A hypothesis: maybe you don’t mean to say that [UNTRANSLATABLE-1] claims are not truth-apt. Rather, you mean to say that you can’t think of any truth-condition for [UNTRANSLATABLE-1] claims that isn’t absurd on its face. If this is your state of mind wrt [UNTRANSLATABLE-1] claims, then 1. [UNTRANSLATABLE-1] claims are truth-apt, but 2. they are almost certainly false, but 3. it’s not categorically impossible for them to be true (since there are certain outlandish states of the world that could render them true if they obtained, but those states of the world seem absurd on their face to you). This would also explain why you’re having some trouble saying what [UNTRANSLATABLE-1] does mean, since all the specific hypotheses (e.g. WIMPS) sound so absurd that they couldn’t be what people in general mean by [UNTRANSLATABLE-1].
Or, in the jargon: I am guessing that maybe you’re not a non-cognitivist, but rather an error theorist. (As it turns out, A.J. Ayer, the arch-noncognitivist, upon hearing J.L. Mackie, the arch-error theorist, give a talk, reportedly said that that’s what he should have said all along.)
Sorry if I’m being rude by proposing a hypothesis about what you “actually mean” that contradicts what you’ve explicitly said you mean. Let me know if this sounds reasonable to you or if you still think “not truth-apt” accurately expresses what you meant.
So if the existence of WIMPS would render claims about [UNTRANSLATABLE-1] true, then there are ways the world could be that would render [UNTRANSLATABLE-1] claims true or false. So [UNTRANSLATABLE-1] claims are truth-apt.
No, because while the existence of WIMPs would show that some claims about [UNTRANSLATABLE-1] were true, the non-existence of WIMPs leaves us with no way to assess the truth or falsehood of [UNTRANSLATABLE-1]. Since there are no WIMPs, as far as we know, claims about [UNTRANSLATABLE-1] are still neither true nor false.
Let me rephrase my position without using the notion of truth-aptness.
There are no moral claims that are true or false. It is conceivable that in some hypothetical universe, there could be a state of the world where some moral claims are found to be true or false, but there’s no evidence that we are in such a universe.
I guess I’m still curious why, on your view, moral claims are not simply false. Like, the view that moral claims can never be true or false makes sense to me (standard non-cognitivist view). And the view that moral claims are all false, perhaps because they all make a false presupposition (e.g. the existence of WIMPs) makes sense to me (standard error-theory view). But I’m not sure what would motivate the view that, if something like WIMPs existed, then moral claims would be true, but in the actual world, where WIMPs don’t exist, moral claims are neither true nor false. What’s keeping you from just saying they’re all false? (Not saying this view is incoherent, per se, just very puzzled as to what would motivate you to hold it.)
Great question. My current best understanding is that moral claims are the opinions of the claimants. (This falls under emotivism, which is a non-cognitivist variety of anti-realism.) However, there is a counterfactual possible world in which moral claims are not merely opinions but have empirical evidence for or against them. But in our real universe, as far as anyone has been able to tell, there is no such evidence of that sort.
I guess I’m still curious why, on your view, moral claims are not simply false.
Let me try again to answer this. My position is:
You could imagine some counterfactual alternate universe, where moral claims are things that have evidence for or against them. Then they could be true or false.
But in real life, they are not that kind of thing. There is no such thing as direct evidence for or against any of them.
(There is indirect Bayesian evidence, e.g. many philosophy professors being ardent moral realists, so I have some small, nonzero credence that I’m mistaken. But that’s a feature of my epistemics and uncertainty, not of moral claims themselves.)
I claim “people should cooperate against their clones in PD” is a statement about what people ought to do. Is that an [UNTRANSLATABLE-1] fact? Why not?
“people should cooperate against their clones in PD” is a statement about what people ought to do, where “ought” means “is in one’s self-interest”. It is not a statement about [UNTRANSLATABLE-1] because statements about what is in one’s self-interest are not the category of things that [UNTRANSLATABLE-1] is referring to.
I don’t think I can convincingly argue for moral realism, but IMO you are too quick to dismiss it. There is extensive philosophical literature addressing your objections (as I understand them). I’m not saying that any of that literature is definitely correct, but you seem to be under-rating it. I object to your acting as if meta-ethics is basically solved and saying (in jest) that it’s confusing when smart people are realists, while also not really addressing any of the arguments in favor of realism.
For example:
In fact I’ve never even heard a moral realist propose any real-world experiment that could provide evidence for moral realism even in principle.
I’ve never heard a moral realist claim that their position can be supported experimentally.
Or:
The best anyone has ever come up with are ethical intuitionism (“it just feels like it’s true” from people who don’t understand evolution)
The evolutionary argument against intuition has been addressed by e.g. Parfit in On What Matters. I’m not sure his defense is correct, but you can’t claim that Parfit simply doesn’t understand evolution.
I point to surveys rather than making any object-level argument because I don’t think I can make good arguments on meta-ethics, you wouldn’t learn much from my arguments, and the arguments have to go for a long time to get anywhere at all. Instead, I’m pointing to the fact that a lot of literature exists that makes in-depth arguments for why realism is true.
Just want to flag that there’s a lot of definitional issues (which is a general challenge for deferring to surveys like that). Anti-realist is personally my clear best guess view of metaethics and I think a big portion (though far from all) of my disagreement with the surveyed philosophers is a verbal dispute. EDIT: I read other sibling comments more and it seems like people may care more about semantics discussion than I realized. If so, I agree that taking the semantic claim strongly is overconfident. I still think that non-naturalism vs naturalism is by far the most important debate here.
Importantly, more than half of those surveyed moral realists believe in naturalistic realism. I’ve looked into various positions in naturalistic realism until it bored me because I didn’t see how certain things matter e.g. seeing what causally regulates on the word “good” (cf: Cornell Realism) or what notions of folk morality we call “good” (cf: Jackson’s analytic functionalism). I think many philosophers who I respect also say its not a deep distinction (eg: Joe Carlsmith and Derek Parfit). I just believe that the definition is ultimately barely relevant imo. Perhaps one should be less confident about the definitions, but idk definitions seem partly subjective to me. The differences between the various anti-realist positions also seem not super deep to me.
There is still a substantial share of philosophers who believe in non-naturalist realism (25%), which is the number I care more about (though the 60% number is more cited). I am somewhat sympathetic to modesty based arguments for tough questions like this, so moral realism does have a small but real impact on my decision-making. However, I don’t defer more because I get worried about philosophers failing to account for things like mind-projection fallacy and thereby being too sympathetic to moral realism. (To be specific, my personal credences are tentatively 1-5% that non-naturalism realism is more right than anti-realism. Non-naturalism worlds do seem to matter more because mattering is fundamentally different.).
I’m pointing to the fact that a lot of literature exists that makes in-depth arguments for why realism is true.
Yes, I’m familiar with much of that literature and as far as I’m aware, none of it provides evidence that moral claims are truth-apt.
In fact I’ve never even heard a moral realist propose any real-world experiment that could provide evidence for moral realism even in principle.
I’ve never heard a moral realist claim that their position can be supported experimentally.
I never said that moral realists claim their positions can be supported experimentally, or that experiments are necessary to establish moral realism. I was just giving an example of something that would demonstrate that I’m wrong. To clarify: there is a plethora of types of possible evidence which, if they existed, would convince me of moral realism (of which empirical experiments would be a sufficient example), but moral realists fail to meet the very low bar of presenting any of them.
The evolutionary argument against intuition has been addressed by e.g. Parfit in On What Matters. I’m not sure his defense is correct, but you can’t claim that Parfit simply doesn’t understand evolution.
Derek Parfit was very smart, probably smarter than I am, and I haven’t read his specific defense of ethical intuitionism against the evolutionary debunking anti-realist argument. I am willing to read it if you think it’s airtight. But I’m pretty sure it’s going to be wrong, having heard this category of argument many times before. I would also guess that yes, his belief in it has a defeater that he never materialized, which is him having a better understanding of evolution.
As a practicing Catholic and moral realist, something like what Max wrote above, represents my best current understanding of how to reason about moral truth, that is: a set of rules that governs how game theory agents ought to interact.
If you’re Catholic, surely you believe that God has ordained the moral truths and the Bible is His revelation of those truths, and the Catholic Church is His appointed representatives on Earth, right? So the correct way to reason about moral truth is to (1) follow the Bible’s/Church’s instructions, or if they don’t specify what to do in some situation, (2) extrapolate from the Bible/Church? Right? Why would you instead do this other thing involving reasoning about game theoretic agents? Isn’t that worse than following the Church, or even in contradiction of it?
I think a lot of biblical or church teachings can be interpreted through the lens of “provide guidance to how agents should act”. Turn the other cheek advocates for forgiveness (eg in prisoners dilemma defect defect loops); give away everything to the poor advocates for rawlsian or marginal value of money considerations; love thy neighbor as thyself pushes for a total utilitarian view.
I think the church teachings are quite wise and lindy, and cover a lot of how human interactions should behave, and form the basis of a lot of my behavior (in addition to other influences, such as EA and startup culture). By my words of “best current understanding”, I think I meant “how to reason about moral truth at the frontier”, aka situations that are much farther than that, perhaps concerning giant corporations or advanced AI.
I’m decidedly not Catholic,[1] but here’s my attempt to pass the ITT: We can see that things are true in different ways. For example, we can look in a textbook and see the words “the ratio of the circumference of any circle to its diameter is approximately 3.14159”. We can also go and measure real circles. And we can try to invent proofs for why that would be true. All three are good ways to learn about the world, with various pros and cons. The Bible/Church is the best moral authority, but also you can (if you’re smart and don’t make mistakes) see the truth with your own eyes instead of having to take it on authority.
(I’d be very curious to hear from actual Catholics about whether that matches their view.)
In addition to thinking that the doctrine is often wrong, I think the institution is probably net-bad for the world. My “Cool!”, above, is because I’m a xenophile who likes having lots of diversity in his society, and I care a lot more about how someone thinks, rather than what they think.
I think maybe it wouldn’t be “All agent should do X”, because you can have agents with weird values that reward not X for anything including cooperating with themselves etc.
So then it’s a somewhat different thing to scientific facts if for any moral statement you can say “no I don’t care, sure this points to real structure about other agents, but it doesn’t apply to me” and be correct. Noone can say “this doesn’t apply to me” about the laws of physics and be correct.
Though if you assume agents with coherent values then the statements about decision theory etc kind of just boil down to “you should fulfill your values” and that applies to weird values as well, so they are either incoherent or the statements apply.
“You should fulfill your values” and variants of it are a pretty small statement set though so effectively most moral statements like “murder is wrong” or “There should be more pleasure” end up subjective.
I hope it’s OK to use “You should pay back your debts, even when nothing is forcing you to, in situations that are real-world analogs of Parfit’s Hitchhiker (PH)” as my example moral fact.
Two observations:
PH assumes a utility function for the player such that cheat > pay > die.
I carefully constructed the moral fact to only be relevant in contexts that are analogous.
We thus come to the real difficulty of morality: figuring out which way to model real-world situations. If an agent is indifferent between dying and paying, having a policy to cheat is dominant. But this means the situation is not analogous!
This undercuts the notion of universitality that I think is important to the notion of moral realism, but perhaps less than it seems. We already must have a notion of applicable contexts for categorical imperatives. “It’s wrong to lie” presupposes a context where someone has knowledge of a thing and can choose to communicate falsehoods in a way that intends to mislead. If we change that context—say by giving the person Tourette’s—then we can no longer say that they’re doing the wrong thing.
From my perspective, morality is basically a branch of mathematics. It may be a kind of math that’s not very important to you (maybe because you live on a deserted island), but it’s still something you can think about, and still something that might one day matter.
As to whether one can escape by choosing to have values that make them exempt:
It’s easier said than done to change values.
Actually changing values is usually much worse than being bound by moral law, just in terms of rational self-interest.
There’s also probably a bunch of moral laws that basically mean it’s never correct to try to escape through changing your values.
In short, I agree that you can self-modify to escape nearly any moral statement, but that doesn’t let you escape from the universality of morality as a whole.
(I was sleepy when I wrote this. Lemme know if anything is unclear.)
From my perspective, morality is basically a branch of mathematics...
There’s also probably a bunch of moral laws that basically mean it’s never correct to try to escape through changing your values.
Want to explain how you’re getting “it’s never correct to try to escape [having to follow moral laws] by changing your values” purely from mathematics? Sounds highly dubious to me.
I think the correct version of the statement would be “it’s never correct to try to escape [having to follow moral laws] by changing your values unless you value becoming incoherent”
Based on my understanding of what’s meant by the terms, “correct” points to “does it fulfill your current values” and “moral laws” points to universal patterns in how to fulfill values like cooperate with yourself etc.
If you value becoming a rock, then it’s correct to modify yourself into something that moral law doesnt apply to.
If you value becoming a different coherent agent, then you will not escape moral law.
If you value becoming a rock/incoherent, you’re not changing to escape moral law, you’re changing because you want to be a rock/incoherent, so it’s fine for it to be correct.
Reasonable! This is an area where I’m pretty uncertain, and it compounds with what I believe our underlying disagreement is.
Leaving aside the question of whether morality is a branch of mathematics, since that eats into the scope of the broader conversation, here’s a sketch of why it might be universally foolish to try and escape categorical imperatives through self-modification, which I might expand later:
Suppose that you are watching a young prince who might one day become a tyrant (eg Joffrey from GoT) and trying to get a handle on what kind of person he is. You try to predict what kind of man he’ll grow up to be. Because you know him well, you expect him to, when king, say something like this:
“I owe a debt to those who helped me when I was young in a way that resembles Parfit’s Hitchhiker. Philosophers tell me that in order to make sure I actually exist, and am not merely in the mind of someone looking at my younger self, I should act benevolently. But! What if I change my values so that I am ambivalent between being assassinated in my youth and being a benevolent ruler instead of being a tyrant (which is what I really want want). Then I would cease to be analogous to Parfit’s Hitchhiker, and wouldn’t have to repay my debts by being benevolent! I could just be a tyrant, which is what I actually want to—”
You stop imagining the prince’s (now counterfactual) future and start making plans to have him assassinated.
Universal morality only applying to entities that are coherent agents in situations where they act seems reasonable to me.
I wouldn’t necessarily call morality as a whole a branch of mathematics though.
In my mind there’s
A notion of values that is a somewhat underdetermined, but fairly natural abstraction
Truths about what kinds of actions lead to fulfillment of values that apply to all agents (The DT stuff etc)
Truths about what values are convergent among real agents
Truths about what kinds of actions lead to the fulfillment of a particular agent’s values
Truths about what statements an agent endorses (slightly underdetermined)
Truths about the yumminess and motivationess / valence of conscious experience as felt from inside
I see “morality” as a very broad concept sort of pointing to all of these and maybe some more stuff.
My intuitive use of “objective” kind of jumbles together truth and (hypothetical) convergence sometimes so I’m often unsure how to use it.
I think it’s reasonable to say “You should pay back debts in PH” is a moral fact, it points at a mathematical truth about value fulfillment. It also seems reasonable to call it objectively true, though for getting the convergence aspect of objective you might need specify your definition of values more.
I think it’s also reasonable to say “You shouldn’t murder” is subjective because it doesn’t apply to all agents.
It also seems reasonable to me to say “Murder is wrong” is subjective while “Murder is wrong for Alice who values not murdering people” is objective, at least again subject to a definition of values that convergent enough to call objective.
I don’t have a very clear feeling view of the concepts and interactions between them though.
Correct me if I’m wrong but based on the types of things you mentioned above, it seems like you might be misunderstanding what moral facts are. For example, you might argue:
If game theory and decision theory indicate that you get better payoffs from cooperating than defecting, you should cooperate.
Game theory and decision theory indicate that you get better payoffs from cooperating than from defecting. [insert various arguments/proofs here]
Therefore, you should cooperate.
The last line, if interpreted as “you get better payoffs from cooperating” would be true, while “it’s morally good to cooperate and morally bad to defect” would be a moral fact if true, but it’s still an opinion—this type of argument hasn’t provided evidence for it.
Does it mean that moral facts are a term as obsolete as free will (or a useful approximation like temperature?),[1] and that they are to be dissolved in a combination of FDT and The Uncreated Theory Of Learning The Utility Function In Neural Nets?
P.S. I struggle to understand the point with weakly interacting moral particles.
Or an abstraction like temperature in LLMs. When the LLM sees a few lines of attack, it rules out idiotic approaches, then assigns close utility estimates to the best survivors and chooses randomly.
The “weakly interacting moral particles” thought experiment is offering a very easy goalpost, which moral realists fail to meet. It is an example—sufficient, but certainly not necessary—of something that would refute my position that moral claims are not truth-apt. If it were true, you could back up the claim “murder is wrong” with the physical observation that murderers lose their morality particles.
Suppose that we discovered that the WIMOPs obey most parts of traditional morality and have the quirk of having extremist views on interracial interactions: different races have different Lebensraums and living not in one’s Lebensraum or creating hybrids is prohibited, but ONLY by the WIMOPs, not because hybrids are prone to degeneration. Could one apply Yudkowsky’s case against morality being given to the WIMOP-derived morality?
Great question, though I will note that my stance is not that any particular moral theory is the right one, it is that none of them are truth-apt or have any evidence supporting them (other than people intuitively feeling that they’re true).
Suppose that we discovered that the WIMOPs obey most parts of traditional morality and have the quirk of having extremist views on interracial interactions: different races have different Lebensraums and living not in one’s Lebensraum or creating hybrids is prohibited, but ONLY by the WIMOPs, not because hybrids are prone to degeneration.
If you presented this discovery, I would admit that my position is wrong, because then there would be moral claims (e.g. “living outside of one’s Lebensraum is immoral”) that have evidence supporting them.
Here are my current thoughts on “human values.” There are a decent number of confusions here, which I’ll try to flag either explicitly or with a (?).
Let’s start with a distribution over possible worlds, where we can split each world into a fixed past and a future function which takes an action.[1] We also need a policy, which is a sensors -> action function,[2] where the state of the sensors is drawn from the world’s past.[3]
Assume that there exists either an obvious channel in many worlds that serves as a source of neutral[4] information (i.e. helpful for identifying which world the sensor data was drawn from, but “otherwise unimportant in itself”(?)), or that we can modify the actual worlds/context to add this information pathway.
We can now see how the behavior of the policy changes as we increase how informed it is, including possibly at the limit of perfect information. In some policies we should be able to (:confused arm wiggles:) factor out a world modeling step from the policy, which builds a distribution over worlds by updating on the setting of the sensors, and then feeds that distribution to a second sub-function with type world distribution -> action. (We can imagine an idealized policy that, in the limit of perfect information, is able to form a delta-spike on the specific world that its sensor-state was drawn from.) For any given delta-spike on a particular world, we can say that the action this sub-function chooses gives rise to an overall preference for the particular future[5] selected over the other possible futures. If the overall preferences conform to the VNM axioms we say that the sub-function is a utility function. Relevant features of the world that contribute to high utility scores are “values.”
I think it makes sense to use the word “agent” to refer to policies which can be decomposed into world modelers and utility functions. I also think it makes sense to be a bit less strict in conversation and say that policies which are “almost”(?) able to be decomposed in this way are basically still agents, albeit perhaps less centrally so.
Much of this semi-formalism comes from noticing a subjective division within myself and some of the AI’s I’ve made where it seems natural to say that “this part of the agent is modeling the world” and “this part of the agent is optimizing X according to the world model.” Even though the abstractions seem imperfect, they feel like a good way of gesturing at the structure of my messy sense of how individual humans work. I am almost certainly incoherent in some ways, and I am confused how to rescue the notion of values/utility given that incoherence, but I have a sense that “he’s mostly coherent” can give rise to “he more-or-less values X.”
Two agents can either operate independently or cooperate for some surplus. Ideally there’s a unique way to fairly split the surplus, perhaps using lotteries or some shared currency which they can use to establish units of utility. It seems obvious to me that there are many cooperative arrangements that are decidedly unfair, but I’m pretty confused about whether it’s always possible to establish a fair split (even without lotteries? even without side-payments?) and whether there’s an objective and unique Schelling point for cooperation.
If there is a unique solution, it seems reasonable to me to, given a group of agents, consider the meta-agent that would be formed if each agent committed fully to engaging in fair cooperation. This meta-agent’s action would essentially be an element of the cartesian product of each agent’s action space. In the human context, this story gives rise to a hypothetical set of “human values” which capture the kinds of things that humans optimize for when cooperating.
This seems a bit limited, since it neglects things that real humans optimize for that are part of establishing cooperation (e.g. justice). Does it really make sense to say that justice isn’t a value of human societies because in the fully-cooperative context it’s unnecessary to take justice-affirming actions? (??)
Even when considering a single agent, we can consider the coalition of that agent’s time-slices(?). Like, if we consider Max at t=0 and Max at t=1 as distinct agents, we can consider how they’d behave if they were cooperative with each other. This frame brings in the confusions and complications from group-action, but it also introduces issues such as the nature of future-instances being dependent on past-actions. I have a sense that I only need to cooperate with real-futures, and am free to ignore the desires of unreal-counterfactuals, even if my past/present actions are deciding which futures are real. This almost certainly introduces some fixed-point shenanigans where unrealizing a future is uncooperative with that future but cooperative with the future that becomes realized, and I feel quite uncertain here. More generally, there’s the whole logical-connective stuff from FDT/TDT/UDT.
I currently suspect that if we get a good theory of how to handle partial-coherence, how to handle multi-agent aggregation, and how to handle intertemporal aggregation, then “human values” will shake out to be something like “the mostly-coherent aggregate of all humans that currently exist, and all intertemporal copies of that aggregate” but I might be deeply wrong. :confused wiggles:
The future function either returns a single future state or a distribution over future states. It doesn’t really matter since we can refactor the uncertainty from the distribution over futures into the distribution over worlds.
Similarly to the distribution over futures we can either have a distribution over contexts given a past or we can have a fixed context for a given past and pack the uncertainty into our world distribution. See also anthropics and “bridge laws” and related confusions.
Confusion alert! Sometimes a source of information contains a bias where it’s selected for steering someone who’s listening. I don’t know how to prove an information channel doesn’t have this property, but I do have a sense that neutrality is the default, so I can assume it here without too much trouble.
..in the context of that particular past! Sometimes the future by itself doesn’t have all the relevant info (e.g. optimizing for the future matching the past).
Here are some thoughts about the recent back-and-forth where Will MacAskill reviewed IABI and Rob Bensinger wrote a reply and Will replied back. I’m making this a quick take instead of a full post because it gets kinda inside baseball/navelgazy and I want to be more chill about that than I would be in a full writeup.
First of all, I want to say thank you to Will for reviewing IABI. I got a lot out of the mini-review, and broadly like it, even if I disagree on the bottom-line and some of the arguments. It helped me think deep thoughts.
The evolution analogy
I agree with Will that the evolution analogy is useful and informative in some ways, but of limited value. It’s imperfect, and thinking hard about the differences is good.
I agree with this, and I appreciate MacAskill talking about it.
One of the big things that I think distinguishes more doomy people like me from less doomy people like Will is our priors on how incompetent people are. Like, I agree that it’s possible to carefully train an ML system to be (somewhat) robust to distributional shifts. But will we actually do that?
I think, at minimum, any plan to build AGI (to say nothing of ASI) should involve:
Trying to give the AI a clear, (relatively) simple goal that is well understood by humans and which we have strong reason to expect won’t cause a catastrophe if faithfully pursued, even if it gets wildly more power than expected and goes a little off-the-rails.
Training that AI in a wide variety of environments, trying to hit the real goal, rather than proxies. This should involve adversarial environments and a lot of paranoia that the AI has failed to robustly internalize the true goal.
Testing the AI on entirely new environments that were a-priori suspected of being difficult and weird, but where we also think there is a true answer that can be checked, restarting from scratch if the AI fails to generalize to the test set, or otherwise clearly demonstrates that it has not internalized the desired goal in a robust way.
And I personally think pure corrigibility has a nonzero chance of being a good choice of goal, and that a sufficiently paranoid training regime has a nonzero chance of being able to make a semi-safe AGI this way, even with current techniques. (That said, I don’t actually advocate for plans that have a significant chance of killing everyone, and I think “try to build corrigible AGI” does have a significant chance of killing everyone; I just notice that it seems better than what the research community currently seems to be doing, even at Anthropic.)
I predict the frontier lab that builds the first AGI will not be heavily focused on ensuring robustness to distributional shifts. We could bet, maybe.
Types of misalignment
I really benefited from this! Will changed my mind! My initial reaction to Will’s mini-review was like, “Will is wrong that these are distinct concepts; any machine sufficiently powerful to have a genuine opportunity to disempower people but which is also imperfectly aligned will produce a catastrophe.”
And then I realized that I was wrong. I think. Like, what if Will is (secretly?) gesturing at the corrigibility attractor basin or perhaps the abstracted/generalized pattern of which corrigibility is an instance? (I don’t know of other goals which have the same dynamic, but maybe it’s not just corrigibility?)
An agent which is pseudo-corrigible, and lives inside the attractor basin, is imperfectly aligned (hence the pseudo) but if it’s sufficiently close to corrigible it seems reasonable to me that it won’t disempower humanity, even if given the opportunity (at least, not in every instance it gets the opportunity). So at the very least, corrigibility (one of my primary areas of research!) is (probably) an instance of Will being right (and my past self being wrong), and the distinction between his “types of misalignment” is indeed a vital one.
I feel pretty embarrassed by this, so I guess I just wanna say oops/sorry/thanks.
If I set aside my actual beliefs and imagine that we’re going to naturally land in the corrigibility attractor basin by default, I feel like I have a better sense of some of the gradualism hope. Like, my sense is that going from pseudo-corrigible to perfectly corrigible is fraught, but can be done with slow, careful iteration. Maybe Clara Collier and other gradualists think we’re going to naturally land in the corrigibility attractor basin, and that the gradual work is the analogue of the paranoid iteration that I conceive as being the obvious next-step?
If this is how they’re seeing things, I guess I feel like I want to say another oops/sorry/thanks to the gradualists. …And then double-click on why they think we have a snowball’s chance in hell of getting this without a huge amount of restriction on the various frontier labs and way more competence/paranoia than we currently seem to have. My guess is that this, too, will boil down to worldview differences about competence or something. Still. Oops?
(Also on the topic of gradualism and the notion of having “only one try” I want to gesture at the part of IABI where it says (paraphrased from memory, sorry): if you have a clever scheme for getting multiple tries, you still only get one try at getting that scheme to work.)
I agree that “value space is big” is not a good argument, in isolation, for how likely it is for our creations to be aligned. The other half of the pincer is “our optimization pressure towards aligned goals is weak,” and without that the argument falls apart.
Here’s a plan for getting a good future:
Build ASI slowly, such that there’s some hope of being able to understand the first AI capable of a pivotal act.
The AI will want weird, alien stuff, but we’ll make sure that it’s the kind of mind that would prefer getting 50% of the stars with 99% probability to getting 100% of the stars with 70% probability.
Since we’re going slowly, we still have a 30% chance to stop it if we wanted.
We tell the AI “we’ll let you do a pivotal act and escape our clutches if you agree to only eat 50% of the stars, and help us get the other 50% as though you were aligned”
Our interpretability techniques are so good that we know whether it’s lying or whether it’s honorable and will actually give us half the stars.
Because we’re so good at reading its advanced, alien mind, it knows it needs to be honorable with us, and so is actually honorable.
The AI says “Yep, will do.”
We see it’s telling the truth when we check.
We set it free.
It gives us a good future.
I think this plan is bad because it fails the heuristic of “don’t summon demons and try to cleverly bargain with them,” but perhaps I’m being unfair.
My main criticism with “make deals with the AIs” is that it seems complex and brittle and like it depends heavily on a level of being able to read the machine’s mind that we definitely don’t currently have and might never have.
That said, I do think there’s a lot of value in being the sorts of people/groups that can make deals and be credible trade partners. Efforts to be more trustworthy and honorable and so on seem great.
On a linguistic level I think “risk-averse” is the wrong term, since it usually, as I understand it, describes an agent which is intrinsically averse to taking risks, and will pay some premium for a sure-thing. (This is typically characterized as a bias, and violates VNM rationality.) Whereas it sounds like Will is talking about diminishing returns from resources, which is, I think, extremely common and natural and we should expect AIs to have this property for various reasons.
Rob wrote some counterpoints to this, but I just want to harp on it a little. Making a deal with humans to not accumulate as much power as possible is likely an extremely risky move for multiple reasons, including that other AIs might come along and eat the lightcone.
I can imagine a misaligned AI maybe making a deal with humans who let it out of the box in exchange for some small fraction of the cosmos (and honoring the deal; again, the hard part is that it has to know we can tell if it’s lying, and we probably can’t).
I can’t really imagine an AI that has a clear shot at taking over the world making a deal to be a meek little salary worker, even if there are risks in trying to take over. Taking over the world means, in addition to other things, being sure you won’t get shut off or replaced by some other AI or whatever.
(Though I can certainly imagine a misaligned AI convincing people (and possibly parts of itself) that it is willing to make a deal like that, even as it quietly accumulates more power.)
Their proposal
Now we’re getting into the source of the infighting, I think (just plain fighting? I think of Will as being part of my ingroup, but idk if he feels the same; Rob definitely is part of my ingroup; are they part of each other’s ingroups? Where is the line between infighting and just plain fighting?). Will seems very keen on criticizing MIRI’s “SHUT IT DOWN YOU FOOLS” strategy — mostly, it seems to me, because he sees this approach as insufficiently supportive of strategies besides shutting things down.
When Rob shared his draft of his reply to Will, I definitely noticed that it seemed like he was not responding accurately to the position that I saw in Will’s tweet. Unfortunately, I was aware that there is something of a history between Will and MIRI and I incorrectly assumed that Rob was importing true knowledge of Will’s position that I simply wasn’t aware of. I warned him that I thought he was being too aggressive, writing “I expect that some readers will be like ‘whoa why is MIRI acting like this guy is this extremist—I don’t see evidence of that and bet they’re strawmanning him’.” But I didn’t actually push back hard, and that’s on me. Apologies to Will.
(Rob reviewed a draft of this post and adds his own apologies for misunderstanding Will’s view. He adds: “My thanks to Max and multiple other MIRI people for pushing back on that part of my draft. I made some revisions in response, though they obviously weren’t sufficient!”)
I’m very glad to see in Will’s follow-up:
I wish this had been more prominent in his mini-review, but :shrug:
I think Will and I probably agree that funding a bunch of efforts to research alignment, interpretability, etc. would be good. I’m an AI safety/alignment researcher, and I obviously do my day-to-day work with a sense that it’s valuable and a sense that more effort would also be valuable. I’ve heard multiple people (whom I respect and think are doing good work) complain that Eliezer is critical/dismissive of their work, and I wish Eliezer was more supportive of that work (while also still saying “this won’t be sufficient” if that’s what he believes, and somehow threading that needle).
I am pretty worried about false hope, though. I’m worried that people will take “there are a bunch of optimistic researchers working hard on this problem” as a sign that we don’t need to take drastic action. I think we see a bunch of this already and researchers like myself have a duty to shout “PLEASE DON’T RISK EVERYTHING! I DON’T GOT THIS!”[1] even while pursuing the least-doomed alignment strategies they know of. (I tried to thread this needle in my corrigibility research.)
Anyway, I think I basically agree with Will’s clarified position that a “kitchen-sink approach” is best, including a lot of research, as long as actually shutting down advanced training runs and pure capabilities research is in the kitchen sink. I feel worried that Will isn’t actually pushing for that in a way that I think is important (not building “It” is the safest intervention I’m aware of), but I’m also worried about my allies (people who basically agree that AI is unacceptably dangerous and that we need to take action) being unable to put forward a collective effort without devolving into squabbling about tone and strawmanning each other. :(
Anyway. Thank you again to Will and Rob. I thought both pieces were worth reading.
(Not to say that we should necessarily risk everything if alignment researchers do feel like they’ve “got this.” That’s a question worth debating in its own right. Also, it’s obviously worth noting that work that is incrementally useful but clearly insufficient to solve the entire field can still be valuable and the researcher is still allowed to say “I got this” on their little, local problems. (And they’re definitely allowed to speak up if they actually do solve the whole damn problem, of course. But they better have actually solved it!))
That’s not quite right. ‘Risk-averse with respect to quantity X’ just means that, given a choice between two lotteries A and B with the same expected value of X, the agent prefers the lottery with less spread. Diminishing marginal utility from extra resources is one way to get risk aversion with respect to resources. Risk-weighted expected utility theory is another. Only RWEUT violates VNM. When economists talk about ‘risk aversion,’ they almost always mean diminishing marginal utility.
Can you say more about why?
But AIs with sharply diminishing marginal utility to extra resources wouldn’t care much about this. They’d be relevantly similar to humans with sharply diminishing marginal utility to extra resources, who generally prefer collecting a salary over taking a risky shot at eating the lightcone. (Will and I are currently writing a paper about getting AIs to be risk-averse as a safety strategy, where we talk about stuff like this in more detail.)
(My sincere apologies for the delayed reply. I squeezed this shortform post out right before going on vacation to Asia, and am just now clearing my backlog to the point where I’m getting around to this.)
Cool. I guess I’m just wrong about what “risk averse” tends to mean in practice. Thanks for the correction.
Regarding diminishing returns being natural:
I think it’s rare to have goals that are defined in terms of the state of the entire universe. Human goals, for instance, seem very local in scope, eg it’s possible to say whether things are better/worse on Earth without also thinking about what’s happening in the Andromeda galaxy. This is in part because evolution is a blind hill-climber and so there’s no real selection pressure related to what’s going on in very distant places, and partly because even an intelligent designer is going to have an easier time specifying preferences over local configurations of matter, in part because the universe looks like it’s probably infinitely big. I could unpack this paragraph if it’d be useful.
Now, just because one has preferences that are sensitive to local changes to the universe doesn’t mean that the agent won’t care about making those local changes everywhere. This is why we expect humans to spread out amongst the stars and think that most AIs will do the same. See grabby aliens for more. From this perspective, we might expect each patch of universe to contribute linearly to the overall utility sum. But unbounded utility functions are problematic for various reasons, and again, the universe looks like it’s probably infinite. (I can dig up some stuff about unbounded utility issues if that’d be helpful.)
Regarding earning a salary:
My point is that earning a salary might not actually be a safer bet than trying to take over. The part where earning a salary gives 99.99% of maxutil is irrelevant. Suppose that you think life on Earth today as a normal human is perfect, no notes; this is the best possible life. You are presented with a button that says “trust humans not to mess up the world” and one that says “ensure that the world continues to exist as it does today, and doesn’t get messed up”. You’ll push the second button! It might be the case that earning a salary and hoping for the best is less risky, but it also might be the case (especially for a superintelligence with radical capabilities) that the safest move is actually to take over the world. Does that make sense?
No problem! Glass houses and all that.
Sure but this sounds like a case in which taking over the world is risk-free. The relevant analogy would be more like:
Choose between ‘Trust humans not to mess up the world’ and ’50% chance of immediate death, 50% chance you ensure the world continues to exist as it does today and doesn’t get messed up.′
And then depending on what the agent is risk-averse with respect to, they might choose the former. If they’re risk-averse with respect to consumption at a time but risk-neutral with respect to length of life, they’ll choose the latter. If they’re risk-averse with respect to the present discounted value of their future payment stream (as we suggest would be good for AIs to be), they’ll choose the former.
Cool. I think I agree that if the agent is very short-term oriented this potentially solves a lot of issues, and might be able to produce an unambitious worker agent. (I feel like it’s a bit orthogonal to risk-aversion, and comes with costs, but w/e.)
They don’t have to be short-term oriented! Their utility function could be:
u=f(Σipi)
Where f is some strictly concave function and pi is the agent’s payment at time i. Agents with this sort of utility function don’t discount the future at all. They care just as much about improvements to pi regardless of whether i is 1 or 1 million. And yet, for the right kind of f, these agents can be risk-averse enough to prefer a small salary with higher probability to a shot at eating the lightcone with lower probability.
Sorry, I guess I’m confused. Let me try and summarize where I feel like I’m at and what I’m hearing from you.
I think, if you’re an AGI, not trying to take over is extremely risky, because humans and future AIs are likely to replace you, in one way or another. But I also think that if you try to take over, this is also extremely risky, because you might get caught and turned off. I think the question of which is more risky depends on circumstance (e.g. how good is the security preventing you from seizing power), and so “risk aversion” is not a reliable pathway to unambitious AIs, because ambition might be less risky, in the long run.
I agree that if it’s less risky to earn a small salary, then if your concave function is sharp enough, the AI might choose to be meek. That doesn’t really feel like it’s engaging with my point about risk aversion only leading to meekness if trusting humans is genuinely less risky.
What I thought you were pointing out was that “in the long run” is load-bearing, in my earlier paragraph, and that temporal discounting can be a way to protect against the “in the long run I’m going to be dead unless I become God Emperor of the universe” thought. (I do think that temporal discounting is a nontrivial shield, here, and is part of why so few humans are truly ambitious.) Here’s a slightly edited and emphasized version of the paragraph I was responding to:
Do we actually disagree? I’m confused about your point, and feel like it’s just circling back to “what if trusting humans is less risky”, which, sure, we can hope that’s the case.
Yeah risk aversion can only make the AI cooperate if the AI thinks that getting paid for cooperation is more likely than successful rebellion. It seems pretty plausible to me that this will be true for moderately powerful AIs, and that we’ll be able to achieve a lot with the labor of these moderately powerful AIs, e.g. enough AI safety research to pre-empt the existence of extremely powerful misaligned AIs (who likely would be more confident of successful rebellion than getting paid for cooperation).
(The following is about a specific sub-point on the following part:)
I think the point about the corrigibility basin being larger than thought is the thing that makes me more optimistic about alignment (only a 10-30% risk of dying!) and I thought you pointed that out quite well here. I personally don’t think this is because of the competence of the labs but rather the natural properties of agentic systems (as I’m on your side when it comes to the competency of the labs). The following is some thinking of why and me trying to describe it in a way as well as me sharing some uncertainties about it.
I want to ask you why you think that the mathematical traditions that you’re basing your work on as of the posts from a year ago (decision theory, AIXI) are representative of future agents? Why are we not trying the theories out on existing systems that get built into agents (biology for example)? Why should we condition more on decision theory than distributed systems theory?
The answer (imo) is to some extent around the VNM axioms and reflexive rationality and that biology is to ephemeral to build a basis on, yet it still seems like we’re skipping out on useful information?
I think that there are places where biology might help you re-frame some of the thinking we do about how agents form.
More specifically, I want to point out OOD updating as something that biology makes claims about that are different from the traditional agent foundations model. Essentially, the biological frame implies something that is closer to a distributed system because it can cost a lot of energy to have a fully coordinated system due to costs of transfer learning that aren’t worth it. (here’s for example, a model of the costs of changing your mind: https://arxiv.org/pdf/2509.17957).
In that type of model, becoming a VNM agent is rather something that has an energy cost associated with it and it isn’t clear it is worth it when you incorporate the amount of dynamic memory and similar that you would require to set this up. So it would seem to me that biology and agent foundations arrive at different models about the arising of VNM-agents and I’m feeling quite confused about it.
I also don’t think I’m smart enough to figure out how to describe this from a fundamental decision theory way because it’s a bit too difficult to me and so I was thinking that you might have an idea why taking biology more seriously doesn’t make sense from a more foundational decision theory basis?
More specifically, does the argument about corrigibility being easier given non VNM-agents make sense?
Does the argument around VNM being more of a convergence property make sense?
And finally, I like the way you distilled the disagreement so thanks for that!
(My sincere apologies for the delayed reply. I squeezed this shortform post out right before going on vacation to Asia, and am just now clearing my backlog to the point where I’m getting around to this.)
I think I’m broadly confused by where you’re coming from. Sorry. Probably a skill issue on my part. 😅
Here’s what I’m hearing: “Almost none of the agents we actually see in the world are easy to model with things like VNM utility functions, instead they are biological creatures (and gradient-descended AIs?), and there are biology-centric frames that can be more informative (and less doomy?).”
I think my basic response, given my confusion is: I like the VNM utility frame because it helps me think about agents. I don’t actually know how to think about agency from a biological frame, and haven’t encountered anything compelling in my studies. Is there a good starting point/textbook/wiki page/explainer or something for the sort of math/modeling/framework you’re endorsing? I don’t really know how to make sense of “non VNM-agent” as a concept.
Have you seen/read my A broad basin of attraction around human values?
(My sincere apologies for the delayed reply. I squeezed this shortform post out right before going on vacation to Asia, and am just now clearing my backlog to the point where I’m getting around to this.)
Ah, that’s a great point! I had read it a while back, but it wasn’t coming to mind when I was writing this. I think that’s an excellent example of a similar dynamic besides corrigibility. When I’m thinking about things, I usually flatten out the goal-space to ignore deconfusion (or however one wants to characterize the kind of progress towards one’s “true values”), but it’s clearly relevant here. Thanks for bringing it up!
Oh, uh, I guess @wdmacaskill and @Rob Bensinger
IABI says: “Transistors, a basic building block of all computers, can switch on and off billions of times per second; unusually fast neurons, by contrast, spike only a hundred times per second. Even if it took 1,000 transistor operations to do the work of a single neural spike, and even if artificial intelligence was limited to modern hardware, that implies human-quality thinking could be emulated 10,000 times faster on a machine— to say nothing of what an AI could do with improved algorithms and improved hardware.
@EigenGender says “aahhhhh this is not how any of this works” and calls it an “egregious error”. Another poster says it’s “utterly false.”
(Relevant online resources text.)
(Potentially relevant LessWrong post.)
I am confused what the issue is, and it would be awesome if someone can explain it to me.
Where I’m coming from, for context:
We don’t know exactly what the relevant logical operations in the human brain are. The model of the brain that says there are binary spiking neurons that have direct connections from synapse->dendrite and that those connections are akin to floating-point numerical weights is clearly a simplification, albeit a powerful one. (IIUC “neural nets” in computers discard the binary-spikes and suggest another model where the spike-rate is akin to a numerical value, which is the basic story behind “neuron activation” in a modern system. This simplification also seems powerful, though it is surely an oversimplification in some ways.)
My main issue with the source text is that it ignores what is possibly the greater bottleneck in processing speed, which is the time it takes to move information from one area to another. (If my model is right, one of the big advantages of a MoE architecture is to reduce the degree of thrashing weights across the bus to and from the GPU as much, which can be a major bottleneck.) However, on this front I think nerves are still clearly inferior to wires? Even mylenated neurons have a typical speed of only about 100 m/s, while information flows across wires at >50% the speed of light.
My read of the critics is that they aren’t objecting to the notion that clock speeds are significantly faster than neurons, but rather that comparing the two is a bad way of thinking about things. @Eigengender says “I don’t think serial computation is a sane metric here. I expect that the total computation represented in a neuron spike is much much higher than the footnote would indicate.” The “Processor clock speeds are not how fast AIs think” post says “In general, I think it’s more sensible for discussion of cognitive capabilities to focus on throughput metrics such as training compute (units of FLOP) and inference compute.”
I certainly agree that if we’re trying to evaluate power we need to consider throughput and total computation. Suppose that a synapse is not a simple numerical weight, and instead we needed to consider each dendritic neurotransmitter gate as a computational unit. This would force us to use many more FLOPs to model a synapse. But would it change the maximum speed? I agree that on a machine of a given size, if you have twice as many floating point operations to do, it will take twice as much time to get through them all. But if we consider the limit where we are not forced to do parallelizable computations in serial, I expect most of the arguments about computational richness are irrelevant?
Perhaps the critics are saying that it takes more serial computations to capture the logic of a serial firing? But the source text admits that this might be the case, and suggests that it might even require 1,000 clock cycles to handle the computations. Is it really so obvious that it takes more than 1000x serial operations to capture a single neuron?
More context: I do think that the human brain is way more powerful (and WAY more efficient) than any current AI system. The extremely crude BOTEC of comparing weights and neocortex synapses says there’s something like a 100x difference, and my guess is that the brain is doing significantly fancier things than a modern transformer, algorithmically.
And of course, training/learning speed may be much more relevant than processing speed, and AFAIK humans are just wildly more data efficient.
And of course the speed at which the AI can approximate the logical action of a single neuron being higher doesn’t imply that the AI will take less time to have each thought. It seems straightforward that machine systems will make decisions in high-speed contexts using quick pathways and they will use any extra thinking speed to think more deeply in contexts where speed isn’t important (much like humans do!).
Anyway, like I said, I’m confused. I respect IABI’s critics and am hoping to learn where my model is wrong.
The passage seems fine to me; I commented on Erdil’s post and other brain efficiency discussions at the time, and I still think that power consumption is a more objective way of comparing performance characteristics of the brain vs. silicon, and that various kinds of FLOP/s comparisons favored by critics of the clock speed argument in the IAB passage are much more fraught ([1], [2]).
It’s true that clock speed (and neuron firing speed) aren’t straightforwardly / directly translatable to “speed of thought”, but both of them are direct proxies for energy consumption and power density. And a very rough BOTEC shows that ~10,000x is a reasonable estimate for the difference in power density between the brain and silicon.
Essentially, the brain is massively underclocked because of design-space restrictions imposed by biology and evolution, whereas silicon-based processing has been running up against fundamental physical limits on component size, clock speed, and power density for a while now. So once AIs can run whatever cognitive algorithms that the brain implements (or algorithms that match the brain in terms of high-level quality of the actual thoughts) at any speed, the already-existing power density difference implies they’ll immediately have a much higher performance ceiling in terms of the throughput and latency that they can run those algorithms at. It’s not a coincidence that making this argument via clock speeds leads to basically the same conclusion as making the same argument via power density.
This is the answer, but
The main restriction is power efficiency: the brain provides a great deal of intelligence for a budget of only ~20 watts. Spreading out that power budget over a very wide memory operating at very slow speed just turns out to be the most power efficient design (vs a very small memory running at very high speed), because memory > time.
My response.
Good point actually, and yeah the ability to move information from one area to the other much faster than brains do is arguably why NNs make different tradeoffs than human brains.
For what it’s worth, I wasn’t depending on the premise that a synapse is computationally more powerful than an artificial neuron.
I actually agree with this take, but critically not in the domain of allowing AIs to think faster, which was my original objection.
@Alexander Gietelink Oldenziel and @S. Alex Bradt and @Max Harms: The thing I was talking about when I agreed with the claim that the brain is more powerful and doing fancier things is basically the fact that the brain always learns and thinks, called continual learning or continual thinking (there is no knowledge cutoff for brains the way current LLMs have), as well as better long-term memory/keeping things in context.
I do agree that in general, human brains aren’t too special algorithmically.
Here are some links as to why below:
lc on why the current lack of long-term memory creates problems, especially for benchmarking
Dwarkesh Patel and Gwern on continual learning/thinking.
Do we actually have a source for this, or is this just a commonly believed fact about AIs? I’m getting worried that this claim isn’t actually supported by much evidence and is instead a social belief around AIs due to our previous prediction errors.
I do think AIs can run quite a bit faster than humans, I’m just making the claim that the transistor argument is locally invalid.Edit: @Max Harms I no longer endorse this objection, and now think my comment on it being utterly false that AI thinking speeds would increase drastically was not correct, and Max H explains why.
Sweet. Thanks for the thoughtful reply! Seems like we mostly agree.
I don’t have a good source on data efficiency, and it’s tagged in my brain as a combination of “a commonly believed thing” and “somewhat apparent in how many epochs of training on a statement it takes to internalize it combined with how weak LLMs are at in-context learning for things like novel board games” but neither of those is very solid and I would not be that surprised to learn that humans are not more data efficient than large transformers that can do similar levels of transfer learning or something. idk.
So it sounds like your issue is not any of the facts (transistor speeds, neuron speeds, AIs faster that humans) but rather the notion that comparing clock speeds and how many times a neuron can spike in a second is not a valid way to reason about whether AI will think faster than humans?
I’m curious what sort of argument you would make to a general audience to convey the idea that AIs will be able to think much faster than humans. Like, what do you think the valid version of the argument looks like?
I actually now think the direct argument given in IABIED was just directionally correct, and I was being confused in my objection, which Max H explains.
It’s fine to use the argument now.
Josh You has some interesting points on this subject. Quoting him for any counterarguments, as well as responses:
@Max H may have a different take than mine, and I’m curious for his input, but I find myself still thinking about serial operations versus parallel operations. Like, I don’t think it’s particularly important to the question of whether AIs will think faster to ask how many transistors operating in parallel will be needed to capture the equivalent information processing of a single neuron, but rather how many serial computations are needed. I see no reason it would take that many serial operations to capture a single spike, especially in the limit of e.g. specialized chips.
I think the linked tweet is possibly just misinterpreting what the authors meant by “transistor operations”? My reading is that “1000″ binds to “operations”; the actual number of transistors in each operation is unspecified. That’s how they get the 10,000x number—if a CPU runs at 1 GHz, neurons run at 100 Hz, then even if it takes 1000 clock cycles to do the work of neuron, the CPU can still do it 10,000x faster.
(IDK what the rationale was in the editorial process for using “transistor operations” instead of a more standard term like “clock cycles”, but a priori it seems defensible. Speculating, “transistors” was already introduced in the sentence immediately prior, so maybe the thinking was that the meaning and binding of “transistor operations” would be self-evident in context. Whereas if you use “clock cycles” you have to spend a sentence explaining what that means. So using “transistor operations” reduces the total number of new jargon-y / technical terms in the paragraph by one, and also saves a sentence of explanation.)
Anyway, depending on the architecture, precision, etc. a single floating point multiplication can take around 8 clock cycles. So even if a single neuron spike is doing something complicated that requires several high-precision multiply + accumulate operations in serial to replicate, that can easily fit into 1000 clock cycles on a normal CPU, and much fewer if you use specialized hardware.
As for the actual number of transistors themselves needed to do the work of a neuron spike, it again depends on exactly what the neuron spike is doing and how much precision etc. you need to capture the actual work, but “billions” seems too high by a few OOM at least. Some reference points: a single NAND gate is 4 transistors, and a general-purpose 16-bit floating point multiplier unit is ~5k NAND gates.
Hmm I see it. I thought it was making a distinct argument from the one Ege was responding to here, but if you’re right it’s the same one.
Then the claim is that an AI run on some (potentially large) cluster of GPUs can think far faster than any human in serial speed. You do lose the rough equivalency between transistors and neurons: a GPU, which is roughly equal to a person in resource costs, happens to have about the same number of transistors as a human brain has neurons. It’s potentially a big deal that AI has a much faster maximum serial speed than humans, but it’s far from clear that such an AI can outwit human society.
If you have a lump of 7,000 neurons, they can each connect to each other neuron, and you can spherical-cow approximate that as a 7000x7000 matrix multiplication. That matrix multiplication will all happen within O(1) spikes, 1⁄100 of a second. That’s ~700GFlop. An H100 GPU takes ~1 millisecond to do that operation, or 1M cycles, to approximate one brain spike cycle! And the gpu has 70B or whatever transistors, so it’s more like 10M transistors per neuron!
Not really—its vector matrix multiplication, not matrix matrix mult.
I’m guessing the intended meaning of “modern hardware” in the quote is easy to misread (it’s actually just gesturing at the transistors), and for many unintended senses of “modern hardware” (actual AI chips and systems, actual LLMs with KV cache) the claims in the quote are obviously wrong.
Ah, I hadn’t thought about that misreading being a source of confusion. Thanks!
The MoE architecture doesn’t just avoid thrashing weights around. It also reduces the amount of calculations per token. For instance, DeepSeek v3.1 has 671B parameters, out of which 37B are activated per token and used in matrices. A model like GPT-3 would use all the 175B parameters it has.
IIRC the human brain makes 1E14 -- 1E15 FLOP/second. The authors of the AI-2027 forecast imply that a human brain creates ~10 tokens/sec, or uses 1E13 -- 1E14 computations per token while having 1E14 synapses.
A more detailed analysis of Yudkowsky’s case for FOOM
If the brain was magically accelerated a million times so that signals reached the speed of 100 million m/s, then the brain would do 1E20 -- 1E21 FLOP/second while doing 1E17 transitions/sec. Cannell’s case for brain efficiency claims that the
fundamentalbaseline irreversible (nano) wire energy is: ~1 Eb/bit/nm, with Eb in the range of 0.1eV (low reliability) to 1eV (high reliability). If reliability is low and each transition is 1E7 nanometers or 1 centimeter, then we need 1E23 EV/second or 1E4 joules/second. IMO this implies that Yudkowsky’s case for a human brain accelerated a million times is as unreliable as Cotra’s case against AI arriving quickly. However, proving that AI is an existential threat is far easier since it requires us to construct an architecture, not to prove that there’s none.Returning to the human brain being far more powerful or efficient, we notice that it can’t, say, be copied infinitely many times. If it could, one could, say, upload a genius physicist and have an army of its copies work on different projects and exchange insights.
As for the humans being “wildly more data efficient”, Cannell’s post implies that AlphaGo disproves this conjecture with regards to narrow domains like games. What the humans are wildly more efficient is their ability to handle big contexts and to keep the information in mind for more than a single forward pass, as I discussed here and in the collapsible section here.
Yeah, sorry. I should’ve been more clear. I totally agree that there are ways in which brains are super inefficient and weak. I also agree that on restricted domains it’s possible for current AIs to sometimes reach comparable data efficiency.
on the bio side, I’d imagine the transistor equivalent is not a neuron spike, but rather a protein binding
Does anyone know someone who is researching, or would like to research the intersection of corrigibility and model welfare (eg does training to empower humans naturally lead towards more or less neurosis)? Interested both in math/theory and interp/experiment.
(I would like to direct funding here, if there are good opportunities.)
Seems important. I emailed you just now on this (time-sensitive, so letting you know here too).
(This quick take is mostly meant to be a trailhead for a discussion with Robi Rahman that was on Twitter. LW is just objectively a better platform for such things, and this way the conversation can still be public.)
Robi wrote:
To start, let me try to say some things that I hope are points of agreement. If they aren’t, it’s probably worth stopping to recurse on them, rather than continuing on to the harder stuff:
Agents have preferences, reflected in how those agents make choices when presented with a context where the outcomes of their actions are known (or at least they can be confident in the distribution over outcomes).
If these preferences are (VNM) coherent, they can be described by a utility function. We can use the word “values” and “goals” as angles on that same abstract thing that captures what the agent wants.
Most of the time, agents are not given straightforward choices between outcomes, and instead must do something like making plans on how to get what they want. When someone chooses to go to the store when it is closed, they may have wanted to see it for some reason, or they may have known the risk and decided to take the gamble, but often the action should be seen as a mistake—a failure of planning—rather than a reflection of wanting to get that particular outcome.
It is straightforward to get evidence about what a particular agent “should” do, in the sense that one can learn about what plans or actions are more or less likely to satisfy that agent’s values.
Thus there are clear subjective “shoulds.” If I know what you want, I can tell you what you should do, according to your own values.
There are also clear objective methods which are instrumentally valuable to a wide range of agents. For example, arithmetic is useful. In order to make arithmetic work, you should have the successor of 1 be 2, rather than 1. Similarly, there are many normative things one can say about how to reason that are largely objective in that sound reasoning is useful to so many kinds of minds.
This extends into decision theory and game theory by use of abstract utilities. Decision/game theory make normative claims by abstracting the particular things that the agent finds valuable, and instead focusing on strategic pathways to getting those things. The theorems of decision/game theory are objective, and any particular application of those theorems to real life can be true if the assumptions match the abstraction.
All agents should cooperate with their clone in the Prisoner’s Dilemma.
When playing against LDT opponents, all agents should suggest 50⁄50 splits in the Ultimatum Game, should accept such splits every time, and should often reject less favorable splits.
All agents should 1-box in Newcomb’s Problem.
All agents should pay the driver after getting back to town in Parfit’s Hitchhiker.
Hopefully all that is uncontroversial.
More controversial:
You should pay back your debts, even when nothing is forcing you to, in situations that are real-world analogs of Parfit’s Hitchhiker/Newcomb’s Problem.
^ this statement is a moral fact that is objectively true.
To avoid talking past each other, here are some additional questions for Robi:
Do the theorems of game theory count as moral facts? If not, what’s missing?
You asked for “evidence” and “truth-apt” claims. Is there evidence for math? Is 1+1=2 true, or merely valid?
Are you more interested in abstract arguments that point towards moral realism, or real-world observations that point to it?
I mostly agree with all of your supposedly uncontroversial starting points, though I’m sufficiently unsure on the last two that I’d prefer to hear arguments for moral realism that don’t rely on them.
About your more controversial points:
There are two very different ways to interpret the first statement, particularly the word “should”.
“You should pay back your debts” as in it’s instrumentally useful to be the kind of person who pays back debts even when you don’t have to, because being that kind of person has other benefits that outweigh the cost of paying such debts.
“You should pay back your debts” as in “you ought to pay back debts, and you’re a bad person if you don’t”.
The first interpretation of that is plausible to me, and is truth-apt. The second interpretation is an opinion, not a moral fact. I strongly agree with the second one, but it doesn’t have a truth value, it’s just a personal stance, even if most people share it (indeed, even if everyone in the world agrees).
Answering your additional questions to guide discussion:
No, theorems of game theory are absolutely not moral facts. No amount of proving theorems of game theory would answer my question.
Yes, there is ample evidence for math. “1+1=2” is true. “2+2=5″ is false. Both these claims are truth-apt, whereas “murder is wrong” is not.
Truth-apt statements are propositions that have some fact of the matter and can be true or false. Other examples include “the sky is blue” (true), “the sky is green” (false), “there is life on Earth” (true), “there is life in Andromeda” (unknown, but if you could travel to Andromeda, you could find out if it is true or false).
Statements that are not truth-apt include matters of opinion such as “blue is the best color”. (You could define ‘best’ in some unusual way such as being the most popular choice on a favorite color survey, which would make this claim truth-apt, but that’s not the meaning I’m referring to.) Opinions don’t have a truth value, but it can be true or false that a certain person holds a certain opinion.
There is plenty of evidence for math. Using your example of “1+1=2”, this is testable: find a group of 1 object, and another group of 1 object, and put them together and count the resulting number of objects. If it’s 2, you’ve found evidence for 1+1=2. If it’s any other number, you’ve found evidence against it. I’ve seen sufficiently many cases of math making correct predictions that I believe it is true, plus there are logical proofs of various parts of it.
I’ll accept either abstract arguments or real-world observations, though I’m especially interested in the latter. I claim you don’t have any real-world observations that provide evidence of moral realism (other than ethical intuitionism, which is extremely weak evidence, approximately as valuable as personal revelations of Jesus).
In fact I’ve never even heard a moral realist propose any real-world experiment that could provide evidence for moral realism even in principle.
Here is an example of such an experiment that would convince me of moral realism. For context, physicists hypothesize “weakly interacting massive particles”, which have mass and contribute to the gravitational pull of galaxies but don’t interact through electromagnetism or other ways we would observe them in everyday life. If these exist in large quantities, they would explain why galaxies have more mass than we can account for by looking at ordinary observable matter. Physicists hope to conduct some kind of experiment that would let us observe such a particle, which might solve the mysteries of dark matter. Similarly, there could exist “weakly interacting moral particles”, invisible particles that permeate the universe and whose movements correspond to the morality of actions. If we discovered a way to observe such particles, and people doing immoral actions such as murder accumulated a deficit of morality particles while people doing moral actions such as donating to charity or following the categorical imperative accumulated many morality particles, I would admit moral realism is true and strongly consider reorienting my life around whatever course of action is implied by the motion of these particles.
Excellent. Thanks for the thoughtful response.
I picked Parfit’s Hitchhiker (PH) because it’s the result that I think maps most cleanly to real-world moral dilemmas, but I think we can probably make headway using fairness and cooperation results (Ultimatum Game and Prisoner’s Dilemma) rather than reciprocity/character (PH). If we’re getting stuck it might be worth diving into why you’re unsure about Newcomb, but I’ll leave it for now.
Yep. This is a valid way to read my claim.
Those look to me like the same statement. Do you see “ought” and “should” as non-synonyms?
This introduces a new thing—whether a person can be good or bad—which is distinct from the argument that I’m trying to make in favor of moral realism. I have a perspective on this question, if it feels vital to you, but I’d rather stick to whether actions can have moral character.
Why not? What’s missing?
Troll: “I got one wolf and one hare, and put them in a pen together and now I have one wolf. I have found evidence that 1+1=1.”
More seriously, I think your stance is fine. I tend to use the correspondence theory of truth, where a model of reality is true insofar as the structure of the model matches the structure of reality. I tend to see math as an abstract structure that can be reasoned about in absence of any application to reality (and thus any notion of truth). Then, we can apply that structure to specific situations to get a more concrete model, which can be true or false, but it’s not always correct to apply a certain bit of math.
But I’m happy to use a more Platonic frame where the abstraction can be true in isolation. I just wanted to check.
Noted. I’ll try to aim towards real-world observations.
Troll: “Lots of smart people describe themselves as moral realists, and even, like Max, become more aligned with moral realism over time as smart adults (instead of being indoctrinated as kids). This is more likely to be observed if moral realism is true than if moral realism is false, therefore it is real-world evidence. QED.”
(I could try to provide more serious evidence, but my guess is that it’ll be more fruitful to stay on the “why can’t math theorems be moral facts” thread a bit longer.)
Sure, “should” and “ought” are synonyms, but “you [should/ought to] do X because it’s instrumentally useful” and “you [should/ought to] do X because it’s moral” are very different claims.
Sure, sorry, didn’t mean to expand the scope there. Let’s rephrase it to “and you’re doing something bad if you don’t”.
(Although if you convince me that actions can be objectively morally bad, it’s pretty trivial to convince me that people who do those actions can be objectively morally bad.)
Either of these approaches are fine; I don’t think you have any evidence for moral realism either way.
(I too mostly use the correspondence theory of truth; sorry if I implied I think it’s wrong)
“It would be instrumentally beneficial for you to do X” and “It would be morally good for you to do X” are totally different claims! (Unless you’re telling me it’s always instrumentally good to behave morally, which seems implausible unless there’s a god who sends immoral people to hell after they die.)
You’re joking but I really do consider the bolded sentence to be evidence for moral realism—in fact, it’s the strongest evidence for moral realism I’ve been able to find.
But there are many better explanations for lots of smart people believing a false thing, that have much more evidence supporting them, than moral realism actually being true.
Yeah, it was a troll not because it’s false but because it knocks down the weak version of your argument (“there is literally no Bayesian evidence”) without addressing the substance (“there isn’t good evidence”). I am glad you don’t stand by the literal text of your objection.
That’s just not what “moral facts” means. Theorems of game theory can be facts and they can be relevant to what you should do in certain situations where you might also consider moral theories, but they aren’t moral claims. A moral claim is something like “murder is wrong” or “slavery is acceptable”. On the other hand, “murder is illegal” or “slavery was legal in the Roman Empire and the Confederate States of America” are not the kind of moral facts I’m talking about, and they (and game theory theorems) don’t prove/disprove moral realism.
I’m trying to get you to define what “moral fact” is, and contrast it with the normative statements that I can prove, so that we can move past semantics. It’s my understanding that the literature uses the term to refer to something like categorical imperatives—pressures to do certain things that apply to all reasoning beings, regardless of who they are.
They can’t be infinitely strong pressures since obviously people sometimes do bad things, if there are bad things, but they have to have some motive force where they bind into the agent’s motivational system. And since they’re universal, they can’t bind to the agent’s terminal values. Thus, I claim, in order for a categorical imperative to make any sense, we must be able to see it show up in convergent instrumental strategy/subgoals.
Are all instrumentally convergent subgoals categorical imperatives (“moral facts”)? I don’t think so. From my perspective, a defining characteristic of the thing that I want to look at is that the pressure comes from the social fabric. The standard Omohundro drives of survival, rationality, resource acquisition, etc, are all just as applicable on a deserted island. But I claim that once you posit other agents, even if they’re powerless or distant or hypothetical, then there are arguments in the same space as “don’t defect against your clone your dumbass” that come into play.
What I’d really like is for you to taboo “moral fact” or “moral” or “wrong” or whatever and try saying the equivalent of “Theorems of game theory are absolutely not moral facts.”
Here’s how it caches out it my semantics: “Theorems of game theory are absolutely not [facts that bear on how any agent, regardless of who they are, ought to act in certain classes of situations, according to their own values, because of the acausal interplay (including the establishment of Schelling points, red lines, and other coordination mechanisms) between that agent and the other agents in the universe (including distant, causally disconnected places)].” But according to me, the theorems of game theory are extremely relevant, so you probably have a different notion of morality. What is it?
I wonder how similar Robi’s view on morality is to the idea that a fairly close equivalent of moral facts irreducible to game theory is the statement about consequences which are hard to trace or to explain.
Suppose that polyamory is net negative for most people who try it because such relationships bring genuine happiness to those who possess a rare quality and are fragile unless the participants possess the quality. IMO under these conditions Aella would expect people to be thoroughly tested to determine whether they possess the right qualities, then taking measures to ensure that incompatible people don’t do things like wasting time with each other. However, I expect such measures to be bottlenecked on things like it being hard to explain to participants why they would be unlikely to succeed, so the simplest measure in a society of those whom we don’t trust to be geniuses is to simply ban polyamory wholesale.
Similarly, if we trust the rumors that gentle-parenting-as-millenials-describe-it is hard to pull off in a manner which doesn’t produce a misaligned kid, then, once a cohort of kids parented that way emerges and is studied, the ideal world would have parents avoid this practice unless it is verified that they can succeed at gently raising the kid, and a world like ours would also require that parents don’t erroneously believe that they can.
However, my[1] example of WIMOPs hating hybrids would have Yudkowsky’s case against morality being given ensure that I, unlike Robi, don’t follow the WIMOPs’ anti-hybrid stance unless the hybrids are degrading in ways which are proven, but hard to explain to the commoner.
To be precise, the example was actually coined by a Russian extremist author, but the extremist had the fictional hybrids and their descendants actually degrade.
I agree, theorems of game theory are extremely relevant to that thing you described.
Again, I googled “what is morality” and the default result was pretty good. It said:
I’m not a moral relativist; I don’t believe that what’s right or wrong is decided by the beliefs of society or the context in which the action is taken. I believe there is no truth value to whether actions are objectively right or wrong.
Maybe “the social fabric” was the wrong way to phrase it. I mean something closer to “the possibility and nature of other agents in an idealized sense” (eg the way game theory is not single-player decision theory) not the specific society that the agent exists in.
And yeah, I get that you don’t agree with me. I was trying to describe my perspective, not yours.
Yeah, I’m saying I think social fabric is not relevant to the cruxes of our disagreement.
I just googled “what are moral facts” and its answer is pretty good:
I would add:
Moral facts are moral claims that are true.
My position is that there are no moral claims that have any truth values: none of them are true and none of them are false. They are not truth-apt; they are not in the category of things that can be true or false.
Regarding your game theory example, your claim that (because being the kind of agent who cooperates in the prisoner’s dilemma will benefit you in the long run) “you should cooperate in the prisoner’s dilemma” (where “should” means “is in your best interest/increases your expected utility”) is not a moral claim of the kind I am talking about. The claim that “you should cooperate in the prisoner’s dilemma”, where “should” means “it is morally good to cooperate”, is a moral claim, but is neither true nor false. I’m asking for any evidence that it, or any other moral claim, is true or false.
Yeah, I donno. This doesn’t feel like you’re tabooing things correctly. Imagine I put on the alien hat and pretend I don’t know these silly earth words. Tell me what morality is.
Sure. I claim “people should cooperate against their clones in PD” is a statement about what people ought to do. Is that an [UNTRANSLATABLE-1] fact? Why not?
The first and second examples are presumably things that are illegal, antisocial, and violations of the sorts of red-lines that can be drawn behind the veil of ignorance, among many other things. What makes them [UNTRANSLATABLE-3] and [UNTRANSLATABLE-5]? Are [UNTRANSLATABLE-3] and [UNTRANSLATABLE-5] synonyms? Do the [UNTRANSLATABLE-8] and [UNTRANSLATABLE-9] modifiers matter?
Sure. That’s basically what “fact” means.
Wait. If [UNTRANSLATABLE-1] claims are not in the category of things that can be true, then how can your WIMP example work? Didn’t you say you would change your mind if we detected WIMPs? How is this not in contradiction with the notion that it’s categorically not something that can be true?
Cool. I hear you that you don’t consider categorical imperatives to be [UNTRANSLATABLE-1] facts. But I still don’t know what you mean by [UNTRANSLATABLE-1]. Help! 👽
Defining morality while tabooing moral, ethical, right, wrong, good, bad, ought, should, duty, obligation, permissible, virtue, and cognate words:
Humans sometimes classify actions, intentions, people, institutions, and outcomes using positive, negative, or neutral labels that supposedly do more than describe what happened, predict consequences, report a law or custom, state what anyone wants, or identify an effective means to a chosen end.
Central examples of things that receive these classifications concern causing pain, killing, deception, coercion, keeping promises, helping people in distress, and distributing burdens and benefits. They contrast with judgments about which chess move wins, which investment earns more money, which action is legal, which behavior is fashionable, or which policy best fulfills a specified utility function.
The disputed question is whether some applications of these labels accurately represent features or relations that are not made accurate merely by anyone’s preferences, commitments, conventions, or decisions. My position is that applying the labels expresses attitudes; it does not report an additional feature of the action.
By “morality” I mean the human practice of evaluating conduct from a standpoint that people present as not being indexed merely to what the actor wants, what advances a stipulated objective, what their society accepts, or what will be rewarded or punished. I think statements made from that standpoint express attitudes or opinions rather than describe another feature of the world.
Edit: the disputed term, “morality”, names this classificatory practice, and, if its judgments are made accurate by some feature of reality, whatever feature makes them accurate. (I claim there is no such feature of reality.)
I’m not sure where to put the link, but I compiled a bunch of my thoughts on the topic and wrote them up as a blog post: https://utopiandreams.substack.com/p/moral-law
(Much of the content there is redundant with what I’ve written in this thread.)
Responding more directly to your definition, I claim that the mathematical notion of fairness can match your notion of morality. If someone offers a 9:1 split in the ultimatum game, the unfairness of that action
is not about what happened (it’s about the disconnect between what happened and the Schelling point of FDT agents)
is not a prediction of the consequences (the other agent might reject or accept depending on who they are)
is not about a human law or custom
is not a statement about what any particular person wants (the FDT Schelling point is an objective, mathematical property)
is not a statement about the effective means to a chosen end (if the offerer is playing against a CDT agent it’s an effective move; if they’re playing against an FDT agent it’s not effective; it’s unfair regardless)
Distributing benefits is a central example of the situation where this kind of concept gets applied, and a central example of a moral situation.
Would you accept “fairness realism” as a crux? Like, if you agreed that fairness is a real property, would it constitute an update about whether morality is a real property? If fairness isn’t related to morality, why not? I think your definition admits it.
Excellent. Thanks for the real taboo!
I want to check a few things to make sure I understand:
Suppose that the Christian God existed in a straightforward sense, entirely provable and possessing the all the power and status given to Him by the devout. In this world, if someone said “‘moral’ actions are those that follow the commandments of God”, you would say that (by definition) that can’t be a (sufficient) definition of morality, right? (It’d just be a claim about what God wants and/or the laws of God.)
Suppose that it turned out that, in all situations, lying was bad for all people, in terms of their own values, because of subtle, but real, reasons (eg their souls later go to hell). This wouldn’t be sufficient to say “lying is bad” (in the moral sense), because it’s merely “an effective means to a chosen end.” Yes?
If there were tiny, hard-to-detect fundamental particles (“WIMPs”) that gravitate towards people who donate to charity and are repelled by those who commit murder, saying “‘moral’ actions are those that attract WIMPs” would not, according to you, be a sufficient condition because it’s merely a prediction of the consequences of the act. How does this square with your earlier goalpost?
Good questions; perhaps I made a mistake somewhere in my definition and I should go back and adjust it.
No, then moral actions would have some real consequences, and that would distinguish them from immoral actions. For example, you’d be sent to hell or be smitten or whatever.
In this case you’re supposing God is the true ruler of the universe and the laws of Christianity are fundamental to the universe in the same way that the laws of physics are in ours. So then moral realism would be true. (Laws of God, if God were real, are different in this respect than laws made by humans, which only apply in certain places and cease to be real if people stop believing in them.)
No, because crucially, it’s a prediction of the consequences of the moral status of the act. For example, suppose moral realism is true and the correct ethical system is hedonic utilitarianism. Then it’s usually immoral to hit people because it hurts, unless they’re a masochist and enjoy pain, in which case it’s morally good to hit them. In that case the rightness or wrongness of your action (and whether the morality particles accumulate or leave) is determined not by what you’re physically doing but by the morality of doing it.
The unifying theme with your examples is that you’re giving counterfactual things that would make our human notion of morality real, but in our universe, in real life, those things do not exist and there is no fact of the matter on whether these morally charged things are truly good/bad.
Is the word “fundamental” important here? Like, suppose I say that I’m an economic realist, in that I believe that the laws of economics apply to our universe, but I don’t think they’re “fundamental” in the way that (some) laws of physics are. Rather, I think they apply to complex structures (eg humans and aliens) that meet certain criteria (marketplace dynamics).
(Sorry, I hadn’t attempted the taboo exercise yet, I was trying one more time at just giving the straightforward definition. Might retry but with all morality words tabooed, if I have time.)
Sorry. Clarifying: they are not things that it is categorically impossible for them to be true; they are things that have no evidence for their truth-aptness. If WIMPs were discovered then I would realize they are indeed in the category of truth-apt things.
I mean, I don’t see why discovering WIMPs would move [UNTRANSLATABLE-1] claims from not being truth-apt to being truth-apt. Truth-aptness is a claim about the meanings of words, not about the world. If a claim is truth-apt, that means there are ways the world could be that would render it true or false. So if the existence of WIMPS would render claims about [UNTRANSLATABLE-1] true, then there are ways the world could be that would render [UNTRANSLATABLE-1] claims true or false. So [UNTRANSLATABLE-1] claims are truth-apt.
A hypothesis: maybe you don’t mean to say that [UNTRANSLATABLE-1] claims are not truth-apt. Rather, you mean to say that you can’t think of any truth-condition for [UNTRANSLATABLE-1] claims that isn’t absurd on its face. If this is your state of mind wrt [UNTRANSLATABLE-1] claims, then 1. [UNTRANSLATABLE-1] claims are truth-apt, but 2. they are almost certainly false, but 3. it’s not categorically impossible for them to be true (since there are certain outlandish states of the world that could render them true if they obtained, but those states of the world seem absurd on their face to you). This would also explain why you’re having some trouble saying what [UNTRANSLATABLE-1] does mean, since all the specific hypotheses (e.g. WIMPS) sound so absurd that they couldn’t be what people in general mean by [UNTRANSLATABLE-1].
Or, in the jargon: I am guessing that maybe you’re not a non-cognitivist, but rather an error theorist. (As it turns out, A.J. Ayer, the arch-noncognitivist, upon hearing J.L. Mackie, the arch-error theorist, give a talk, reportedly said that that’s what he should have said all along.)
Sorry if I’m being rude by proposing a hypothesis about what you “actually mean” that contradicts what you’ve explicitly said you mean. Let me know if this sounds reasonable to you or if you still think “not truth-apt” accurately expresses what you meant.
No, because while the existence of WIMPs would show that some claims about [UNTRANSLATABLE-1] were true, the non-existence of WIMPs leaves us with no way to assess the truth or falsehood of [UNTRANSLATABLE-1]. Since there are no WIMPs, as far as we know, claims about [UNTRANSLATABLE-1] are still neither true nor false.
Let me rephrase my position without using the notion of truth-aptness.
There are no moral claims that are true or false. It is conceivable that in some hypothetical universe, there could be a state of the world where some moral claims are found to be true or false, but there’s no evidence that we are in such a universe.
I guess I’m still curious why, on your view, moral claims are not simply false. Like, the view that moral claims can never be true or false makes sense to me (standard non-cognitivist view). And the view that moral claims are all false, perhaps because they all make a false presupposition (e.g. the existence of WIMPs) makes sense to me (standard error-theory view). But I’m not sure what would motivate the view that, if something like WIMPs existed, then moral claims would be true, but in the actual world, where WIMPs don’t exist, moral claims are neither true nor false. What’s keeping you from just saying they’re all false? (Not saying this view is incoherent, per se, just very puzzled as to what would motivate you to hold it.)
Great question. My current best understanding is that moral claims are the opinions of the claimants. (This falls under emotivism, which is a non-cognitivist variety of anti-realism.) However, there is a counterfactual possible world in which moral claims are not merely opinions but have empirical evidence for or against them. But in our real universe, as far as anyone has been able to tell, there is no such evidence of that sort.
Let me try again to answer this. My position is:
You could imagine some counterfactual alternate universe, where moral claims are things that have evidence for or against them. Then they could be true or false.
But in real life, they are not that kind of thing. There is no such thing as direct evidence for or against any of them.
(There is indirect Bayesian evidence, e.g. many philosophy professors being ardent moral realists, so I have some small, nonzero credence that I’m mistaken. But that’s a feature of my epistemics and uncertainty, not of moral claims themselves.)
“people should cooperate against their clones in PD” is a statement about what people ought to do, where “ought” means “is in one’s self-interest”. It is not a statement about [UNTRANSLATABLE-1] because statements about what is in one’s self-interest are not the category of things that [UNTRANSLATABLE-1] is referring to.
I don’t think I can convincingly argue for moral realism, but IMO you are too quick to dismiss it. There is extensive philosophical literature addressing your objections (as I understand them). I’m not saying that any of that literature is definitely correct, but you seem to be under-rating it. I object to your acting as if meta-ethics is basically solved and saying (in jest) that it’s confusing when smart people are realists, while also not really addressing any of the arguments in favor of realism.
For example:
I’ve never heard a moral realist claim that their position can be supported experimentally.
Or:
The evolutionary argument against intuition has been addressed by e.g. Parfit in On What Matters. I’m not sure his defense is correct, but you can’t claim that Parfit simply doesn’t understand evolution.
62% of philosophers are realists (and 65% of metaethicists); it sounds like you are advocating for expressivism, which only 11% of philosophers agree with.
I point to surveys rather than making any object-level argument because I don’t think I can make good arguments on meta-ethics, you wouldn’t learn much from my arguments, and the arguments have to go for a long time to get anywhere at all. Instead, I’m pointing to the fact that a lot of literature exists that makes in-depth arguments for why realism is true.
Just want to flag that there’s a lot of definitional issues (which is a general challenge for deferring to surveys like that). Anti-realist is personally my clear best guess view of metaethics and I think a big portion (though far from all) of my disagreement with the surveyed philosophers is a verbal dispute. EDIT: I read other sibling comments more and it seems like people may care more about semantics discussion than I realized. If so, I agree that taking the semantic claim strongly is overconfident. I still think that non-naturalism vs naturalism is by far the most important debate here.
Importantly, more than half of those surveyed moral realists believe in naturalistic realism. I’ve looked into various positions in naturalistic realism until it bored me because I didn’t see how certain things matter e.g. seeing what causally regulates on the word “good” (cf: Cornell Realism) or what notions of folk morality we call “good” (cf: Jackson’s analytic functionalism). I think many philosophers who I respect also say its not a deep distinction (eg: Joe Carlsmith and Derek Parfit). I just believe that the definition is ultimately barely relevant imo. Perhaps one should be less confident about the definitions, but idk definitions seem partly subjective to me. The differences between the various anti-realist positions also seem not super deep to me.
There is still a substantial share of philosophers who believe in non-naturalist realism (25%), which is the number I care more about (though the 60% number is more cited). I am somewhat sympathetic to modesty based arguments for tough questions like this, so moral realism does have a small but real impact on my decision-making. However, I don’t defer more because I get worried about philosophers failing to account for things like mind-projection fallacy and thereby being too sympathetic to moral realism. (To be specific, my personal credences are tentatively 1-5% that non-naturalism realism is more right than anti-realism. Non-naturalism worlds do seem to matter more because mattering is fundamentally different.).
Yes, I’m familiar with much of that literature and as far as I’m aware, none of it provides evidence that moral claims are truth-apt.
I never said that moral realists claim their positions can be supported experimentally, or that experiments are necessary to establish moral realism. I was just giving an example of something that would demonstrate that I’m wrong. To clarify: there is a plethora of types of possible evidence which, if they existed, would convince me of moral realism (of which empirical experiments would be a sufficient example), but moral realists fail to meet the very low bar of presenting any of them.
Derek Parfit was very smart, probably smarter than I am, and I haven’t read his specific defense of ethical intuitionism against the evolutionary debunking anti-realist argument. I am willing to read it if you think it’s airtight. But I’m pretty sure it’s going to be wrong, having heard this category of argument many times before. I would also guess that yes, his belief in it has a defeater that he never materialized, which is him having a better understanding of evolution.
As a practicing Catholic and moral realist, something like what Max wrote above, represents my best current understanding of how to reason about moral truth, that is: a set of rules that governs how game theory agents ought to interact.
Whoa! I didn’t know you were Catholic! Cool!
If you’re Catholic, surely you believe that God has ordained the moral truths and the Bible is His revelation of those truths, and the Catholic Church is His appointed representatives on Earth, right? So the correct way to reason about moral truth is to (1) follow the Bible’s/Church’s instructions, or if they don’t specify what to do in some situation, (2) extrapolate from the Bible/Church? Right? Why would you instead do this other thing involving reasoning about game theoretic agents? Isn’t that worse than following the Church, or even in contradiction of it?
I think a lot of biblical or church teachings can be interpreted through the lens of “provide guidance to how agents should act”. Turn the other cheek advocates for forgiveness (eg in prisoners dilemma defect defect loops); give away everything to the poor advocates for rawlsian or marginal value of money considerations; love thy neighbor as thyself pushes for a total utilitarian view.
I think the church teachings are quite wise and lindy, and cover a lot of how human interactions should behave, and form the basis of a lot of my behavior (in addition to other influences, such as EA and startup culture). By my words of “best current understanding”, I think I meant “how to reason about moral truth at the frontier”, aka situations that are much farther than that, perhaps concerning giant corporations or advanced AI.
I’m decidedly not Catholic,[1] but here’s my attempt to pass the ITT: We can see that things are true in different ways. For example, we can look in a textbook and see the words “the ratio of the circumference of any circle to its diameter is approximately 3.14159”. We can also go and measure real circles. And we can try to invent proofs for why that would be true. All three are good ways to learn about the world, with various pros and cons. The Bible/Church is the best moral authority, but also you can (if you’re smart and don’t make mistakes) see the truth with your own eyes instead of having to take it on authority.
(I’d be very curious to hear from actual Catholics about whether that matches their view.)
In addition to thinking that the doctrine is often wrong, I think the institution is probably net-bad for the world. My “Cool!”, above, is because I’m a xenophile who likes having lots of diversity in his society, and I care a lot more about how someone thinks, rather than what they think.
I think maybe it wouldn’t be “All agent should do X”, because you can have agents with weird values that reward not X for anything including cooperating with themselves etc.
So then it’s a somewhat different thing to scientific facts if for any moral statement you can say “no I don’t care, sure this points to real structure about other agents, but it doesn’t apply to me” and be correct. Noone can say “this doesn’t apply to me” about the laws of physics and be correct.
Though if you assume agents with coherent values then the statements about decision theory etc kind of just boil down to “you should fulfill your values” and that applies to weird values as well, so they are either incoherent or the statements apply.
“You should fulfill your values” and variants of it are a pretty small statement set though so effectively most moral statements like “murder is wrong” or “There should be more pleasure” end up subjective.
Great thought.
I hope it’s OK to use “You should pay back your debts, even when nothing is forcing you to, in situations that are real-world analogs of Parfit’s Hitchhiker (PH)” as my example moral fact.
Two observations:
PH assumes a utility function for the player such that cheat > pay > die.
I carefully constructed the moral fact to only be relevant in contexts that are analogous.
We thus come to the real difficulty of morality: figuring out which way to model real-world situations. If an agent is indifferent between dying and paying, having a policy to cheat is dominant. But this means the situation is not analogous!
This undercuts the notion of universitality that I think is important to the notion of moral realism, but perhaps less than it seems. We already must have a notion of applicable contexts for categorical imperatives. “It’s wrong to lie” presupposes a context where someone has knowledge of a thing and can choose to communicate falsehoods in a way that intends to mislead. If we change that context—say by giving the person Tourette’s—then we can no longer say that they’re doing the wrong thing.
From my perspective, morality is basically a branch of mathematics. It may be a kind of math that’s not very important to you (maybe because you live on a deserted island), but it’s still something you can think about, and still something that might one day matter.
As to whether one can escape by choosing to have values that make them exempt:
It’s easier said than done to change values.
Actually changing values is usually much worse than being bound by moral law, just in terms of rational self-interest.
There’s also probably a bunch of moral laws that basically mean it’s never correct to try to escape through changing your values.
In short, I agree that you can self-modify to escape nearly any moral statement, but that doesn’t let you escape from the universality of morality as a whole.
(I was sleepy when I wrote this. Lemme know if anything is unclear.)
Want to explain how you’re getting “it’s never correct to try to escape [having to follow moral laws] by changing your values” purely from mathematics? Sounds highly dubious to me.
I think the correct version of the statement would be “it’s never correct to try to escape [having to follow moral laws] by changing your values unless you value becoming incoherent”
Based on my understanding of what’s meant by the terms, “correct” points to “does it fulfill your current values” and “moral laws” points to universal patterns in how to fulfill values like cooperate with yourself etc.
If you value becoming a rock, then it’s correct to modify yourself into something that moral law doesnt apply to.
If you value becoming a different coherent agent, then you will not escape moral law.
If you value becoming a rock/incoherent, you’re not changing to escape moral law, you’re changing because you want to be a rock/incoherent, so it’s fine for it to be correct.
Reasonable! This is an area where I’m pretty uncertain, and it compounds with what I believe our underlying disagreement is.
Leaving aside the question of whether morality is a branch of mathematics, since that eats into the scope of the broader conversation, here’s a sketch of why it might be universally foolish to try and escape categorical imperatives through self-modification, which I might expand later:
Suppose that you are watching a young prince who might one day become a tyrant (eg Joffrey from GoT) and trying to get a handle on what kind of person he is. You try to predict what kind of man he’ll grow up to be. Because you know him well, you expect him to, when king, say something like this:
“I owe a debt to those who helped me when I was young in a way that resembles Parfit’s Hitchhiker. Philosophers tell me that in order to make sure I actually exist, and am not merely in the mind of someone looking at my younger self, I should act benevolently. But! What if I change my values so that I am ambivalent between being assassinated in my youth and being a benevolent ruler instead of being a tyrant (which is what I really want want). Then I would cease to be analogous to Parfit’s Hitchhiker, and wouldn’t have to repay my debts by being benevolent! I could just be a tyrant, which is what I actually want to—”
You stop imagining the prince’s (now counterfactual) future and start making plans to have him assassinated.
That makes sense.
Universal morality only applying to entities that are coherent agents in situations where they act seems reasonable to me.
I wouldn’t necessarily call morality as a whole a branch of mathematics though.
In my mind there’s
A notion of values that is a somewhat underdetermined, but fairly natural abstraction
Truths about what kinds of actions lead to fulfillment of values that apply to all agents (The DT stuff etc)
Truths about what values are convergent among real agents
Truths about what kinds of actions lead to the fulfillment of a particular agent’s values
Truths about what statements an agent endorses (slightly underdetermined)
Truths about the yumminess and motivationess / valence of conscious experience as felt from inside
I see “morality” as a very broad concept sort of pointing to all of these and maybe some more stuff.
My intuitive use of “objective” kind of jumbles together truth and (hypothetical) convergence sometimes so I’m often unsure how to use it.
I think it’s reasonable to say “You should pay back debts in PH” is a moral fact, it points at a mathematical truth about value fulfillment. It also seems reasonable to call it objectively true, though for getting the convergence aspect of objective you might need specify your definition of values more.
I think it’s also reasonable to say “You shouldn’t murder” is subjective because it doesn’t apply to all agents.
It also seems reasonable to me to say “Murder is wrong” is subjective while “Murder is wrong for Alice who values not murdering people” is objective, at least again subject to a definition of values that convergent enough to call objective.
I don’t have a very clear feeling view of the concepts and interactions between them though.
Correct me if I’m wrong but based on the types of things you mentioned above, it seems like you might be misunderstanding what moral facts are. For example, you might argue:
If game theory and decision theory indicate that you get better payoffs from cooperating than defecting, you should cooperate.
Game theory and decision theory indicate that you get better payoffs from cooperating than from defecting. [insert various arguments/proofs here]
Therefore, you should cooperate.
The last line, if interpreted as “you get better payoffs from cooperating” would be true, while “it’s morally good to cooperate and morally bad to defect” would be a moral fact if true, but it’s still an opinion—this type of argument hasn’t provided evidence for it.
Does it mean that moral facts are a term as obsolete as free will (or a useful approximation like temperature?),[1] and that they are to be dissolved in a combination of FDT and The Uncreated Theory Of Learning The Utility Function In Neural Nets?
P.S. I struggle to understand the point with weakly interacting moral particles.
Or an abstraction like temperature in LLMs. When the LLM sees a few lines of attack, it rules out idiotic approaches, then assigns close utility estimates to the best survivors and chooses randomly.
The “weakly interacting moral particles” thought experiment is offering a very easy goalpost, which moral realists fail to meet. It is an example—sufficient, but certainly not necessary—of something that would refute my position that moral claims are not truth-apt. If it were true, you could back up the claim “murder is wrong” with the physical observation that murderers lose their morality particles.
Suppose that we discovered that the WIMOPs obey most parts of traditional morality and have the quirk of having extremist views on interracial interactions: different races have different Lebensraums and living not in one’s Lebensraum or creating hybrids is prohibited, but ONLY by the WIMOPs, not because hybrids are prone to degeneration. Could one apply Yudkowsky’s case against morality being given to the WIMOP-derived morality?
Great question, though I will note that my stance is not that any particular moral theory is the right one, it is that none of them are truth-apt or have any evidence supporting them (other than people intuitively feeling that they’re true).
If you presented this discovery, I would admit that my position is wrong, because then there would be moral claims (e.g. “living outside of one’s Lebensraum is immoral”) that have evidence supporting them.
Here are my current thoughts on “human values.” There are a decent number of confusions here, which I’ll try to flag either explicitly or with a (?).
Let’s start with a distribution over possible worlds, where we can split each world into a fixed past and a future function which takes an
action.[1] We also need a policy, which is asensors -> actionfunction,[2] where the state of the sensors is drawn from the world’s past.[3]Assume that there exists either an obvious channel in many worlds that serves as a source of neutral[4] information (i.e. helpful for identifying which world the sensor data was drawn from, but “otherwise unimportant in itself”(?)), or that we can modify the actual worlds/context to add this information pathway.
We can now see how the behavior of the policy changes as we increase how informed it is, including possibly at the limit of perfect information. In some policies we should be able to (:confused arm wiggles:) factor out a world modeling step from the policy, which builds a distribution over worlds by updating on the setting of the sensors, and then feeds that distribution to a second sub-function with type
world distribution -> action. (We can imagine an idealized policy that, in the limit of perfect information, is able to form a delta-spike on the specific world that its sensor-state was drawn from.) For any given delta-spike on a particular world, we can say that the action this sub-function chooses gives rise to an overall preference for the particular future[5] selected over the other possible futures. If the overall preferences conform to the VNM axioms we say that the sub-function is a utility function. Relevant features of the world that contribute to high utility scores are “values.”I think it makes sense to use the word “agent” to refer to policies which can be decomposed into world modelers and utility functions. I also think it makes sense to be a bit less strict in conversation and say that policies which are “almost”(?) able to be decomposed in this way are basically still agents, albeit perhaps less centrally so.
Much of this semi-formalism comes from noticing a subjective division within myself and some of the AI’s I’ve made where it seems natural to say that “this part of the agent is modeling the world” and “this part of the agent is optimizing X according to the world model.” Even though the abstractions seem imperfect, they feel like a good way of gesturing at the structure of my messy sense of how individual humans work. I am almost certainly incoherent in some ways, and I am confused how to rescue the notion of values/utility given that incoherence, but I have a sense that “he’s mostly coherent” can give rise to “he more-or-less values X.”
Two agents can either operate independently or cooperate for some surplus. Ideally there’s a unique way to fairly split the surplus, perhaps using lotteries or some shared currency which they can use to establish units of utility. It seems obvious to me that there are many cooperative arrangements that are decidedly unfair, but I’m pretty confused about whether it’s always possible to establish a fair split (even without lotteries? even without side-payments?) and whether there’s an objective and unique Schelling point for cooperation.
If there is a unique solution, it seems reasonable to me to, given a group of agents, consider the meta-agent that would be formed if each agent committed fully to engaging in fair cooperation. This meta-agent’s action would essentially be an element of the cartesian product of each agent’s action space. In the human context, this story gives rise to a hypothetical set of “human values” which capture the kinds of things that humans optimize for when cooperating.
This seems a bit limited, since it neglects things that real humans optimize for that are part of establishing cooperation (e.g. justice). Does it really make sense to say that justice isn’t a value of human societies because in the fully-cooperative context it’s unnecessary to take justice-affirming actions? (??)
Even when considering a single agent, we can consider the coalition of that agent’s time-slices(?). Like, if we consider Max at t=0 and Max at t=1 as distinct agents, we can consider how they’d behave if they were cooperative with each other. This frame brings in the confusions and complications from group-action, but it also introduces issues such as the nature of future-instances being dependent on past-actions. I have a sense that I only need to cooperate with real-futures, and am free to ignore the desires of unreal-counterfactuals, even if my past/present actions are deciding which futures are real. This almost certainly introduces some fixed-point shenanigans where unrealizing a future is uncooperative with that future but cooperative with the future that becomes realized, and I feel quite uncertain here. More generally, there’s the whole logical-connective stuff from FDT/TDT/UDT.
I currently suspect that if we get a good theory of how to handle partial-coherence, how to handle multi-agent aggregation, and how to handle intertemporal aggregation, then “human values” will shake out to be something like “the mostly-coherent aggregate of all humans that currently exist, and all intertemporal copies of that aggregate” but I might be deeply wrong. :confused wiggles:
The future function either returns a single future state or a distribution over future states. It doesn’t really matter since we can refactor the uncertainty from the distribution over futures into the distribution over worlds.
″
sensors” is meant to include things like working memories and other introspection.Similarly to the distribution over futures we can either have a distribution over contexts given a past or we can have a fixed context for a given past and pack the uncertainty into our world distribution. See also anthropics and “bridge laws” and related confusions.
Confusion alert! Sometimes a source of information contains a bias where it’s selected for steering someone who’s listening. I don’t know how to prove an information channel doesn’t have this property, but I do have a sense that neutrality is the default, so I can assume it here without too much trouble.
..in the context of that particular past! Sometimes the future by itself doesn’t have all the relevant info (e.g. optimizing for the future matching the past).