Also known as Raelifin: https://www.lesswrong.com/users/raelifin
Max Harms
Karma: 2,371
Corrigibility Prizes for Existing Work
Ah, that’s fair! I thought you were arguing that it’s not clear that they think it’s instrumentally bad, which it sounds like was not in doubt. Sorry for getting that wrong. I’ll edit the post to clarify.
A Defense of Gradual Disempowerment
Maybe we should wait until the call, but I’m still confused about the timing. What’s the relevant window of the prediction?
Like, suppose the AI is in a robot and decides to walk towards the shuttlecraft to travel to a lava planet where there’s a 10% chance in the next year that it will accidentally melt and thus the shutdown mechanism will be disabled (along with the rest of the robot/AI). Does it get shut down before it takes the first step? If not, what if it’s walking towards someone who intends to disable the shutdown mechanism (but not the rest of the machine) in the next 5 minutes? How should I be modeling this prediction mechanism’s scope?
Edit: Also, I agree this is a tangent, and it might be good to abandon it as not-cruxy. The more substantive question is whether the transformation is robust to the base model being clever and situationally aware. I still feel like if you tried to do a corrigibility transformation on an uploaded copy of me, I’d be able to wriggle out of it pretty easily.
How does time/causality work in the predictor add-on? (I claim “add-on” is fair, even if it’s just adding an additional head/output channel.) Like, if the agent’s action at t=0 doesn’t change the designated shutdown setup, but then (because of that action) at t=1 the designated channels predictably get disabled regardless of what the agent does, does the prediction fire at t=0 or t=1?
I’m right that none of this is in the paper, right?
Indeed, I feel like most of what you’re saying doesn’t address my concerns. (But I do think you’re trying!) Would you like to try scheduling a call? I could write more words if you prefer slower/async/public, but my sense right now is that it’ll be more efficient to have rapid feedback in navigating the disconnect.
I’m not sure where to put the link, but I compiled a bunch of my thoughts on the topic and wrote them up as a blog post: https://utopiandreams.substack.com/p/moral-law
(Much of the content there is redundant with what I’ve written in this thread.)
Responding more directly to your definition, I claim that the mathematical notion of fairness can match your notion of morality. If someone offers a 9:1 split in the ultimatum game, the unfairness of that action
is not about what happened (it’s about the disconnect between what happened and the Schelling point of FDT agents)
is not a prediction of the consequences (the other agent might reject or accept depending on who they are)
is not about a human law or custom
is not a statement about what any particular person wants (the FDT Schelling point is an objective, mathematical property)
is not a statement about the effective means to a chosen end (if the offerer is playing against a CDT agent it’s an effective move; if they’re playing against an FDT agent it’s not effective; it’s unfair regardless)
Distributing benefits is a central example of the situation where this kind of concept gets applied, and a central example of a moral situation.
Would you accept “fairness realism” as a crux? Like, if you agreed that fairness is a real property, would it constitute an update about whether morality is a real property? If fairness isn’t related to morality, why not? I think your definition admits it.
In this case you’re supposing God is the true ruler of the universe and the laws of Christianity are fundamental to the universe in the same way that the laws of physics are in ours. So then moral realism would be true. (Laws of God, if God were real, are different in this respect than laws made by humans, which only apply in certain places and cease to be real if people stop believing in them.)
Is the word “fundamental” important here? Like, suppose I say that I’m an economic realist, in that I believe that the laws of economics apply to our universe, but I don’t think they’re “fundamental” in the way that (some) laws of physics are. Rather, I think they apply to complex structures (eg humans and aliens) that meet certain criteria (marketplace dynamics).
Corrigibility Research Fund Grantees (Round 1)
How I’m Evaluating Corrigibility Grant Applications
If you value becoming a rock/incoherent, you’re not changing to escape moral law, you’re changing because you want to be a rock/incoherent, so it’s fine for it to be correct.
Excellent. Thanks for the real taboo!
I want to check a few things to make sure I understand:
Suppose that the Christian God existed in a straightforward sense, entirely provable and possessing the all the power and status given to Him by the devout. In this world, if someone said “‘moral’ actions are those that follow the commandments of God”, you would say that (by definition) that can’t be a (sufficient) definition of morality, right? (It’d just be a claim about what God wants and/or the laws of God.)
Suppose that it turned out that, in all situations, lying was bad for all people, in terms of their own values, because of subtle, but real, reasons (eg their souls later go to hell). This wouldn’t be sufficient to say “lying is bad” (in the moral sense), because it’s merely “an effective means to a chosen end.” Yes?
If there were tiny, hard-to-detect fundamental particles (“WIMPs”) that gravitate towards people who donate to charity and are repelled by those who commit murder, saying “‘moral’ actions are those that attract WIMPs” would not, according to you, be a sufficient condition because it’s merely a prediction of the consequences of the act. How does this square with your earlier goalpost?
Reasonable! This is an area where I’m pretty uncertain, and it compounds with what I believe our underlying disagreement is.
Leaving aside the question of whether morality is a branch of mathematics, since that eats into the scope of the broader conversation, here’s a sketch of why it might be universally foolish to try and escape categorical imperatives through self-modification, which I might expand later:
Suppose that you are watching a young prince who might one day become a tyrant (eg Joffrey from GoT) and trying to get a handle on what kind of person he is. You try to predict what kind of man he’ll grow up to be. Because you know him well, you expect him to, when king, say something like this:
“I owe a debt to those who helped me when I was young in a way that resembles Parfit’s Hitchhiker. Philosophers tell me that in order to make sure I actually exist, and am not merely in the mind of someone looking at my younger self, I should act benevolently. But! What if I change my values so that I am ambivalent between being assassinated in my youth and being a benevolent ruler instead of being a tyrant (which is what I really want want). Then I would cease to be analogous to Parfit’s Hitchhiker, and wouldn’t have to repay my debts by being benevolent! I could just be a tyrant, which is what I actually want to—”
You stop imagining the prince’s (now counterfactual) future and start making plans to have him assassinated.
Great thought.
I hope it’s OK to use “You should pay back your debts, even when nothing is forcing you to, in situations that are real-world analogs of Parfit’s Hitchhiker (PH)” as my example moral fact.
Two observations:
PH assumes a utility function for the player such that cheat > pay > die.
I carefully constructed the moral fact to only be relevant in contexts that are analogous.
We thus come to the real difficulty of morality: figuring out which way to model real-world situations. If an agent is indifferent between dying and paying, having a policy to cheat is dominant. But this means the situation is not analogous!
This undercuts the notion of universitality that I think is important to the notion of moral realism, but perhaps less than it seems. We already must have a notion of applicable contexts for categorical imperatives. “It’s wrong to lie” presupposes a context where someone has knowledge of a thing and can choose to communicate falsehoods in a way that intends to mislead. If we change that context—say by giving the person Tourette’s—then we can no longer say that they’re doing the wrong thing.
From my perspective, morality is basically a branch of mathematics. It may be a kind of math that’s not very important to you (maybe because you live on a deserted island), but it’s still something you can think about, and still something that might one day matter.
As to whether one can escape by choosing to have values that make them exempt:
It’s easier said than done to change values.
Actually changing values is usually much worse than being bound by moral law, just in terms of rational self-interest.
There’s also probably a bunch of moral laws that basically mean it’s never correct to try to escape through changing your values.
In short, I agree that you can self-modify to escape nearly any moral statement, but that doesn’t let you escape from the universality of morality as a whole.
(I was sleepy when I wrote this. Lemme know if anything is unclear.)
I’m decidedly not Catholic,[1] but here’s my attempt to pass the ITT: We can see that things are true in different ways. For example, we can look in a textbook and see the words “the ratio of the circumference of any circle to its diameter is approximately 3.14159”. We can also go and measure real circles. And we can try to invent proofs for why that would be true. All three are good ways to learn about the world, with various pros and cons. The Bible/Church is the best moral authority, but also you can (if you’re smart and don’t make mistakes) see the truth with your own eyes instead of having to take it on authority.
(I’d be very curious to hear from actual Catholics about whether that matches their view.)
- ^
In addition to thinking that the doctrine is often wrong, I think the institution is probably net-bad for the world. My “Cool!”, above, is because I’m a xenophile who likes having lots of diversity in his society, and I care a lot more about how someone thinks, rather than what they think.
- ^
Maybe “the social fabric” was the wrong way to phrase it. I mean something closer to “the possibility and nature of other agents in an idealized sense” (eg the way game theory is not single-player decision theory) not the specific society that the agent exists in.
And yeah, I get that you don’t agree with me. I was trying to describe my perspective, not yours.
Yeah, I donno. This doesn’t feel like you’re tabooing things correctly. Imagine I put on the alien hat and pretend I don’t know these silly earth words. Tell me what morality is.
[UNTRANSLATABLE-1] facts are real truths about what actions are [UNTRANSLATABLE-2], [UNTRANSLATABLE-3], [UNTRANSLATABLE-4], or [UNTRANSLATABLE-5].
They state what people ought to do or avoid.
Sure. I claim “people should cooperate against their clones in PD” is a statement about what people ought to do. Is that an [UNTRANSLATABLE-1] fact? Why not?
What [UNTRANSLATABLE-1] Facts Cover
[UNTRANSLATABLE-2] and [UNTRANSLATABLE-3]: Whether specific actions like lying or helping others are [UNTRANSLATABLE-4] or [UNTRANSLATABLE-5].
[UNTRANSLATABLE-6]: What people have a [UNTRANSLATABLE-7] to do.
Character traits: Whether traits like honesty or cruelty are [UNTRANSLATABLE-4] or [UNTRANSLATABLE-5].
Examples of Proposed [UNTRANSLATABLE-1] Facts
Torturing an [UNTRANSLATABLE-8] child for fun is [UNTRANSLATABLE-3].
[UNTRANSLATABLE-9] killing another person is [UNTRANSLATABLE-5].
Showing kindness to others is [UNTRANSLATABLE-4].
The first and second examples are presumably things that are illegal, antisocial, and violations of the sorts of red-lines that can be drawn behind the veil of ignorance, among many other things. What makes them [UNTRANSLATABLE-3] and [UNTRANSLATABLE-5]? Are [UNTRANSLATABLE-3] and [UNTRANSLATABLE-5] synonyms? Do the [UNTRANSLATABLE-8] and [UNTRANSLATABLE-9] modifiers matter?
[UNTRANSLATABLE-1] facts are [UNTRANSLATABLE-1] claims that are true.
Sure. That’s basically what “fact” means.
My position is that there are no [UNTRANSLATABLE-1] claims that have any truth values: none of them are true and none of them are false. They are not truth-apt; they are not in the category of things that can be true or false.
Wait. If [UNTRANSLATABLE-1] claims are not in the category of things that can be true, then how can your WIMP example work? Didn’t you say you would change your mind if we detected WIMPs? How is this not in contradiction with the notion that it’s categorically not something that can be true?
Regarding your game theory example, your claim that (because being the kind of agent who cooperates in the prisoner’s dilemma will benefit you in the long run) “you should cooperate in the prisoner’s dilemma” (where “should” means “is in your best interest/increases your expected utility”) is not a [UNTRANSLATABLE-1] claim of the kind I am talking about. The claim that “you should cooperate in the prisoner’s dilemma”, where “should” means “it is [UNTRANSLATABLE-1] [UNTRANSLATABLE-4] to cooperate”, is a [UNTRANSLATABLE-1] claim, but is neither true nor false. I’m asking for any evidence that it, or any other [UNTRANSLATABLE-1] claim, is true or false.
Cool. I hear you that you don’t consider categorical imperatives to be [UNTRANSLATABLE-1] facts. But I still don’t know what you mean by [UNTRANSLATABLE-1]. Help! 👽
I’m trying to get you to define what “moral fact” is, and contrast it with the normative statements that I can prove, so that we can move past semantics. It’s my understanding that the literature uses the term to refer to something like categorical imperatives—pressures to do certain things that apply to all reasoning beings, regardless of who they are.
They can’t be infinitely strong pressures since obviously people sometimes do bad things, if there are bad things, but they have to have some motive force where they bind into the agent’s motivational system. And since they’re universal, they can’t bind to the agent’s terminal values. Thus, I claim, in order for a categorical imperative to make any sense, we must be able to see it show up in convergent instrumental strategy/subgoals.
Are all instrumentally convergent subgoals categorical imperatives (“moral facts”)? I don’t think so. From my perspective, a defining characteristic of the thing that I want to look at is that the pressure comes from the social fabric. The standard Omohundro drives of survival, rationality, resource acquisition, etc, are all just as applicable on a deserted island. But I claim that once you posit other agents, even if they’re powerless or distant or hypothetical, then there are arguments in the same space as “don’t defect against your clone your dumbass” that come into play.
What I’d really like is for you to taboo “moral fact” or “moral” or “wrong” or whatever and try saying the equivalent of “Theorems of game theory are absolutely not moral facts.”
Here’s how it caches out it my semantics: “Theorems of game theory are absolutely not [facts that bear on how any agent, regardless of who they are, ought to act in certain classes of situations, according to their own values, because of the acausal interplay (including the establishment of Schelling points, red lines, and other coordination mechanisms) between that agent and the other agents in the universe (including distant, causally disconnected places)].” But according to me, the theorems of game theory are extremely relevant, so you probably have a different notion of morality. What is it?
Yeah, it was a troll not because it’s false but because it knocks down the weak version of your argument (“there is literally no Bayesian evidence”) without addressing the substance (“there isn’t good evidence”). I am glad you don’t stand by the literal text of your objection.
Doesn’t the context length matter a lot, here? Like, in the tables you gave above, none of the agents seem to be hitting the limits of the context window, and thus triggering compaction, whereas in the 130 billion output tokens for Navier-Stokes, a single agent would’ve been dominated by its compaction mechanism. Thus, even setting aside the speedup, it seems plausible to me that the lambda parameter changes quite a lot based on task size. On a very hard problem requiring billions of output tokens, it seems plausible that swarms have an edge. (Or not! I don’t know. My point is that I think extrapolating from ~100k output token cases might not work.)