Researcher at the Center on Long-Term Risk. All opinions my own.
Anthony DiGiovanni
Karma: 1,220
A high-level model of AI bargaining
[Linkpost] Evals for “SPI-incompatible” behavior & reasoning: Guide to initial research
It seems unnecessarily confusing to use the word “caring” for “putting weight on something in a way that isn’t ‘objective’, in the sense of empirical evidence plus logic”. (I know this has precedent in this community, tbc, I’m pushing back on that too.) I don’t assign very high probability to the sun rising tomorrow because I “care” a lot more about sun-rises-tomorrow hypotheses, I do that because I find the epistemological norms that ground induction intuitive. These are just different things, even if they share the property of being non-objective.
I believe basically everything can be formulated as a “bet”, and I don’t quite see what could be there about probabilities that can’t be phrased this way.
”What do you anticipate happening?” From my perspective, anticipation is nothing else than thinking about the consequences of an event. That’s useful if the event happens, and a waste of time if it doesn’t. Therefore, whether I anticipate an event translates to whether I want to bet my time on thinking about it.
”Aren’t you surprised by this event?” To me, surprisal is just getting into a situation that I didn’t make plans for. It’s equivalent to losing a bet: I wagered my time on thinking about the consequences of the other possibility, but the outcome that I didn’t bet on had come to pass.You can reformulate the anticipation/surprisal interpretations of probability in terms of bets, but I don’t think this is much of a positive argument for that approach. I would say, you should bet in some way for good reasons. The anticipation/surprisal interpretations at least gesture at what those reasons are: you expect better consequences from betting that way.
I think this is important, because the “probabilities are just betting odds” meme unnecessarily rules out (e.g.) imprecise probabilities by fiat.
(I’m not sure I endorse the anticipation/surprisal framings either, exactly. I think of probability more in terms of degrees of plausibility. See here for a bit more.)
good odds of changing stuff
Part of the problem OP points out is that “changing stuff” =/= “changing stuff positively”. (“Here’s one. Crucial considerations and sign flips are common.”)
These don’t seem like esoteric failures where it’s plausible that the hypothetical isn’t fair, or where the dynamic inconsistency is plausibly explained as being caused by changing values, or where there’s a fundamental tradeoff between the harms and value of new information. The errors happen in very mundane situations
Setting aside whether these count as “errors” (I basically agree with Jesse’s comment), I’m not sure why you think the above. Paul’s case involves this assumption: “We live in a very big universe where many copies of me all face the exact same decision. This seems plausible for a variety of reasons; the best one is accepting an interpretation of quantum mechanics without collapse (a popular view).”
In one sense this is mundane, since if you buy any of the large world theories, you’d think we’ve always been in such a large world. Maybe that’s what you mean. But it’s very non-mundane in the sense that these large worlds are still clearly out-of-distribution for our intuitions. We haven’t directly experienced the consequences of lots of copies of ourselves doing things. This is one big reason I’m pretty suspicious of my pre-theoretic intuitions about Paul’s case.
Lesser-known LLMisms, in my experience:
“gloss”
“My lean:”
“Does that land?”
“Net:”
Using a verb without a direct object, when that verb usually is paired with a direct object. (Not incorrect, but it sounds weird.)
“posture”
“throat-clearing”
Sorry, I don’t yet see how this engages with the argument in my post.
Your original post’s claims, as I understand them: “There’s a policy such that (1) if your counterpart best-responds to that policy, you all split things fairly, and (2) if your counterpart doesn’t best-respond, you all don’t totally destroy the pie. And (sec 2.4) it’s fine that other agents might make commitments before you make any decisions, because it’s not rational for other agents to commit to threaten you if you have this policy.”
My post’s counterargument: “That second sentence ignores the problem. You’re saying, it’s not rational ex post for some agent Bob to commit to threaten you if they know you have this policy. But the whole problem is that Bob might have thought ex ante that you would use some other policy.”
Why I don’t think you’ve addressed my counterargument:
(Your first bullet seems orthogonal to my counterargument.)
Second bullet: Bob might think ex ante that (a) you won’t follow FDT, [1] and (b) you’ll be more likely to meet his demands if he sticks with his unfair demand than if he doesn’t. Hence we haven’t dodged “the whole problem” above.
Third bullet: I think this is a really strong (empirical) claim, which you’ve only asserted rather than argued for.
- ↩︎
(Or rather, whichever version of “FDT” is stipulated by definition to never give in — I think this is debatable under the standard definition of FDT.)
There’s more explanation of why work on AI conflict is time-sensitive / not purely deferrable to AIs in this sequence, which I link to in the intro. And as I note in the intro:
We agree SPIs will likely be used by default. However, this is arguably not overwhelmingly likely, because AIs or humans in the loop might mistakenly lock out the opportunity to use SPIs later. It’s unclear if default capabilities progress will generalize to careful reasoning about novel bargaining approaches. So, given the large stakes of conflicts that SPIs could prevent, making SPI implementation even more likely seems promising overall.
To be clear, I don’t expect this to be super compelling on its own. I’m assuming readers have background on CLR’s agenda more generally. And this agenda is meant to motivate and explain our strategy on an approach that’s particularly promising if you already share CLR’s general prioritization.
Re: this:
And how would testing current AIs be relevant to the bargaining behavior of the powerful AIs you’re worried about?
My sense is that it’s fairly common in AI safety generally to study current AIs in preparation for the actually dangerous AIs. I don’t understand why you think this methodology is especially ill-suited for our threat model.
Can you say more how you’d respond to the problems with that kind of proposal discussed here?
CLR’s Safe Pareto Improvements Research Agenda
(I think there are probably some significant differences in general philosophical framings that make this discussion tricky, so I’m not sure if I’ll reply further.)
Because our semantic intuitions give us internal access to semantic facts (whe know what we mean),
I would say, we also know what we value — broadly construed, i.e., meant to encompass epistemic and decision-theoretic norms too. These normative attitudes are things we can directly introspect, and they’re neither merely semantic (as people usually use that term) nor empirical.
If you want to say claims about our values are really just semantic claims about our evaluative concepts, then I think:
That will be confusing to most people, based on how we usually use the term “semantic”.
That loses something important about the substance of bedrock normative attitudes: They’re, well, attitudes. To me it seems clear that a claim like “Other things being equal, suffering is bad” is doing something more than expressing meanings of terms. Someone who says “I understand what bad means, but I don’t see why suffering is bad” would be making a normative error by our lights, not a semantic one — unlike “I understand what bachelor means, but I don’t see why bachelors are unmarried”.
(“Bedrock” is the operative word here. Meant to exclude the “vast majority of ethical statements [that] also have some degree of empirical content”.)
Nice post!
This process typically involves some back-and-forth between our intuitive judgments about how a concept like “knowledge” or “morality” applies to a given case, and the more explicit and systematized analysis that the philosopher has proposed. Typically, intuitions are given substantive but finite amounts of weight, such that they can be revised/discarded if necessary; and considerations other than fit with intuitions (for example, the parsimony and intuitive appeal of the higher-level principles themselves) can play a role as well. The hope is to eventually reach a “reflective equilibrium” that best harmonizes one’s intuitions and one’s higher level analysis/understanding – though philosophers can still differ, here, in which form of harmonization they ultimately end up endorsing.
I’d be very interested in your takes on my critique of reflective equilibrium here.[1] I personally suspect it could be a philosophical mistake if we baked reflective equilibrium into the definition of “human-like philosophy” that AIs are supposed to do. (Not to say you’re claiming we should do that — you do say “typically”, which I appreciate.)
- ^
Some of my favorite examples of academic critiques:
Singer (1974) (largely an exegesis of Sidgwick)
McMahan (2013), sec. “A Sketch of a Foundationalist Conception of Moral Justification”
- ^
not just coextensive, but necessarily coextensive.
I meant to include necessarily coextensive, too.
It’s also epistemically rational to be certain of a tautology, even if you think (like I do) that being a tautology is a semantic fact
This seems like a non sequitur. I wasn’t claiming “norms of epistemic rationality can’t be functions of [whether a given proposition is a semantic fact]”. (Edit:) I was claiming “norms of epistemic rationality themselves aren’t merely semantic”. The semantic fact that X is a tautology doesn’t, by itself, dictate that you should have P(X) = 1. You need an (in this case, very obviously compelling) norm to tell you that.
Indeed, it’s hard to see where a priori justification could come from if not from semantic facts.
It can come from your intuitions in favor of some foundational norms. (Cf. “non-pragmatic principles” here.) Do you think that when, say, moral anti-realists talk about their bedrock moral intuitions, they’re just talking about semantic facts?
Freedom from human independent concepts is supposed to support the idea of a mathematical universe, not a relational one
Tegmark says (p. 10 here) “the only intrinsic properties of a mathematical structure are its relations”. So I think I am representing his actual argument for MUH in the OP. I think “the claim that the idea that some , but only some, maths exists materially, is unnecessary baggage” is meant to derive the Level IV multiverse from MUH.
It isn’t obvious that maths isn’t a human invention. It isn’t obvious that an external world has to be independent of human concepts, as well as human imagination. It isn’t obvious that maths can exist immaterially
FWIW I think I agree with all the non-obvious propositions here. I don’t think they’re the load-bearing premises of Tegmark’s argument.
Why I’m unconvinced by Tegmark’s argument for the mathematical universe hypothesis
The basic argument seems to be:
A complete description of external reality must be free of human-idiosyncratic concepts (“baggage”).
Any intrinsic properties of external reality, i.e. properties other than relational properties, are baggage.
Therefore a complete description of external reality must be purely relational. (And a mathematical structure is the only candidate for “purely relational”.)
I don’t see why we should buy (2).
As far as I know, there are two arguments for (2). The first: “Every time our theories of physics have improved, we’ve dissolved what we thought was ‘stuff’ into abstract relations between more fundamental stuff. That is, we’ve dissolved intrinsic properties into relational properties.”
But our evidence is “we keep dissolving concepts of non-fundamental intrinsic properties into relations”, not “we keep dissolving all the intrinsic properties that those concepts cover”. E.g. when we learned that the concept of an “atom” is nothing over and above a relation between subatomic particles, we weren’t getting rid of any of the fundamental stuff that composes an atom. The claim that there’s no fundamental stuff to be found is metaphysical, not something we have inductive evidence for.
Which brings us to the second argument: “We never directly observe this intrinsic fundamental stuff. Physics only needs relational properties to explain our observations. By Occam’s razor, we should drop the stuff.”
But I don’t know what it even means to have relations without stuff (e.g. parts of spacetime) to be related — to have structure without substance. So our observations do seem to entail intrinsic properties. I’m open to the correct ontology being very counterintuitive. But compared to the cost of dropping an ontological view as basic as “relations need stuff for the relations to apply to”, the simplicity gains from doing so seem pretty weak.
then that’s presumably a semantic fact about what you mean with these words
I don’t think so. This seems to conflate “A is coextensive with B” and “A is semantically identical to B”. My intuition in favor of assigning equal subjective probability to symmetric outcomes (assuming there’s a unique partition etc.) seems to be about the epistemic right-ness of doing so. (Where “right-ness” is compatible with normative anti-realism, I think.) That goes beyond semantics.
Thanks for clarifying! Back to your first comment then:
Worlds a) and b) and worlds a) and c) are incomparable from one standpoint because you are still radically clueless about alignment research being good
I’m still not sure I understand, sorry if I’m missing something basic. Let:
(a*) = “recommend donating to alignment research [without doing research about the sign of alignment beforehand]”
(b*) = “recommend not donating [without doing research about the sign of alignment beforehand]”
As you say, Lukas’s argument implies (a) > (b). (Independently of whether (b) ~ (c).) This holds even if (a*) is incomparable with (b*) (that’s precisely Lukas’s point). I don’t see how “you are still radically clueless about alignment research being good” — i.e., (a*) is incomparable with (b*) — tells me that (a) is incomparable with (b) (or that (a) incomp (c)).
Thanks! To clarify, my counterpoint wasn’t meant to address any objections to incomparability in principle. It was meant to respond to Lukas’s claim that even if you buy the arguments for incomparability in principle and for severe imprecision, some strategies are still comparable.
Anyway, to your point:
Worlds a) and b) and worlds a) and c) are incomparable from one standpoint because you are still radically clueless about alignment research being good.
Could you spell out this claim more? What does it mean for one pair of strategies ((a) and (b)) to be incomparable to another ((a) and (c))? It sounds like you want to say “from one standpoint, (b) and (c) are incomparable”, but I’m not sure.
Have you asked Katja what she’d prefer here? (I worry people might get a negative impression of her from this even though she didn’t intend it.)