Personally, the author believes that SPI might “add up to normality”—that it will be a sort of reformulation of existing (informal) approaches used by humans, with similar benefits and limitations.
I’m a bit confused by this claim. To me it’s a bit unclear what you mean by “adding up to normality”. (E.g.: Are you claiming that A) humans in current-day strategic interactions shouldn’t change their behavior in response to learning about SPIs (because 1) they are already using them or 2) doing things that are somehow equivalent to them)? Or are you claiming that B) they don’t fundamentally change game-theoretic analysis (of any scenario/most scenarios)? Or C) are you saying they are irrelevant for AI v. AI interactions? Or D) that the invention of SPIs will not revolutionize human society, make peace in the middle east, …) Some of the versions seem clearly false to me. (E.g., re C, even if you think that the requirements for the use of SPIs are rarely satisfied in practice, it’s still easy to construct simple, somewhat plausible scenarios / assumptions (see our paper) under which SPIs do seem do matter substantially for game-theoretic analysis.) Some just aren’t justified at all in your post. (E.g., re A1, you’re saying that (like myself) you find this all confusing and hard to say.) And some are probably not contrary to what anyone else believes about surrogate goals / SPIs. (E.g., I don’t know anyone who makes particularly broad or grandiose claims about the use of SPIs by humans.)
I definitely don’t think (C) and the “any” variant of (B). Less sure about the “most” variant of (B), but I wouldn’t bet on that either.
I do believe (D), mostly because I don’t think that humans will be able to make the necessary commitments (in the sense mentioned in the thread with Rohin). I am not super sure about (A). My bet is that to the extent that SPI can work for humans, we are already using it (or something equivalent) in most situations. But perhaps some exceptions will work, like the lawyer example? (Although I suspect that our skill at picking hawkish lawyers is stronger than we realize. Or there might be existing incentives where lawyers are being selected for hawkishness, because we are already using them for someting-like-SPI? Overall, I guess that the more one-time-only an event is, the higher is the chance that the pre-existing selection pressures will be weak, and (A) might work.)
Overall I’d have appreciated more detailed discussion of when this is realistic (or of why you think it rarely is realistic).
That is a good point. I will try to expand on it, perhaps at least in a comment here once I have time, or so :-).
I definitely don’t think (C) and the “any” variant of (B). Less sure about the “most” variant of (B), but I wouldn’t bet on that either.
I do believe (D), mostly because I don’t think that humans will be able to make the necessary commitments (in the sense mentioned in the thread with Rohin). I am not super sure about (A). My bet is that to the extent that SPI can work for humans, we are already using it (or something equivalent) in most situations. But perhaps some exceptions will work, like the lawyer example? (Although I suspect that our skill at picking hawkish lawyers is stronger than we realize. Or there might be existing incentives where lawyers are being selected for hawkishness, because we are already using them for someting-like-SPI? Overall, I guess that the more one-time-only an event is, the higher is the chance that the pre-existing selection pressures will be weak, and (A) might work.)
That is a good point. I will try to expand on it, perhaps at least in a comment here once I have time, or so :-).