The problem is that the two human player’s minds aren’t logically related. Each human player in this game wants his AI to play defect, because their decisions are logically independent of each other’s.
I don’t think so. Each player wants to do the Winning Thing, and there is only one Winning Thing (their situations are symmetrical), so if they’re both good at Winning (a significantly lower bar than successfully building an AI with their preferences), their decisions are related.
So what you’re saying is, given two players who can successfully build AIs with their preferences (and that’s common knowledge), they will likely (surely?) play cooperate in one-shot PD against each other. Do I understand you correctly?
Suppose what you say is correct, that the Winning Thing is to play cooperate in one-shot PD. Then what happens when some player happens to get a brain lesion that causes him to unconsciously play defect without affecting his AI building abilities? He would take everyone else’s lunch money. Or if he builds his AI to play defect while everyone else builds their AIs to play cooperate, his AI then takes over the world. I hope that’s a sufficient reductio ad absurdum.
Hmm, I just noticed that you’re only saying “their decisions are related” and not explicitly making the conclusion they should play cooperative. Well, that’s fine, as long as they would play defect in one-shot PD, then they would also program their AIs to play defect in one-shot PD (assuming each AI can’t prove its source code to the other). That’s all I need for my argument.
So what you’re saying is, given two players who can successfully build AIs with their preferences (and that’s common knowledge), they will likely (surely?) play cooperate in one-shot PD against each other. Do I understand you correctly?
Yes.
Suppose what you say is correct, that the Winning Thing is to play cooperate in one-shot PD. Then what happens when some player happens to get a brain lesion that causes him to unconsciously play defect without affecting his AI building abilities? He would take everyone else’s lunch money. Or if he builds his AI to play defect while everyone else builds their AIs to play cooperate, his AI then takes over the world. I hope that’s a sufficient reductio ad absurdum.
Good idea. Hmm. It sounds like this is the same question as: what if, instead of “TDT with defection patch” and “pure TDT”, the available options are “TDT with defection patch” and “TDT with tiny chance of defection patch”? Alternately: what if the abstract computations that are the players have a tiny chance of being embodied in such a way that their embodiments always defect on one-shot PD, whatever the abstract computation decides?
It seems to me that Lesion Man just got lucky. This doesn’t mean people can win by giving themselves lesions, because that’s deliberately defecting / being an abstract computation that defects, which is bad. Whether everyone else should defect / program their AIs to defect due to this possibility depends on the situation; I would think they usually shouldn’t. (If it’s a typical PD payoff matrix, there are many players, and they care about absolute, not relative, scores, defecting isn’t worth it even if it’s guaranteed there’ll be one Lesion Man.)
This still sounds disturbingly like envying Lesion Man’s mere choices – but the effect of the lesion isn’t really his choice (right?). It’s only the illusion of unitary agency, bounded at the skin rather than inside the brain, that makes it seem like it is. The Cartesian dualism of this view (like AIXI, dropping an anvil on its own head) is also disturbing, but I suspect the essential argument is still sound, even as it ultimately needs to be more sophisticated.
I guess my reductio ad absurdum wasn’t quite sufficient. I’ll try to think this through more thoroughly and carefully. Let me know which steps, if any, you disagree with, or are unclear, in the following line of reasoning.
Until a few years ago, almost everyone on Earth was running some sort of non-TDT which plays defect in one-shot PD.
It’s possible that upon learning about TDT, some people might spontaneously switch to running it, depending on whatever meta-DT controls this, and whether the human brain is malleable enough to run TDT.
If, in any identifiable group of people, a sufficient fraction switches to TDT, and that proportion is public knowledge, the TDT-running individuals in that group should start playing cooperate in one-shot PD with other members of the group.
The threshold proportion is higher if the remaining defectors can cause greater damage. If the remaining defectors can use their gains from defection to better reproduce themselves, or to gather more resources that will let them increase their gains/damage, then the threshold proportion must be close to 1, because even a single defector can start a chain reaction that causes all the resources of the group to become held by defectors.
What proportion of skilled AI designers would switch to TDT is ultimately an empirical question, but it seems to me that it’s unlikely to be close to unity.
TDT-running AI designers will design their AIs to run TDT. Non-TDT-running AI designers will design their AIs to run non-TDT (not necessarily the same non-TDT).
Assume that a TDT-running AI (TAI) can’t tell which other AIs are running TDT and which ones aren’t, so in every game it faces the decision described in steps 4 and 5. A TDT AI will cooperate in some situations where the benefit from cooperation is relatively high and damage from defection relatively low, and not in other situations.
As a result, non-TAI will do better than TAI, but the damage to TAIs will be limited.
Only if a TAI is sure that all AIs are TAIs, will it play cooperate unconditionally.
If a TAI encounters an AI of alien origin, the same logic applies. The alien AI will be TAI if-and-only-if its creator was running TDT. If the TAI knows nothing about the alien creator, then it has to estimate what fraction of AI-builders in the universe runs TDT. Taking into account that TDT can’t arise from evolution, and not seeing any reason for evolution to create a meta-DT that would pick TDT upon discovering it, this fraction seems pretty low, and so the TAI will likely play defect against the alien AI.
Hmm, this exercise has cleared a lot of my own confusion. Obviously a lot more work needs to be done to make the reasoning rigorous, but hopefully I’ve gotten the gist of it right.
ETA: According to this line of argument, your hypothesis that all skilled AI designers play cooperate in one-shot PD against each other is equivalent to saying that skilled AI designers have minds malleable enough to run TDT, and have a meta-DT that causes them to switch to running TDT. But I do not see an evolutionary reason for this, so if it’s true, it must be true by luck. Do you agree?
It looks like in this discussion you assume that switching to “TDT” (it’s highly uncertain what this means) immediately gives the decision to cooperate in “true PD”. I don’t see why it should be so. Summarizing mypreviouscomments, exactly what the players know about each other, exactly in what way they know it, may make their decisions go either way. That the players switch from CDT to some kind of more timeless decision theory doesn’t determine the answer to be “cooperate”, it merely opens up the possibility that previously was decreed irrational, and I suspect that what’s important in the new setting for making the decision go either way isn’t captured properly in the problem statement of “true PD”.
Also, the way you treat “agents with TDT” seems more appropriate for “agents with Cooperator prefix” from cousin_it’s Formalizing PD. And this is a simplified thing far removed from a complete decision theory, although a step in the right direction.
I don’t assume that switching to TDT immediately gives the decision to cooperate in “true PD”. I assume that an AI running TDT would decide to cooperate if it thinks the expected utility of cooperating is higher than the EU of defecting, and that is true if its probability of facing another TDT is sufficiently high compared to its probability of facing a defector (how high is sufficient depends on the payoffs of the game). Well, this is necessary but not sufficient. For example if the other TDT doesn’t think its probability of facing a TDT is high enough, it won’t cooperate, so we need some common knowledge of the relevant probabilities and payoffs.
Does my line of reasoning make sense now, given this additional explanation?
Actually it makes less sense now, since your explanation seems to agree that two “TDT” algorithms that know each of them is “TDT” won’t necessarily cooperate, which undermines my hypothesis for why you were talking about cooperation as a sure thing in some relation to “TDT”. I still think you make that assumption though. Citation from your argument:
4. If, in any identifiable group of people, a sufficient fraction switches to TDT, and that proportion is public knowledge, the TDT-running individuals in that group should start playing cooperate in one-shot PD with other members of the group.
I’m having trouble understanding what you’re talking about again. Do you agree or disagree with step 4? To rephrase it a bit, if an identifiable group of people contains a high fraction of individuals running TDT, and that proportion is public knowledge, then TDT-running individuals in that group should play cooperate in one-shot PD with other members of the group in games where the payoffs are such that potential gains from mutual cooperation is large compared to potential loses from being defected against. (Assuming being in such a group is the best evidence available about whether someone is running TDT or not.)
If you disagree, why do you think a TDT-running individual might not play cooperate in this situation? Can you give an example to help me understand?
I disagree with step 4, I think sometimes the TDT players that know they both are TDT players won’t cooperate, but this discussion stirred up some of the relevant issues, so I’ll answer later when I figure out what I should believe now.
I don’t see why TDT players would fail to cooperate under conditions of common knowledge. Are you talking about a case where they each know the other is TDT but think the other doesn’t know they know, or something like that?
I don’t know the whole answer, but for example consider what happens with Pareto-efficiency in PD when you allow mixed strategies (and mixed strategy is essentially the presence of nontrivial dependence of the played move on the agent’s state of knowledge, beyond what is restricted by the experiment, so there is no actual choice about allowing mixed strategies, mixed strategies are what’s there by default even if the problem states that players select some certain play). Now, the Pareto-efficient plays are those where one player cooperates with certainty, while the other cooperates or defects with some probability. These strategies correspond to bargaining between the players. I don’t know how to solve the bargaining problem (aka fairness problem aka first-mover problem in TDT), but I see no good reason to expect that the solution in this case is going to be exactly pure cooperation. Which is what I meant by the insufficiency in correspondence between true PD and pure cooperation: true PD seems to give too little info, leaving uncertainty about the outcome, at least in this sense. This example doesn’t allow both players to defect, but it’s not pure cooperation either.
It’s too elegant to arise by evolution, and it also deals with one-shot PDs with no knock-on effects which is an extremely nonancestral condition—evolution by its nature deals with events that repeat many times; sexual evolution by its nature deals with organisms that interbreed; so “one-shot true PDs” is in general a condition unlikely to arise with sufficient frequency that evolution deals with it at all.
Taking into account that TDT can’t arise from evolution, and not seeing any reason for evolution to create a meta-DT that would pick TDT upon discovering it
This may perhaps embody the main point of disagreement. A self-modifying CDT which, at 7am, expects to encounter a future Newcomb’s Problem or Parfit’s Hitchhiker in which the Omega gets a glimpse at the source code after 7am, will modify to use TDT for all decisions in which Omega glimpses the source code after 7am. A bit of “common sense” would tell you to just realize that “you should have been using TDT from the beginning regardless of when Omega glimpsed your source code and the whole CDT thing was a mistake” but this kind of common sense is not embodied in CDT. Nonetheless, TDT is a unique reflectively consistent answer for a certain class of decision problems, and a wide variety of initial points is likely to converge to it. The exact proportion, which determines under what conditions of payoff and loss stranger-AIs will cooperate with each other, is best left up to AIs to calculate, I think.
Nonetheless, TDT is a unique reflectively consistent answer for a certain class of decision problems, and a wide variety of initial points is likely to converge to it.
The main problem I see with this thesis (to restate my position in a hopefully clear form) is that an agent that starts off with a DT that unconditionally plays D in one-shot PD will not self-modify into TDT, unless it has some means of giving trustworthy evidence that it has done so. Suppose there is no such means, then any other agent must treat it the same, whether it self-modifies into TDT or not. Suppose it expects to face a TDT agent in the future. Whether that agent will play C or D against it is independent of what it decides now. If it does self-modify into TDT, then it might play C against the other TDT where it otherwise would have played D, and since the payoff for C is lower than for D, holding the other player’s choice constant, it will decide not to self-modify into TDT.
If it expects to face Newcomb’s Problem, then it would self-modify into something that handles it better, but that something must still unconditionally play D in one-shot PD.
Do you still think “a wide variety of initial points is likely to converge to it”? If so, do you agree that (ETA: in a world where proving source code isn’t possible) those initial points exclude any DT that unconditionally plays D in one-shot PD?
BTW, there are a number of decision theorists in academia. Should we try to get them to work on our problems? Unfortunately, I have no skill/experience/patience/willpower for writing academic papers. I tried to write such a paper about cryptography once and submitted it to a conference, got back a rejection with nonsensical review comments, and that was that. (I guess I could have tried harder but then that would probably have put me on a different career path where I wouldn’t be working these problems today.)
Also, there ought to be lots of mathematicians and philosophers who would be interested in the problem of logical uncertainty. How can we get them to work on it?
Suppose it expects to face a TDT agent in the future. Whether that agent will play C or D against it is independent of what it decides now.
Unless that agent already knows or can guess your source code, in which case it is simulating you or something highly correlated to you, and in which case “modify to play C only if I expect that other agent simulating me to play C iff I modify to play C” is a superior strategy to “just D” because an agent who simulates you making the former choice (and which expects to be correctly simulated itself) will play C against you, while if it simulates you making the latter choice it will play D against you.
If it does self-modify into TDT, then it might play C against the other TDT where it otherwise would have played D, and since the payoff for C is lower than for D, holding the other player’s choice constant, it will decide not to self-modify into TDT.
The whole point is that the other player’s choice is not constant. Otherwise there is no reason ever for anyone to play C in a one-shot true PD! Simulation introduces logical dependencies—that’s the whole point and to the extent it is not true even TDT agents will play D.
“Holding the other player’s choice constant” here is the equivalent of “holding the contents of the boxes constant” in Newcomb’s Problem. It presumes the answer.
Unless that agent already knows or can guess your source code, in which case it is simulating you or something highly correlated to you
I think you’re invoking TDT-style reasoning here, before the agent has self-modified into TDT.
Besides, I’m assuming a world where agents can’t know or guess each others’ source codes. I thought I made that clear. If this assumption doesn’t make sense to you, consider this: What evidence can one AI use to infer the source code of another AI or its creator? What if any such evidence can be faked near perfectly by the other AI? What about for two AIs of different planetary origins meeting in space?
I know you’d like to assume a world where guessing each others’ source code is possible, since that makes everything work out nicely and everyone can “live happily ever after”. But why shouldn’t we consider both possibilities, instead of ignoring the less convenient one?
ETA: I think it may be possible to show that a CDT won’t self-modify into a TDT as long as it believes there is a non-zero probability that it lives in a world where it will encounter at least one agent that won’t know or guess its current or future source code, but in the limit as that probability goes to zero, the DT it self-modifies into converges to TDT.
I think you’re invoking TDT-style reasoning here, before the agent has self-modified into TDT.
I already said that agents which start out as pure CDT won’t modify into pure TDTs—they’ll only cooperate if someone gets a peek at their source code after they self-modified. However, humans, at least, are not pure CDT agents—they feel at least the impulse to one-box on Newcomb’s Problem if you raise the stakes high enough.
This has nothing to do with evolutionary contexts of honor and cooperation and defection and temptation, and everything to do with our evolved instincts governing abstract logic and causality, which is what governs what sort of source code you think has what sort of effect. Even unreasonably pure CDT agents recognize that if they modify their source code at 7am, they should modify to play TDT against any agent that has looked at their source code after 7am. To humans, who are not pure CDT agents, the idea that you should play essentially the same way if Omega glimpsed your source code at exactly 6:59am, seems like common sense given the intuitions we have about logic and causality and elegance and winning. If you’re going to all the trouble to invent TDT anyway, it seems like a waste of effort to two-box against Omega if he perfectly saw your source code 5 seconds before you self-modified. (These being the kind of ineffable meta-decision considerations that we both agree are important, but which are hard to formalize.)
Besides, I’m assuming a world where agents can’t know or guess each others’ source codes.
You are guessing their source code every time you argue that they’ll choose D. If I can’t make you see this as an instance of “guessing the other agent’s source code” then indeed you will not see the large correlations at the start point, and if the agents start out highly uncorrelated then the rare TDT agents will choose the correct maximizing action, D. They will be rare because, by assumption in this case, most agents end up choosing to cooperate or defect for all sorts of different reasons, rather than by following highly regular lines of logic in nearly all cases—let alone the same line of logic that kept on predictably ending up at D.
There’s a wide variety of cases where philosophers go astray by failing to recognize an instance of an everyday concept as an abstract concept. For example, they say in one breath that “God is unfalsifiable”, and in the next breath talk about how God spoke to them in their heart, because they don’t recognize “God spoke to me in my heart” as an instance of “God allegedly made something observable happen”. Philosophers talk about qualia being epiphenomenal in one breath, and then in the next speak of how they know themselves to be conscious, because they don’t recognize this self-observation as an instance of “something making something else happen” aka “cause and effect”. The only things recognized as matching the formal-sounding phrase “cause and effect” are big formal things officially labeled “causal”, not just stuff that makes other stuff happen.
In the same sense, you have this idea about modeling other agents as this big official affair that requires poring over their source code with a magnifying glass and then furthermore verifying that they can’t change it while you aren’t looking.
You need to recognize the very thought processes you are carrying out right now in arguing that just about anyone will choose D as an instance of guessing the outputs of the other agents’ source codes and moreover guessing that most such codes and outputs are massively logically correlated.
This is witnessed by the fact that if we did get to see some interstellar transactions, and you saw that the first three transactions were (C, C), you would say, “Wow, guess Eliezer was right” and expect the next one to be (C, C) as well. (And of course if I witnessed three cases of (D, D) I would say “Guess I was wrong.”) Even though the initial conditions are not physically correlated, we expect a correlation. What is this correlation, then? It is a logical correlation. We expect different species to end up following similar lines of reasoning, that is, performing similar computations, like factorizing 123,456 in spacelike separated galaxies.
It occurs to me that the problem is important enough that even if we can reach intuitive agreement, we should still do the math. But it doesn’t help to solve the wrong problem, so do you think the following is the right formalization of the problem?
Assume a “no physical proof of source code” universe.
Assume three types of intelligent life can arise in this universe.
In a Type A species, Eliezer’s intuition is obvious to everyone, so they build AIs running TDT without further consideration.
In a Type B species, my intuition is obvious to everyone so, so they build AIs running XDT, or AIs running CDT which immediately self-modify into XDT. Assume (or prove) that XDT behaves like TDT except it unconditionally plays D in PD.
In a Type C species, different people have different intuitions, and some (Type D individuals) don’t have strong intuitions or prefer to use a formal method to make this meta-decision. We human beings obviously belong to this type of species, and let’s say we at LessWrong belong to this last subgroup (Type D).
Does this make sense so far?
Let me say where my intuition expects this to lead to, so you don’t think I’m setting a trap for you to walk into. Whatever meta-decision we make, it can be logically correlated only to AIs running TDT and other Type D individuals in the universe. If the proportion of Type D individuals in the universe is low, then it’s obviously better for us to implement XDT instead of TDT. That’s because whether we use TDT or XDT will have little effect on how often other TDTs play cooperate. (They can predict what Type D individuals will decide, but since there are few of us and they can’t tell which AIs were created by Type D individuals, it won’t affect their decisions much.)
Unfortunately we don’t know the proportions of different types of species/individuals. So we should program an AI to estimate them, and have it make the decision of what to self-modify into.
ETA: Just realized that the decisions of Type D individuals can also correlate with the intuitions of others, since intuitions come from unconscious mental computations and they may be of a similar nature with our explicit decisions. But this correlation will be imperfect, so the above reasoning still applies, at least to some extent.
ETA2: This logical correlation stuff is hard to think about. Can we make any sense of these types of problems before having a good formal theory of logical correlation?
ETA3: The thing that’s weird here is that assuming everyone’s intuitions/decisions aren’t perfectly correlated, some will build TDTs and some will build XDTs. And it will be the ones who end up deciding to build XDTs that defect who will win. How to make sense of this, if that’s the wrong decision?
ETA4: I’ll be visiting Mt. Rainier for the rest of the day, so that’s it. :) Sorry for the over-editing.
Maybe cousin_it is right and we really have to settle this by formal math. But I’m lazy and will give words one more try. If we don’t reach agreement after this I’m going to the math.
So, right now we have different intuitions. Let’s say you have the correct intuition and convince everyone of it, and I have the incorrect one but I’m too stupid to realize it. So you and your followers go on to create a bunch of AIs with TDT. I go on to create an AI which is like TDT except it plays defect in PD. Lets say I pretended to be your follower and we never had this conversation, so there is no historical evidence that I would create such an AI. When my AI is born, it modifies my brain so that I start to believe I created an AI with TDT, thus erasing the last shred of evidence. My AI will then go on and win against every other AI.
Given the above, why should I change my mind now, and not win?
ETA: Ok, I realize this is pretty much the same scenario as the brain lesion one, except it’s not just possible, it’s likely. Someone is bound to have my intuition and be resistant to your persuasion. If you say that smart agents win, then he must be the smart one, right?
This I think connects one more terminological distinction. When you talked earlier about something like “reasoning about the output of platonic computation” as a key insight that started your version of TDT, you meant basically the same thing as me talking about how even knowing about the existence of the other agent, little things that you can use to even talk about it, is already the logical dependence between you and the other agent that could in some cases be used to stage cooperation.
A self-modifying CDT which, at 7am, expects to encounter a future Newcomb’s Problem or Parfit’s Hitchhiker in which the Omega gets a glimpse at the source code after 7am, will modify to use TDT for all decisions in which Omega glimpses the source code after 7am.
No, I disagree. You seem to have missed this comment, or do you disagree with it?
But there are analogs of one-shot true PD everywhere.
Name a single one-shot true PD that any human has ever encountered in the history of time, and be sure to calculate the payoffs in inclusive fitness terms.
Of course that’s a rigged question—if you can tell me the name of the villain, I can either say “look how they didn’t have any children” or “their children suffered from the dishonor brought upon their parent”. But still, I think you are taking far too liberal a view of what constitutes one-shotness.
Empirically, humans ended up with both a sense of temptation and a sense of honor that, to the extent it holds, holds when no one is looking. We have separate impulses for “cooperate because I might get caught” and “cooperate because it’s the honorable thing to do”.
Regarding your other comment, “Do what my programmer would want me to do” is not formally defined enough for me to handle it—all the complexity is hidden in “would want”. Can you walk me through what you think a CDT agent self-modifies to if it’s not “use TDT for future decisions where Omega glimpsed my code after 7am and use CDT for future decisions where Omega glimpsed my code before 7am”? (Note that calculations about general population frequency count as “before 7am” from the crazed CDT’s perspective, because you’re reasoning from initial conditions that correlate to the AI’s state before 7am rather than after it.)
By “analog of one-shot true PD” I meant any game where the Nash equilibrium isn’t Pareto-optimal. The two links in my last comment gave plenty of examples.
all the complexity is hidden in “would want”
I think I formalized it already, but to say it again, suppose the creator had the option of creating a giant lookup table in place of S. What choice of GLT would have maximized his expected utility at the time of coding, under the creator’s own decision theory? S would compute that and then return whatever the GLT entry for X is.
ETA:
Can you walk me through what you think a CDT agent self-modifies to
It self-modifies to the S described above, with a description of itself embedded as the creator. Or to make it even simpler but less realistic, a CDT just replaces itself by a GLT, chosen to maximize its current expected utility.
By “analog of one-shot true PD” I meant any game where the Nash equilibrium isn’t Pareto-optimal. The two links in my last comment gave plenty of examples.
Suppose we have an indefinitely iterated PD with an unknown bound and hard-to-calculate but small probabilities of each round being truly unobserved. Do you call that “a game where the Nash equilibrium isn’t a Pareto optimum”? Do you think evolution has handled it by programming us to just defect?
I’ve done some informal psychological experiments to check human conformance with timeless decision theory on variants of the original Newcomb’s Problem, btw, and people who one-box on Newcomb’s Problem seem to have TDT intuitions in other ways. Not that this is at all relevant to the evolutionary dilemmas, which we seem to’ve been programmed to handle by being temptable, status-conscious, and honorable to variant quantitative degrees.
But programming an AI to cooperate with strangers on oneshot true PDs out of a human sense of honor would be the wrong move—our sense of honor isn’t the formal “my C iff (opponent C iff my C)”, so a TDT agent would then defect against us.
I just don’t see human evolution—status, temptation, honor—as being very relevant here. An AI’s decision theory will be, and should be, decided by our intuitions about logic and causality, not about status, temptation, and honor. Honor enters as a human terminal value, not as a decider of the structure of the decision theory.
How do you play “cooperate iff (the opponent cooperates iff I cooperate)” in a GLT? Is the programmer supposed to be modeling the opponent AI in sufficient resolution to guess how much the opponent AI knows about the programmer’s decision, and how many other possible programmers that the AI is modeling are likely to correlate with it? Does S compute the programmer’s decision using S’s knowledge or only the programmer’s knowledge? Does S compute the opponent inaccurately as if it were modeling only the programmer, or accurately as if it were modeling both the programmer and S?
I suppose that a strict CDT could replace itself with a GLT, if that GLT can take into account all info where the opponent AI gets a glimpse at the GLT after it’s written. Then the GLT behaves just like the code I specified before on e.g. Newcomb’s Problem—one-box if Omega glimpses the GLT or gets evidence about it after the GLT was written, two-box if Omega perfectly knows your code 5 seconds before the GLT gets written.
[Edit: Don’t bother responding to this yet. I need to think this through.]
How do you play “cooperate iff (the opponent cooperates iff I cooperate)” in a GLT?
I’m not sure this question makes sense. Can you give an example?
Does S compute the programmer’s decision using S’s knowledge or only the programmer’s knowledge?
S should take the programmer R’s prior and memories/sensory data at the time of coding, and compute a posterior probability distribution using them (assuming it would do a better job at this than R). Then use that to compute R’s expected utility for the purpose of computing the optimal GLT. This falls out of the idea that S is trying to approximate what the GLT would be if R had logical omniscience.
Is the programmer supposed to be modeling the opponent AI in sufficient resolution to guess how much the AI knows about the programmer?
No, S will do it.
Does S compute the opponent as if it were modeling only the programmer, or both the programmer and S?
I guess both, but I don’t understand the significance of this question.
Suppose what you say is correct, that the Winning Thing is to play cooperate in one-shot PD. Then what happens when some player happens to get a brain lesion that causes him to unconsciously play defect without affecting his AI building abilities? He would take everyone else’s lunch money.
Possibly. But it has to be an unpredictable brain lesion—one that is expected to happen with very low frequency. A predictable decision to do this just means that TDTs defect against you. If enough AI-builders do this then TDTs in general defect against each other (with a frequency threshold dependent on relative payoffs) because they have insufficient confidence that they are playing against TDTs rather than special cases in code.
Or if he builds his AI to play defect while everyone else builds their AIs to play cooperate, his AI then takes over the world.
No one is talking about building AIs to cooperate. You do not want AIs that cooperate on the one-shot true PD. You want AIs that cooperate if and only if the opponent cooperates if and only if your AI cooperates. So yes, if you defect when others expect you to cooperate, you can pwn them; but why do you expect that AIs would expect you to cooperate (conditional on their cooperation) if “the smart thing to do” is to build an AI that defects? AIs with good epistemic models would then just expect other AIs that defect.
The comment you responded to was mostly obsoleted by this one, which represents my current position. Please respond to that one instead. Sorry for making you waste your time!
I don’t think so. Each player wants to do the Winning Thing, and there is only one Winning Thing (their situations are symmetrical), so if they’re both good at Winning (a significantly lower bar than successfully building an AI with their preferences), their decisions are related.
So what you’re saying is, given two players who can successfully build AIs with their preferences (and that’s common knowledge), they will likely (surely?) play cooperate in one-shot PD against each other. Do I understand you correctly?
Suppose what you say is correct, that the Winning Thing is to play cooperate in one-shot PD. Then what happens when some player happens to get a brain lesion that causes him to unconsciously play defect without affecting his AI building abilities? He would take everyone else’s lunch money. Or if he builds his AI to play defect while everyone else builds their AIs to play cooperate, his AI then takes over the world. I hope that’s a sufficient reductio ad absurdum.
Hmm, I just noticed that you’re only saying “their decisions are related” and not explicitly making the conclusion they should play cooperative. Well, that’s fine, as long as they would play defect in one-shot PD, then they would also program their AIs to play defect in one-shot PD (assuming each AI can’t prove its source code to the other). That’s all I need for my argument.
Yes.
Good idea. Hmm. It sounds like this is the same question as: what if, instead of “TDT with defection patch” and “pure TDT”, the available options are “TDT with defection patch” and “TDT with tiny chance of defection patch”? Alternately: what if the abstract computations that are the players have a tiny chance of being embodied in such a way that their embodiments always defect on one-shot PD, whatever the abstract computation decides?
It seems to me that Lesion Man just got lucky. This doesn’t mean people can win by giving themselves lesions, because that’s deliberately defecting / being an abstract computation that defects, which is bad. Whether everyone else should defect / program their AIs to defect due to this possibility depends on the situation; I would think they usually shouldn’t. (If it’s a typical PD payoff matrix, there are many players, and they care about absolute, not relative, scores, defecting isn’t worth it even if it’s guaranteed there’ll be one Lesion Man.)
This still sounds disturbingly like envying Lesion Man’s mere choices – but the effect of the lesion isn’t really his choice (right?). It’s only the illusion of unitary agency, bounded at the skin rather than inside the brain, that makes it seem like it is. The Cartesian dualism of this view (like AIXI, dropping an anvil on its own head) is also disturbing, but I suspect the essential argument is still sound, even as it ultimately needs to be more sophisticated.
I guess my reductio ad absurdum wasn’t quite sufficient. I’ll try to think this through more thoroughly and carefully. Let me know which steps, if any, you disagree with, or are unclear, in the following line of reasoning.
TDT couldn’t have arisen by evolution.
Until a few years ago, almost everyone on Earth was running some sort of non-TDT which plays defect in one-shot PD.
It’s possible that upon learning about TDT, some people might spontaneously switch to running it, depending on whatever meta-DT controls this, and whether the human brain is malleable enough to run TDT.
If, in any identifiable group of people, a sufficient fraction switches to TDT, and that proportion is public knowledge, the TDT-running individuals in that group should start playing cooperate in one-shot PD with other members of the group.
The threshold proportion is higher if the remaining defectors can cause greater damage. If the remaining defectors can use their gains from defection to better reproduce themselves, or to gather more resources that will let them increase their gains/damage, then the threshold proportion must be close to 1, because even a single defector can start a chain reaction that causes all the resources of the group to become held by defectors.
What proportion of skilled AI designers would switch to TDT is ultimately an empirical question, but it seems to me that it’s unlikely to be close to unity.
TDT-running AI designers will design their AIs to run TDT. Non-TDT-running AI designers will design their AIs to run non-TDT (not necessarily the same non-TDT).
Assume that a TDT-running AI (TAI) can’t tell which other AIs are running TDT and which ones aren’t, so in every game it faces the decision described in steps 4 and 5. A TDT AI will cooperate in some situations where the benefit from cooperation is relatively high and damage from defection relatively low, and not in other situations.
As a result, non-TAI will do better than TAI, but the damage to TAIs will be limited.
Only if a TAI is sure that all AIs are TAIs, will it play cooperate unconditionally.
If a TAI encounters an AI of alien origin, the same logic applies. The alien AI will be TAI if-and-only-if its creator was running TDT. If the TAI knows nothing about the alien creator, then it has to estimate what fraction of AI-builders in the universe runs TDT. Taking into account that TDT can’t arise from evolution, and not seeing any reason for evolution to create a meta-DT that would pick TDT upon discovering it, this fraction seems pretty low, and so the TAI will likely play defect against the alien AI.
Hmm, this exercise has cleared a lot of my own confusion. Obviously a lot more work needs to be done to make the reasoning rigorous, but hopefully I’ve gotten the gist of it right.
ETA: According to this line of argument, your hypothesis that all skilled AI designers play cooperate in one-shot PD against each other is equivalent to saying that skilled AI designers have minds malleable enough to run TDT, and have a meta-DT that causes them to switch to running TDT. But I do not see an evolutionary reason for this, so if it’s true, it must be true by luck. Do you agree?
Btw, agree with steps 3-9.
It looks like in this discussion you assume that switching to “TDT” (it’s highly uncertain what this means) immediately gives the decision to cooperate in “true PD”. I don’t see why it should be so. Summarizing my previous comments, exactly what the players know about each other, exactly in what way they know it, may make their decisions go either way. That the players switch from CDT to some kind of more timeless decision theory doesn’t determine the answer to be “cooperate”, it merely opens up the possibility that previously was decreed irrational, and I suspect that what’s important in the new setting for making the decision go either way isn’t captured properly in the problem statement of “true PD”.
Also, the way you treat “agents with TDT” seems more appropriate for “agents with Cooperator prefix” from cousin_it’s Formalizing PD. And this is a simplified thing far removed from a complete decision theory, although a step in the right direction.
I don’t assume that switching to TDT immediately gives the decision to cooperate in “true PD”. I assume that an AI running TDT would decide to cooperate if it thinks the expected utility of cooperating is higher than the EU of defecting, and that is true if its probability of facing another TDT is sufficiently high compared to its probability of facing a defector (how high is sufficient depends on the payoffs of the game). Well, this is necessary but not sufficient. For example if the other TDT doesn’t think its probability of facing a TDT is high enough, it won’t cooperate, so we need some common knowledge of the relevant probabilities and payoffs.
Does my line of reasoning make sense now, given this additional explanation?
Actually it makes less sense now, since your explanation seems to agree that two “TDT” algorithms that know each of them is “TDT” won’t necessarily cooperate, which undermines my hypothesis for why you were talking about cooperation as a sure thing in some relation to “TDT”. I still think you make that assumption though. Citation from your argument:
I’m having trouble understanding what you’re talking about again. Do you agree or disagree with step 4? To rephrase it a bit, if an identifiable group of people contains a high fraction of individuals running TDT, and that proportion is public knowledge, then TDT-running individuals in that group should play cooperate in one-shot PD with other members of the group in games where the payoffs are such that potential gains from mutual cooperation is large compared to potential loses from being defected against. (Assuming being in such a group is the best evidence available about whether someone is running TDT or not.)
If you disagree, why do you think a TDT-running individual might not play cooperate in this situation? Can you give an example to help me understand?
See my reply to Eliezer.
I disagree with step 4, I think sometimes the TDT players that know they both are TDT players won’t cooperate, but this discussion stirred up some of the relevant issues, so I’ll answer later when I figure out what I should believe now.
I don’t see why TDT players would fail to cooperate under conditions of common knowledge. Are you talking about a case where they each know the other is TDT but think the other doesn’t know they know, or something like that?
I don’t know the whole answer, but for example consider what happens with Pareto-efficiency in PD when you allow mixed strategies (and mixed strategy is essentially the presence of nontrivial dependence of the played move on the agent’s state of knowledge, beyond what is restricted by the experiment, so there is no actual choice about allowing mixed strategies, mixed strategies are what’s there by default even if the problem states that players select some certain play). Now, the Pareto-efficient plays are those where one player cooperates with certainty, while the other cooperates or defects with some probability. These strategies correspond to bargaining between the players. I don’t know how to solve the bargaining problem (aka fairness problem aka first-mover problem in TDT), but I see no good reason to expect that the solution in this case is going to be exactly pure cooperation. Which is what I meant by the insufficiency in correspondence between true PD and pure cooperation: true PD seems to give too little info, leaving uncertainty about the outcome, at least in this sense. This example doesn’t allow both players to defect, but it’s not pure cooperation either.
It’s too elegant to arise by evolution, and it also deals with one-shot PDs with no knock-on effects which is an extremely nonancestral condition—evolution by its nature deals with events that repeat many times; sexual evolution by its nature deals with organisms that interbreed; so “one-shot true PDs” is in general a condition unlikely to arise with sufficient frequency that evolution deals with it at all.
This may perhaps embody the main point of disagreement. A self-modifying CDT which, at 7am, expects to encounter a future Newcomb’s Problem or Parfit’s Hitchhiker in which the Omega gets a glimpse at the source code after 7am, will modify to use TDT for all decisions in which Omega glimpses the source code after 7am. A bit of “common sense” would tell you to just realize that “you should have been using TDT from the beginning regardless of when Omega glimpsed your source code and the whole CDT thing was a mistake” but this kind of common sense is not embodied in CDT. Nonetheless, TDT is a unique reflectively consistent answer for a certain class of decision problems, and a wide variety of initial points is likely to converge to it. The exact proportion, which determines under what conditions of payoff and loss stranger-AIs will cooperate with each other, is best left up to AIs to calculate, I think.
The main problem I see with this thesis (to restate my position in a hopefully clear form) is that an agent that starts off with a DT that unconditionally plays D in one-shot PD will not self-modify into TDT, unless it has some means of giving trustworthy evidence that it has done so. Suppose there is no such means, then any other agent must treat it the same, whether it self-modifies into TDT or not. Suppose it expects to face a TDT agent in the future. Whether that agent will play C or D against it is independent of what it decides now. If it does self-modify into TDT, then it might play C against the other TDT where it otherwise would have played D, and since the payoff for C is lower than for D, holding the other player’s choice constant, it will decide not to self-modify into TDT.
If it expects to face Newcomb’s Problem, then it would self-modify into something that handles it better, but that something must still unconditionally play D in one-shot PD.
Do you still think “a wide variety of initial points is likely to converge to it”? If so, do you agree that (ETA: in a world where proving source code isn’t possible) those initial points exclude any DT that unconditionally plays D in one-shot PD?
BTW, there are a number of decision theorists in academia. Should we try to get them to work on our problems? Unfortunately, I have no skill/experience/patience/willpower for writing academic papers. I tried to write such a paper about cryptography once and submitted it to a conference, got back a rejection with nonsensical review comments, and that was that. (I guess I could have tried harder but then that would probably have put me on a different career path where I wouldn’t be working these problems today.)
Also, there ought to be lots of mathematicians and philosophers who would be interested in the problem of logical uncertainty. How can we get them to work on it?
Unless that agent already knows or can guess your source code, in which case it is simulating you or something highly correlated to you, and in which case “modify to play C only if I expect that other agent simulating me to play C iff I modify to play C” is a superior strategy to “just D” because an agent who simulates you making the former choice (and which expects to be correctly simulated itself) will play C against you, while if it simulates you making the latter choice it will play D against you.
The whole point is that the other player’s choice is not constant. Otherwise there is no reason ever for anyone to play C in a one-shot true PD! Simulation introduces logical dependencies—that’s the whole point and to the extent it is not true even TDT agents will play D.
“Holding the other player’s choice constant” here is the equivalent of “holding the contents of the boxes constant” in Newcomb’s Problem. It presumes the answer.
I think you’re invoking TDT-style reasoning here, before the agent has self-modified into TDT.
Besides, I’m assuming a world where agents can’t know or guess each others’ source codes. I thought I made that clear. If this assumption doesn’t make sense to you, consider this: What evidence can one AI use to infer the source code of another AI or its creator? What if any such evidence can be faked near perfectly by the other AI? What about for two AIs of different planetary origins meeting in space?
I know you’d like to assume a world where guessing each others’ source code is possible, since that makes everything work out nicely and everyone can “live happily ever after”. But why shouldn’t we consider both possibilities, instead of ignoring the less convenient one?
ETA: I think it may be possible to show that a CDT won’t self-modify into a TDT as long as it believes there is a non-zero probability that it lives in a world where it will encounter at least one agent that won’t know or guess its current or future source code, but in the limit as that probability goes to zero, the DT it self-modifies into converges to TDT.
I already said that agents which start out as pure CDT won’t modify into pure TDTs—they’ll only cooperate if someone gets a peek at their source code after they self-modified. However, humans, at least, are not pure CDT agents—they feel at least the impulse to one-box on Newcomb’s Problem if you raise the stakes high enough.
This has nothing to do with evolutionary contexts of honor and cooperation and defection and temptation, and everything to do with our evolved instincts governing abstract logic and causality, which is what governs what sort of source code you think has what sort of effect. Even unreasonably pure CDT agents recognize that if they modify their source code at 7am, they should modify to play TDT against any agent that has looked at their source code after 7am. To humans, who are not pure CDT agents, the idea that you should play essentially the same way if Omega glimpsed your source code at exactly 6:59am, seems like common sense given the intuitions we have about logic and causality and elegance and winning. If you’re going to all the trouble to invent TDT anyway, it seems like a waste of effort to two-box against Omega if he perfectly saw your source code 5 seconds before you self-modified. (These being the kind of ineffable meta-decision considerations that we both agree are important, but which are hard to formalize.)
You are guessing their source code every time you argue that they’ll choose D. If I can’t make you see this as an instance of “guessing the other agent’s source code” then indeed you will not see the large correlations at the start point, and if the agents start out highly uncorrelated then the rare TDT agents will choose the correct maximizing action, D. They will be rare because, by assumption in this case, most agents end up choosing to cooperate or defect for all sorts of different reasons, rather than by following highly regular lines of logic in nearly all cases—let alone the same line of logic that kept on predictably ending up at D.
There’s a wide variety of cases where philosophers go astray by failing to recognize an instance of an everyday concept as an abstract concept. For example, they say in one breath that “God is unfalsifiable”, and in the next breath talk about how God spoke to them in their heart, because they don’t recognize “God spoke to me in my heart” as an instance of “God allegedly made something observable happen”. Philosophers talk about qualia being epiphenomenal in one breath, and then in the next speak of how they know themselves to be conscious, because they don’t recognize this self-observation as an instance of “something making something else happen” aka “cause and effect”. The only things recognized as matching the formal-sounding phrase “cause and effect” are big formal things officially labeled “causal”, not just stuff that makes other stuff happen.
In the same sense, you have this idea about modeling other agents as this big official affair that requires poring over their source code with a magnifying glass and then furthermore verifying that they can’t change it while you aren’t looking.
You need to recognize the very thought processes you are carrying out right now in arguing that just about anyone will choose D as an instance of guessing the outputs of the other agents’ source codes and moreover guessing that most such codes and outputs are massively logically correlated.
This is witnessed by the fact that if we did get to see some interstellar transactions, and you saw that the first three transactions were (C, C), you would say, “Wow, guess Eliezer was right” and expect the next one to be (C, C) as well. (And of course if I witnessed three cases of (D, D) I would say “Guess I was wrong.”) Even though the initial conditions are not physically correlated, we expect a correlation. What is this correlation, then? It is a logical correlation. We expect different species to end up following similar lines of reasoning, that is, performing similar computations, like factorizing 123,456 in spacelike separated galaxies.
It occurs to me that the problem is important enough that even if we can reach intuitive agreement, we should still do the math. But it doesn’t help to solve the wrong problem, so do you think the following is the right formalization of the problem?
Assume a “no physical proof of source code” universe.
Assume three types of intelligent life can arise in this universe.
In a Type A species, Eliezer’s intuition is obvious to everyone, so they build AIs running TDT without further consideration.
In a Type B species, my intuition is obvious to everyone so, so they build AIs running XDT, or AIs running CDT which immediately self-modify into XDT. Assume (or prove) that XDT behaves like TDT except it unconditionally plays D in PD.
In a Type C species, different people have different intuitions, and some (Type D individuals) don’t have strong intuitions or prefer to use a formal method to make this meta-decision. We human beings obviously belong to this type of species, and let’s say we at LessWrong belong to this last subgroup (Type D).
Does this make sense so far?
Let me say where my intuition expects this to lead to, so you don’t think I’m setting a trap for you to walk into. Whatever meta-decision we make, it can be logically correlated only to AIs running TDT and other Type D individuals in the universe. If the proportion of Type D individuals in the universe is low, then it’s obviously better for us to implement XDT instead of TDT. That’s because whether we use TDT or XDT will have little effect on how often other TDTs play cooperate. (They can predict what Type D individuals will decide, but since there are few of us and they can’t tell which AIs were created by Type D individuals, it won’t affect their decisions much.)
Unfortunately we don’t know the proportions of different types of species/individuals. So we should program an AI to estimate them, and have it make the decision of what to self-modify into.
ETA: Just realized that the decisions of Type D individuals can also correlate with the intuitions of others, since intuitions come from unconscious mental computations and they may be of a similar nature with our explicit decisions. But this correlation will be imperfect, so the above reasoning still applies, at least to some extent.
ETA2: This logical correlation stuff is hard to think about. Can we make any sense of these types of problems before having a good formal theory of logical correlation?
ETA3: The thing that’s weird here is that assuming everyone’s intuitions/decisions aren’t perfectly correlated, some will build TDTs and some will build XDTs. And it will be the ones who end up deciding to build XDTs that defect who will win. How to make sense of this, if that’s the wrong decision?
ETA4: I’ll be visiting Mt. Rainier for the rest of the day, so that’s it. :) Sorry for the over-editing.
Maybe cousin_it is right and we really have to settle this by formal math. But I’m lazy and will give words one more try. If we don’t reach agreement after this I’m going to the math.
So, right now we have different intuitions. Let’s say you have the correct intuition and convince everyone of it, and I have the incorrect one but I’m too stupid to realize it. So you and your followers go on to create a bunch of AIs with TDT. I go on to create an AI which is like TDT except it plays defect in PD. Lets say I pretended to be your follower and we never had this conversation, so there is no historical evidence that I would create such an AI. When my AI is born, it modifies my brain so that I start to believe I created an AI with TDT, thus erasing the last shred of evidence. My AI will then go on and win against every other AI.
Given the above, why should I change my mind now, and not win?
ETA: Ok, I realize this is pretty much the same scenario as the brain lesion one, except it’s not just possible, it’s likely. Someone is bound to have my intuition and be resistant to your persuasion. If you say that smart agents win, then he must be the smart one, right?
This I think connects one more terminological distinction. When you talked earlier about something like “reasoning about the output of platonic computation” as a key insight that started your version of TDT, you meant basically the same thing as me talking about how even knowing about the existence of the other agent, little things that you can use to even talk about it, is already the logical dependence between you and the other agent that could in some cases be used to stage cooperation.
But there are analogs of one-shot true PD everywhere.
No, I disagree. You seem to have missed this comment, or do you disagree with it?
Name a single one-shot true PD that any human has ever encountered in the history of time, and be sure to calculate the payoffs in inclusive fitness terms.
Of course that’s a rigged question—if you can tell me the name of the villain, I can either say “look how they didn’t have any children” or “their children suffered from the dishonor brought upon their parent”. But still, I think you are taking far too liberal a view of what constitutes one-shotness.
Empirically, humans ended up with both a sense of temptation and a sense of honor that, to the extent it holds, holds when no one is looking. We have separate impulses for “cooperate because I might get caught” and “cooperate because it’s the honorable thing to do”.
Regarding your other comment, “Do what my programmer would want me to do” is not formally defined enough for me to handle it—all the complexity is hidden in “would want”. Can you walk me through what you think a CDT agent self-modifies to if it’s not “use TDT for future decisions where Omega glimpsed my code after 7am and use CDT for future decisions where Omega glimpsed my code before 7am”? (Note that calculations about general population frequency count as “before 7am” from the crazed CDT’s perspective, because you’re reasoning from initial conditions that correlate to the AI’s state before 7am rather than after it.)
By “analog of one-shot true PD” I meant any game where the Nash equilibrium isn’t Pareto-optimal. The two links in my last comment gave plenty of examples.
I think I formalized it already, but to say it again, suppose the creator had the option of creating a giant lookup table in place of S. What choice of GLT would have maximized his expected utility at the time of coding, under the creator’s own decision theory? S would compute that and then return whatever the GLT entry for X is.
ETA:
It self-modifies to the S described above, with a description of itself embedded as the creator. Or to make it even simpler but less realistic, a CDT just replaces itself by a GLT, chosen to maximize its current expected utility.
Is that sufficiently clear?
Suppose we have an indefinitely iterated PD with an unknown bound and hard-to-calculate but small probabilities of each round being truly unobserved. Do you call that “a game where the Nash equilibrium isn’t a Pareto optimum”? Do you think evolution has handled it by programming us to just defect?
I’ve done some informal psychological experiments to check human conformance with timeless decision theory on variants of the original Newcomb’s Problem, btw, and people who one-box on Newcomb’s Problem seem to have TDT intuitions in other ways. Not that this is at all relevant to the evolutionary dilemmas, which we seem to’ve been programmed to handle by being temptable, status-conscious, and honorable to variant quantitative degrees.
But programming an AI to cooperate with strangers on oneshot true PDs out of a human sense of honor would be the wrong move—our sense of honor isn’t the formal “my C iff (opponent C iff my C)”, so a TDT agent would then defect against us.
I just don’t see human evolution—status, temptation, honor—as being very relevant here. An AI’s decision theory will be, and should be, decided by our intuitions about logic and causality, not about status, temptation, and honor. Honor enters as a human terminal value, not as a decider of the structure of the decision theory.
How do you play “cooperate iff (the opponent cooperates iff I cooperate)” in a GLT? Is the programmer supposed to be modeling the opponent AI in sufficient resolution to guess how much the opponent AI knows about the programmer’s decision, and how many other possible programmers that the AI is modeling are likely to correlate with it? Does S compute the programmer’s decision using S’s knowledge or only the programmer’s knowledge? Does S compute the opponent inaccurately as if it were modeling only the programmer, or accurately as if it were modeling both the programmer and S?
I suppose that a strict CDT could replace itself with a GLT, if that GLT can take into account all info where the opponent AI gets a glimpse at the GLT after it’s written. Then the GLT behaves just like the code I specified before on e.g. Newcomb’s Problem—one-box if Omega glimpses the GLT or gets evidence about it after the GLT was written, two-box if Omega perfectly knows your code 5 seconds before the GLT gets written.
[Edit: Don’t bother responding to this yet. I need to think this through.]
I’m not sure this question makes sense. Can you give an example?
S should take the programmer R’s prior and memories/sensory data at the time of coding, and compute a posterior probability distribution using them (assuming it would do a better job at this than R). Then use that to compute R’s expected utility for the purpose of computing the optimal GLT. This falls out of the idea that S is trying to approximate what the GLT would be if R had logical omniscience.
No, S will do it.
I guess both, but I don’t understand the significance of this question.
Or:
Possibly. But it has to be an unpredictable brain lesion—one that is expected to happen with very low frequency. A predictable decision to do this just means that TDTs defect against you. If enough AI-builders do this then TDTs in general defect against each other (with a frequency threshold dependent on relative payoffs) because they have insufficient confidence that they are playing against TDTs rather than special cases in code.
No one is talking about building AIs to cooperate. You do not want AIs that cooperate on the one-shot true PD. You want AIs that cooperate if and only if the opponent cooperates if and only if your AI cooperates. So yes, if you defect when others expect you to cooperate, you can pwn them; but why do you expect that AIs would expect you to cooperate (conditional on their cooperation) if “the smart thing to do” is to build an AI that defects? AIs with good epistemic models would then just expect other AIs that defect.
The comment you responded to was mostly obsoleted by this one, which represents my current position. Please respond to that one instead. Sorry for making you waste your time!