Great post. I’m going to riff on it to talk about what it would look like to have an epistemology which formally explains/predicts the stuff in this essay.
Paranoia is a hard thing to model from a Bayesian perspective, because there’s no slot to insert an adversary who might fuck you over in ways you can’t model (and maybe this explains why people were so confused about the Market for Lemons paper? Not sure). However, I think it’s a very natural concept from a Knightian perspective. My current guess is that the correct theory of Knightian uncertainty will be able to formulate the concept of paranoia in a very “natural” way (and also subsume Bayesian uncertainty as a special case where you need zero paranoia because you’re working in a closed domain which you have a mechanistic understanding of).
The worst-case assumption in infra-Bayesianism (and the maximin algorithm more generally, e.g. as used in chess engines) is one way of baking in a high level of paranoia. However, two drawbacks of that approach:
There’s no good way to “dial down” the level of paranoia. I.e. we don’t have an elegant version of maximin to apply to settings where your adversary isn’t always choosing the worst possibility for you.
The closest I have is the Hurwicz criterion, which basically sets the ratio of focusing on the worst outcome to focusing on the best outcome. But this is very hacky—the thing you actually care about is all the intermediate outcomes.
It’s not actually paranoid in a Knightian way, because what if your adversary does something that you didn’t even think of?
Another way of being paranoid is setting large bid-ask spreads. I assume that finance people have a lot to say about how to set bid-ask spreads, but I haven’t heard of any very elegant theory.
I think of Sahil’s live theory as being a theory of anti-paranoia. It’s the approach you take in a world which is fundamentally “friendly” to you. It’s still not very pinned-down, though.
I think your three approaches to dealing with an adversarial world all gestures to valuable directions for formal investigation. I think of “blinding yourself” in terms of maintaining a boundary. The more paranoid you are, the stronger a boundary you need to have between yourself and the outside world. Boundaries are Knightian in the sense that they allow you to get stuff done without actually knowing much about what’s in the external world. My favorite example here (maybe from Sahil?) is the difference between a bacterium and a cell inside a human body. A bacterium is in a hostile world and therefore needs to maintain strong boundaries. Conversely, a cell inside a body can mostly assume that the chemicals in its environment are there for its benefit, and so can be much more permeable to them. We want to be able to make similar adjustments on an information level (and also on a larger-scale physical level).
I think of “purging the untrustworthy” in terms of creating/maintaining group identity. I expect that this can be modeled in terms of creating commitments to behave certain ways. The “healthy” version is creating a reputation which you don’t want to undermine because it’s useful for coordination (as discussed e.g. here). The unhealthy version is to traumatize people into changing their identities, by inducing enough suffering to rearrange their internal coalitions (I have a long post coming up on how this explains the higher education system; the short version is here).
I think of “become unpredictable” in terms of asymmetric strategies which still work against entities much more intelligent than you. Ivan has a recent essay about encryption as an asymmetric weapon which is robust to extremely powerful adversaries. I’m reminded of an old Eliezer essay about how, if you’re using noise in an algorithm, you’re doing it wrong. That’s true from a Bayesian perspective, but it’s very untrue from a (paranoid) Knightian perspective. Another example of an asymmetric weapon: no matter how “clever” your drone is, it probably can’t figure out a way to fly directly towards a sufficiently powerful fan (because the turbulence is too chaotic to exploit).
I think that the good version of “become vindictive” is something to do with virtue ethics. I think of virtue ethics as a strategy for producing good outcomes even when dealing with entities (particularly collectives) that are much more capable than you. This is also true of deontology (see passage in HPMOR where Hermione keeps getting obliviated). I think consequentialism works pretty well in low-adversarialness environments, virtue ethics works in medium-adversarialness environments, and then deontology is most important in the most adversarial environments, because as you go from the former to the latter you are making decisions in ways which have fewer and fewer degrees of freedom to exploit.
Hopefully much more on all of this soon, but thank you for inspiring me to get out at least a rough set of pointers.
Paranoia is a hard thing to model from a Bayesian perspective, because there’s no slot to insert an adversary who might fuck you over in ways you can’t model (and maybe this explains why people were so confused about the Market for Lemons paper? Not sure).
I don’t see why you’d think that. The market for lemons model works perfectly in a bayesian framework. It’s very easy to model the ways that the adversary fucks you over. (They do it by lying to you about what state of the world you’re in, in a setting where you can’t observe any evidence to distinguish those states of the world.)
Corrolary: You’d have an inconsitent definition of “paranoia” if you simultaneously want to say that “market for lemons” involves paranoia, and also want to treat “Bayesian uncertainty as a special case where you need zero paranoia”.
I agree with Richard here, in that the market for lemon’s model is really hard to extend to the full “open source game theory case”.
Proving this isn’t very hard. Bayesianism has a realizability assumption, but of course if you have an adversary that is smarter than you, you can’t model them, and so you can’t form a bayesian hypothesis about what they will do.
This then extends into logical uncertainty, and solutions to logical uncertainty produce things that are kind of bayesian but not quite (like logical induction), and how to actually translate that into real-life thinking feels still like a largely open question.
Both the market for lemons in econ and adverse selection in trading are simple examples of models of adversarial dynamics. I would call these non-central examples of paranoia insofar as you know the variable about which your adversary is hiding information (the quality of the car/the price the stock should be). This makes them too simple to get at the heart of the phenomenon.
I think Habyrka is gesturing at something similar in his paragraph starting “All that said, in reality, navigating a lemon market isn’t too hard.” And I take him to be gesturing at a more central description of paranoia in his subsequent description: “What do you do in a world in which there are not only sketchy used car salesmen, but also sketchy used car inspectors, and sketchy used car inspector rating agencies, or more generally, competent adversaries who will try to predict whatever method you will use to orient to the world, and aim to subvert it for their own aims?”
This is similar to my criticism of maximin as a model of paranoia: “It’s not actually paranoid in a Knightian way, because what if your adversary does something that you didn’t even think of?”
Here’s a gesture at making this more precise: what makes something a central example of paranoia in my mind is when even your knowledge of how your adversary is being adversarial is also something that has been adversarially optimized. Thus chess is not a central example of paranoia (except insofar as your opponent has been spying on your preparations, say) and even markets for lemons aren’t a central example (except insofar as buyers weren’t even tracking that dishonesty was a strategy sellers might use—which is notably a dynamic not captured by the economic model).
I agree with almost all of this, though I think you don’t need to invoke knightian uncertainty. I think it’s simply enough to model there being a very large attack surface combined with a more intelligent adversary.
See this section of my reply to Unnamed about some semi-formal models in the space:
I would definitely love a better model of whether these really are the exhaustively correct strategies. I have some handwavy pointers to why I roughly think they are, but they are pretty handwavy at this point. Trying to elucidate them a tiny bit right now:
The fundamental issue that paranoia is trying to deal with is the act of an adversary predicting your outputs well-enough that to them, you can basically be treated as part of the environment (in MIRI-adjacent circles I’ve sometimes heard this referred to as “diagonalization”).
If I think about this in a Computer-Scienc-y way, I am imagining a bigger agent that is simulating a smaller agent, with a bunch of input channels that represent the observations the smaller agent makes of the world. Some fraction of those input channels can be controlled. The act of diagonalization is basically finding some set of controllable inputs that, no matter what the uncontrollable parts of the input say[1], result in the smaller agent doing what the bigger agent wants.
Now, in this context, three strategies stand out to me that conceptually make sense:
You cut off your internal dependence to the controlled input channels
You reduce the amount of information that your adversary has about your internals so they can model your internals less well
You make yourself harder to predict, either by performing complicated computations to determine your actions, or making what kind of computation you perform to arrive at the result highly dependent on input channels you know are definitely uncontrolled
And like… in this very highly simplified CS model, those are roughly the three strategies that make sense to me at all? I can’t think of anything else that makes sense to do, though maybe it’s just a lack of imagination. Like, I feel like you have varied all the variables that make sense to vary in this toy-model.
And of course, it’s really unclear how well this toy-model translates to reality! But it’s one of the big generators that made me think the “3 strategies” claim makes sense.
Or maybe not full independence but very strong correlation. The details of this actually matter a lot, and this is where the current magic lives, but we can look at the “guaranteed output” case for now.
though I think you don’t need to invoke knightian uncertainty. I think it’s simply enough to model there being a very large attack surface combined with a more intelligent adversary.
One of the problems I’m pointing to is that you don’t know what the attack surface is. This puts you in a pretty different situation than if you have a known large attack surface to defend, even against a smarter adversary (e.g. the whole length of a border; or every possible sequence of Go moves).
Separately, I may be being a bit sloppy by using “Knightian uncertainty” as a broad handle for cases where you have important “unknown unknowns”, aka you don’t even know what ontology to use. But it feels close enough that I’m by default planning to continue describing the research project outlined above as trying to develop a theory of Knightian uncertainty in which Bayesian uncertainty is a special case.
I think consequentialism works pretty well in low-adversarialness environments, virtue ethics works in medium-adversarialness environments, and then deontology is most important in the most adversarial environments, because as you go from the former to the latter you are making decisions in ways which have fewer and fewer degrees of freedom to exploit.
I’ve been thinking about this a lot recently. It seems we could generalize this beyond adversarialness to uncertainty more broadly: In a low-uncertainty environment, consequentialism seems more compelling; in a high-uncertainty environment, deontology makes sense (because as you go from the former to the latter you are making decisions in ways which rest on fewer and fewer error-prone assumptions).
However, this still feels unsatisfying to me for a couple of reasons: (1) In a low-uncertainty environment, there is still some uncertainty. It doesn’t seem to make sense for an actor to behave in violation of their felt sense of morality to achieve a “good” outcome unless they are omniscient and can perfectly predict all indirect effects of their actions.[1] And, if they were truly omniscient, then deontic and consequentialist approaches might converge on similar actions—at least Derek Parfit argues this. I don’t know if I buy this, because (2) why do we value outcomes over the experiences by which we arrive at them? This presupposes consequentialism, which seems increasingly clearly misaligned with human psychology—e.g., the finding that maximizers are unhappier than satisficers, despite achieving “objectively” better outcomes, or the finding that happiness-seeking is associated with reduced happiness.
Relating this back to the question of reasoning in high-adversarial environments, it seems to me that the most prudent (and psychologically protective) approach is a deontological one, not only because it is more robust to outcome-thwarting by adversaries but more importantly because it is (a) positively associated with wellbeing and empathy and (b) inversely associated with power-seeking. See also here.
Moreover, one would need to be omniscient to accurately judge the uncertainty/adversarialness of their environment, so it probably makes sense to assume a high-uncertainty/high-adversarialness environment regardless (at least, if one cares about this sort of thing).
Great post. I’m going to riff on it to talk about what it would look like to have an epistemology which formally explains/predicts the stuff in this essay.
Paranoia is a hard thing to model from a Bayesian perspective, because there’s no slot to insert an adversary who might fuck you over in ways you can’t model (and maybe this explains why people were so confused about the Market for Lemons paper? Not sure). However, I think it’s a very natural concept from a Knightian perspective. My current guess is that the correct theory of Knightian uncertainty will be able to formulate the concept of paranoia in a very “natural” way (and also subsume Bayesian uncertainty as a special case where you need zero paranoia because you’re working in a closed domain which you have a mechanistic understanding of).
The worst-case assumption in infra-Bayesianism (and the maximin algorithm more generally, e.g. as used in chess engines) is one way of baking in a high level of paranoia. However, two drawbacks of that approach:
There’s no good way to “dial down” the level of paranoia. I.e. we don’t have an elegant version of maximin to apply to settings where your adversary isn’t always choosing the worst possibility for you.
The closest I have is the Hurwicz criterion, which basically sets the ratio of focusing on the worst outcome to focusing on the best outcome. But this is very hacky—the thing you actually care about is all the intermediate outcomes.
It’s not actually paranoid in a Knightian way, because what if your adversary does something that you didn’t even think of?
Another way of being paranoid is setting large bid-ask spreads. I assume that finance people have a lot to say about how to set bid-ask spreads, but I haven’t heard of any very elegant theory.
I think of Sahil’s live theory as being a theory of anti-paranoia. It’s the approach you take in a world which is fundamentally “friendly” to you. It’s still not very pinned-down, though.
I think your three approaches to dealing with an adversarial world all gestures to valuable directions for formal investigation. I think of “blinding yourself” in terms of maintaining a boundary. The more paranoid you are, the stronger a boundary you need to have between yourself and the outside world. Boundaries are Knightian in the sense that they allow you to get stuff done without actually knowing much about what’s in the external world. My favorite example here (maybe from Sahil?) is the difference between a bacterium and a cell inside a human body. A bacterium is in a hostile world and therefore needs to maintain strong boundaries. Conversely, a cell inside a body can mostly assume that the chemicals in its environment are there for its benefit, and so can be much more permeable to them. We want to be able to make similar adjustments on an information level (and also on a larger-scale physical level).
I think of “purging the untrustworthy” in terms of creating/maintaining group identity. I expect that this can be modeled in terms of creating commitments to behave certain ways. The “healthy” version is creating a reputation which you don’t want to undermine because it’s useful for coordination (as discussed e.g. here). The unhealthy version is to traumatize people into changing their identities, by inducing enough suffering to rearrange their internal coalitions (I have a long post coming up on how this explains the higher education system; the short version is here).
I think of “become unpredictable” in terms of asymmetric strategies which still work against entities much more intelligent than you. Ivan has a recent essay about encryption as an asymmetric weapon which is robust to extremely powerful adversaries. I’m reminded of an old Eliezer essay about how, if you’re using noise in an algorithm, you’re doing it wrong. That’s true from a Bayesian perspective, but it’s very untrue from a (paranoid) Knightian perspective. Another example of an asymmetric weapon: no matter how “clever” your drone is, it probably can’t figure out a way to fly directly towards a sufficiently powerful fan (because the turbulence is too chaotic to exploit).
I think that the good version of “become vindictive” is something to do with virtue ethics. I think of virtue ethics as a strategy for producing good outcomes even when dealing with entities (particularly collectives) that are much more capable than you. This is also true of deontology (see passage in HPMOR where Hermione keeps getting obliviated). I think consequentialism works pretty well in low-adversarialness environments, virtue ethics works in medium-adversarialness environments, and then deontology is most important in the most adversarial environments, because as you go from the former to the latter you are making decisions in ways which have fewer and fewer degrees of freedom to exploit.
Hopefully much more on all of this soon, but thank you for inspiring me to get out at least a rough set of pointers.
I don’t see why you’d think that. The market for lemons model works perfectly in a bayesian framework. It’s very easy to model the ways that the adversary fucks you over. (They do it by lying to you about what state of the world you’re in, in a setting where you can’t observe any evidence to distinguish those states of the world.)
Corrolary: You’d have an inconsitent definition of “paranoia” if you simultaneously want to say that “market for lemons” involves paranoia, and also want to treat “Bayesian uncertainty as a special case where you need zero paranoia”.
I agree with Richard here, in that the market for lemon’s model is really hard to extend to the full “open source game theory case”.
Proving this isn’t very hard. Bayesianism has a realizability assumption, but of course if you have an adversary that is smarter than you, you can’t model them, and so you can’t form a bayesian hypothesis about what they will do.
This then extends into logical uncertainty, and solutions to logical uncertainty produce things that are kind of bayesian but not quite (like logical induction), and how to actually translate that into real-life thinking feels still like a largely open question.
Yeah, I don’t disagree with anything in this comment. I was just reacting to the market for lemons comparison.
Ah, cool, makes sense!
Fair point. Let me be more precise here.
Both the market for lemons in econ and adverse selection in trading are simple examples of models of adversarial dynamics. I would call these non-central examples of paranoia insofar as you know the variable about which your adversary is hiding information (the quality of the car/the price the stock should be). This makes them too simple to get at the heart of the phenomenon.
I think Habyrka is gesturing at something similar in his paragraph starting “All that said, in reality, navigating a lemon market isn’t too hard.” And I take him to be gesturing at a more central description of paranoia in his subsequent description: “What do you do in a world in which there are not only sketchy used car salesmen, but also sketchy used car inspectors, and sketchy used car inspector rating agencies, or more generally, competent adversaries who will try to predict whatever method you will use to orient to the world, and aim to subvert it for their own aims?”
This is similar to my criticism of maximin as a model of paranoia: “It’s not actually paranoid in a Knightian way, because what if your adversary does something that you didn’t even think of?”
Here’s a gesture at making this more precise: what makes something a central example of paranoia in my mind is when even your knowledge of how your adversary is being adversarial is also something that has been adversarially optimized. Thus chess is not a central example of paranoia (except insofar as your opponent has been spying on your preparations, say) and even markets for lemons aren’t a central example (except insofar as buyers weren’t even tracking that dishonesty was a strategy sellers might use—which is notably a dynamic not captured by the economic model).
I agree with almost all of this, though I think you don’t need to invoke knightian uncertainty. I think it’s simply enough to model there being a very large attack surface combined with a more intelligent adversary.
See this section of my reply to Unnamed about some semi-formal models in the space:
Or maybe not full independence but very strong correlation. The details of this actually matter a lot, and this is where the current magic lives, but we can look at the “guaranteed output” case for now.
One of the problems I’m pointing to is that you don’t know what the attack surface is. This puts you in a pretty different situation than if you have a known large attack surface to defend, even against a smarter adversary (e.g. the whole length of a border; or every possible sequence of Go moves).
Separately, I may be being a bit sloppy by using “Knightian uncertainty” as a broad handle for cases where you have important “unknown unknowns”, aka you don’t even know what ontology to use. But it feels close enough that I’m by default planning to continue describing the research project outlined above as trying to develop a theory of Knightian uncertainty in which Bayesian uncertainty is a special case.
I’ve been thinking about this a lot recently. It seems we could generalize this beyond adversarialness to uncertainty more broadly: In a low-uncertainty environment, consequentialism seems more compelling; in a high-uncertainty environment, deontology makes sense (because as you go from the former to the latter you are making decisions in ways which rest on fewer and fewer error-prone assumptions).
However, this still feels unsatisfying to me for a couple of reasons: (1) In a low-uncertainty environment, there is still some uncertainty. It doesn’t seem to make sense for an actor to behave in violation of their felt sense of morality to achieve a “good” outcome unless they are omniscient and can perfectly predict all indirect effects of their actions.[1] And, if they were truly omniscient, then deontic and consequentialist approaches might converge on similar actions—at least Derek Parfit argues this. I don’t know if I buy this, because (2) why do we value outcomes over the experiences by which we arrive at them? This presupposes consequentialism, which seems increasingly clearly misaligned with human psychology—e.g., the finding that maximizers are unhappier than satisficers, despite achieving “objectively” better outcomes, or the finding that happiness-seeking is associated with reduced happiness.
Relating this back to the question of reasoning in high-adversarial environments, it seems to me that the most prudent (and psychologically protective) approach is a deontological one, not only because it is more robust to outcome-thwarting by adversaries but more importantly because it is (a) positively associated with wellbeing and empathy and (b) inversely associated with power-seeking. See also here.
Moreover, one would need to be omniscient to accurately judge the uncertainty/adversarialness of their environment, so it probably makes sense to assume a high-uncertainty/high-adversarialness environment regardless (at least, if one cares about this sort of thing).