We discussed this post in the AISafety.com Reading Group, and have a few questions about it and infra-bayesianism:
The image on top of the sequence on Infra-Bayesianism shows a tree, which we interpret as a game-tree, with Murphy and an agent alternating in taking actions. Can we say anything about such a tree? E.g. Complexity, Pruning, etc?
There was some discussion about if an infra-bayesian agent could be Dutch-booked. Is this possible?
Your introduction makes no attempt to explain “convexity”, which seems like a central part of Infra-Bayesianism. If it is central, what would be a good one-paragraph summary?
No idea. I don’t think it’s computationally very tractable. If I understand correctly, l Vanessa hopes there will be computationally feasible approximations, but there wasn’t much research into computational complexity yet, because there are more basic unsolved questions.
I’m pretty sure that no. An IB agent (with enough compute) plans for the long run and doesn’t go into a chain of deals that leaves it worse of than not doing anything. In general, IB solves the “not exactly Bayesian expected utility maximizer but still can’t be Dutch booked problem” by potentially refusing to take either side of a bet: if it has Knightian uncertainty about whether a probability is lower or higher than 50%, it will refuse to bet at even odds either for or against. This is something that humans actually often do, and I agree with Vanessa that a decision theory can be allowed to do that.
I had a paragraph about it: ”Here is where convex sets come in: The law constrains Murphy to choose the probability distribution of outcomes from a certain set in the space of probability distributions. Whatever the loss function is, the worst probability distribution Murphy can choose from the set is the same as if he could choose from the convex hull of the set. So we might as well start by saying that the law must be constraining Murphy to a convex set of probability distributions.” As far as I can tell, this is the reason behind considering convex sets. This makes convexity pretty central: laws are very central, and now we are assuming that every law is a convex set in the space of probability distributions.
Vanessa said that her guess is yes. In the terms of the linked Arbital article, IB is intended to be an example of “There could be some superior alternative to probability theory and decision theory that is Bayesian-incoherent”. Personally, I don’t know, I think that the article’s “A cognitively powerful agent might not be sufficiently optimized” possibility feels more likely in the current paradigm, I can absolutely imagine the first AIs to become a world-ending threat not being very coherent. Also, IB is just an ideal, real-world smart agents will be at best approximations of infra-Bayesian agents (same holds for Bayesianism). Vanessa’s guess is that understanding IB better will still give us useful insights into these real-world models if we view them as IB approximations, I’m pretty doubtful, but maybe. Also, I feel that the problem I write about in my post on the monotonicity principle points at some deeper problem in IB which makes me doubtful whether sufficiently optimized agents will actually use (approximations of) the minimax thinking prescribed by IB.
Regarding 4: given that infra-Bayesianism is maximally paranoid, shouldn’t it have lower performance relative to decision-making theories like regular Bayes under many non-adversarial conditions? If the training set does not contain many instances of adversarial information, then shouldn’t we expect agents to adopt Bayes instead of infra-Bayes?
I think Vanessa would argue that “Bayesianism” is not really an option. The non-realizability problem in Bayesianism is not just some weird special case, but the normal state of things: Bayesianism assumes that we have hypotheses fully describing the world, which we very definitely don’t have in real life. IB tries to be less demanding, and the laws in the agent’s hypothesis class don’t necessarily need to be that detailed. I am relatively skeptical of this, and I believe that for an IB agent to work well, the laws in its hypothesis class probably also need to be unfeasibly detailed. So both “adopting Bayes” and “adopting infra-Bayes” fully is impossible. We probably won’t have such a nice mathematical model for the messy decision process a superintelligence actually adopts, the question is whether thinking about it as an approximation of Bayes or infra-Bayes gives us a more clear picture. It’s a hard question, and IB has an advantage in that the laws need to be less detailed, and a disadvantage that I think you are right about it being unnecessarily paranoid. My personal guess is that nothing besides the basic insight of Bayesianism (“the agent seems to update on evidence, sort of following Bayes-rule”) will be actually useful in understanding the way an AI will think.
We discussed this post in the AISafety.com Reading Group, and have a few questions about it and infra-bayesianism:
The image on top of the sequence on Infra-Bayesianism shows a tree, which we interpret as a game-tree, with Murphy and an agent alternating in taking actions. Can we say anything about such a tree? E.g. Complexity, Pruning, etc?
There was some discussion about if an infra-bayesian agent could be Dutch-booked. Is this possible?
Your introduction makes no attempt to explain “convexity”, which seems like a central part of Infra-Bayesianism. If it is central, what would be a good one-paragraph summary?
Will any sufficiently smart agent be infra-bayesian? To be precise, can you replace “Bayesian” with “Infra-Bayesian” in this article: https://arbital.com/p/optimized_agent_appears_coherent/ ?
No idea. I don’t think it’s computationally very tractable. If I understand correctly, l Vanessa hopes there will be computationally feasible approximations, but there wasn’t much research into computational complexity yet, because there are more basic unsolved questions.
I’m pretty sure that no. An IB agent (with enough compute) plans for the long run and doesn’t go into a chain of deals that leaves it worse of than not doing anything. In general, IB solves the “not exactly Bayesian expected utility maximizer but still can’t be Dutch booked problem” by potentially refusing to take either side of a bet: if it has Knightian uncertainty about whether a probability is lower or higher than 50%, it will refuse to bet at even odds either for or against. This is something that humans actually often do, and I agree with Vanessa that a decision theory can be allowed to do that.
I had a paragraph about it:
”Here is where convex sets come in: The law constrains Murphy to choose the probability distribution of outcomes from a certain set in the space of probability distributions. Whatever the loss function is, the worst probability distribution Murphy can choose from the set is the same as if he could choose from the convex hull of the set. So we might as well start by saying that the law must be constraining Murphy to a convex set of probability distributions.”
As far as I can tell, this is the reason behind considering convex sets. This makes convexity pretty central: laws are very central, and now we are assuming that every law is a convex set in the space of probability distributions.
Vanessa said that her guess is yes. In the terms of the linked Arbital article, IB is intended to be an example of “There could be some superior alternative to probability theory and decision theory that is Bayesian-incoherent”. Personally, I don’t know, I think that the article’s “A cognitively powerful agent might not be sufficiently optimized” possibility feels more likely in the current paradigm, I can absolutely imagine the first AIs to become a world-ending threat not being very coherent. Also, IB is just an ideal, real-world smart agents will be at best approximations of infra-Bayesian agents (same holds for Bayesianism). Vanessa’s guess is that understanding IB better will still give us useful insights into these real-world models if we view them as IB approximations, I’m pretty doubtful, but maybe. Also, I feel that the problem I write about in my post on the monotonicity principle points at some deeper problem in IB which makes me doubtful whether sufficiently optimized agents will actually use (approximations of) the minimax thinking prescribed by IB.
Regarding 4: given that infra-Bayesianism is maximally paranoid, shouldn’t it have lower performance relative to decision-making theories like regular Bayes under many non-adversarial conditions? If the training set does not contain many instances of adversarial information, then shouldn’t we expect agents to adopt Bayes instead of infra-Bayes?
I think Vanessa would argue that “Bayesianism” is not really an option. The non-realizability problem in Bayesianism is not just some weird special case, but the normal state of things: Bayesianism assumes that we have hypotheses fully describing the world, which we very definitely don’t have in real life. IB tries to be less demanding, and the laws in the agent’s hypothesis class don’t necessarily need to be that detailed. I am relatively skeptical of this, and I believe that for an IB agent to work well, the laws in its hypothesis class probably also need to be unfeasibly detailed. So both “adopting Bayes” and “adopting infra-Bayes” fully is impossible. We probably won’t have such a nice mathematical model for the messy decision process a superintelligence actually adopts, the question is whether thinking about it as an approximation of Bayes or infra-Bayes gives us a more clear picture. It’s a hard question, and IB has an advantage in that the laws need to be less detailed, and a disadvantage that I think you are right about it being unnecessarily paranoid. My personal guess is that nothing besides the basic insight of Bayesianism (“the agent seems to update on evidence, sort of following Bayes-rule”) will be actually useful in understanding the way an AI will think.