And so I’d say that AIXI-like agents and coalitional agents converge in the limit of optimality, but before that coalitional agency will be a much better framework for understanding realistic agents, including significantly superhuman agents.
So the thing that coalitional agents are robust at is acting approximately like belief/goal agents, and you’re only making a structural claim about agency?
Oh, I see what you mean now. In that case, no, I disagree. Right now this notion of robustness is pre-theoretic. I suspect that we can characterize robustness as “acting like a belief/goal agent” in the limit, but part of my point is that we don’t even know what it means to act “approximately like belief/goal agents” in realistic regimes, because e.g. belief/goal agents as we currently characterize them can’t learn new concepts.
Update: I am increasingly convinced that Bayesianism is not a complete theory of intelligence and may not be the best fundamental basis for agent foundations research, but I am still not convinced that coalitional agency is the right direction.
Mostly talking to you, talking to Abram, and reading Tom Sterkenburg’s thesis.
Briefly: I am now less confident that realizability assumptions are ever satisfied for embedded agents in our universe (Vanessa Kosoy / Diffractor argue this fairly convincingly). In fact this is probably similar to a standard observation about the scientific method (I read Alchin’s “theory of knowledge”, Hutter recommends avoiding editions 3rd and after). As an example intuition, with runtime restrictions it seems to be impossible to construct universal mixtures (Vladimir Vovk impressed this on me). In the unrealizable case, I now appreciate Bayesian learning as one specific expert advice aggregator (albeit an abnormally principled one equipped with now-standard analysis). I appreciate the advantages of other approaches with partial experts, with Garrabrant induction as an extreme case.
I still endorse the Bayesian approach in many cases, in particular when it is at least possible to formulate a reasonable hypothesis class that contains the truth.
What is this coalitional structure for if not to approximate an EU maximizing agent?
This quote from my comment above addresses this:
So the thing that coalitional agents are robust at is acting approximately like belief/goal agents, and you’re only making a structural claim about agency?
If so, I find your model pretty plausible.
Oh, I see what you mean now. In that case, no, I disagree. Right now this notion of robustness is pre-theoretic. I suspect that we can characterize robustness as “acting like a belief/goal agent” in the limit, but part of my point is that we don’t even know what it means to act “approximately like belief/goal agents” in realistic regimes, because e.g. belief/goal agents as we currently characterize them can’t learn new concepts.
Relatedly, see the dialogue in this post.
Update: I am increasingly convinced that Bayesianism is not a complete theory of intelligence and may not be the best fundamental basis for agent foundations research, but I am still not convinced that coalitional agency is the right direction.
Interesting. Got a short summary of what’s changing your mind?
I now have a better understanding of coalitional agency, which I will be interested in your thoughts on when I write it up.
Mostly talking to you, talking to Abram, and reading Tom Sterkenburg’s thesis.
Briefly: I am now less confident that realizability assumptions are ever satisfied for embedded agents in our universe (Vanessa Kosoy / Diffractor argue this fairly convincingly). In fact this is probably similar to a standard observation about the scientific method (I read Alchin’s “theory of knowledge”, Hutter recommends avoiding editions 3rd and after). As an example intuition, with runtime restrictions it seems to be impossible to construct universal mixtures (Vladimir Vovk impressed this on me). In the unrealizable case, I now appreciate Bayesian learning as one specific expert advice aggregator (albeit an abnormally principled one equipped with now-standard analysis). I appreciate the advantages of other approaches with partial experts, with Garrabrant induction as an extreme case.
I still endorse the Bayesian approach in many cases, in particular when it is at least possible to formulate a reasonable hypothesis class that contains the truth.