Yeah, this might be getting at a similar sort of difficulty as we have when looking for a universal/natural/objective criterion to compare decision theories. (I remember a post on this topic by Caspar Oesterheld, but can’t find it now.)
My operationalization would be something along the lines of: For all the worlds with “a free market containing lots of different smart AI agents making decisions in lots of different ways”, what is the total probability measure of those in which the most powerful agents are the ones satisfying the independence axiom?
Or: across all such worlds, what is the total measure of independence-obeying agents relative to the total measure of “most powerful agents”?
(Well, I guess this is just binning/discretizing power, so as to avoid blowing up its expected value.)
Yeah, this might be getting at a similar sort of difficulty as we have when looking for a universal/natural/objective criterion to compare decision theories. (I remember a post on this topic by Caspar Oesterheld, but can’t find it now.)
My operationalization would be something along the lines of: For all the worlds with “a free market containing lots of different smart AI agents making decisions in lots of different ways”, what is the total probability measure of those in which the most powerful agents are the ones satisfying the independence axiom?
Or: across all such worlds, what is the total measure of independence-obeying agents relative to the total measure of “most powerful agents”?
(Well, I guess this is just binning/discretizing power, so as to avoid blowing up its expected value.)
“The lack of performance metrics for CDT versus EDT, etc.” (An important post! FWIW I discuss some related implications for forecasting AIs’ decision theories here.)