Jessica Taylor. CS undergrad and Master’s at Stanford; former research fellow at MIRI.
I work on decision theory, social epistemology, strategy, naturalized agency, mathematical foundations, decentralized networking systems and applications, theory of mind, and functional programming languages.
Blog: unstableontology.com
Twitter: https://twitter.com/jessi_cata
This is nice! I actually don’t remember exactly what formulation of this I discussed with Paul at the workshop, but I really like this version. I do think that it may be useful (at least from a pedagogical perspective) to look at what happens in the finite case, where we have a finite set of halting stochastic programs that may query the oracle to ask about the output probabilities of other programs in this set. This restricted version is powerful enough to express the liar’s paradox and some multi-agent problems.
In this case, as long as we are assured that each program always halts, I believe the problem is computable. We can enumerate all possible calls that any program might give to the oracle. Then, we can write the desired 1-returning probabilities for each call as a set-valued function of the oracle’s 1-returning probabilities for each call. This function will be in closed form using only addition, subtraction, multiplication, and a set-valued version of the unit step function. So, it will be possible to find the fixed point of this function using a combination of brute-force search and polynomial solving, similar to how it’s possible to find Nash equilibria.