This seems like it’s building on or inspired by work you’ve done? Or was this team interested in embeddedness and reflective oracles for other reasons?
It’s ridiculously long (which is great, I’ll read through it when I get a chance), do you have any pointers to sections that you think have particularly valuable insights?
I believe they were mainly inspired by Demski and Garrabrant, but we were in contact for the last few months and I’m glad to see that some of my recent work was applicable. We arrived at the idea of using a joint distribution with a grain of truth independently, and they introduce a novel “RUI” construction, but also study (what I’ve been calling) AEDT wrt rOSI in section 5.2. The differences are pretty technical, IMO the RUI approach is halfway between rOSI and logical induction.
It’s so long that even I’m still reading it, and I got a copy early. Assuming you’re familiar with Solomonoff induction / AIXI / embedded agency (which it sounds like you are) the core of it is section 3 and section 5 (particularly 5.1-5.3 I think). The appendix is like 100 pages and so far doesn’t seem essential unless you want to extend the results (also some of it will be familiar if you read my GOT paper).
Author here. We were heavily inspired by multiple things, including Demski and Garrabrant, the 1990′s work of Kalai and Lehrer, empirical work in our group inspired by neuroscience pointing towards systems that predict their own actions, and the earlier work on reflective oracles by Leike . We were not aware of @Cole Wyeth et al.’s excellent 2025 paper which puts the reflective oracle work on firmer theoretical footing, as our work was (largely but not entirely) done before this paper appeared.
Hey Jeremy! Our team’s interest is mainly on multi-agent learning, and self-modeling and theory of mind. As properly formalizing a coherent theory for these topics turned out to be quite difficult, we dived deeper and deeper and ultimately arrived at the AIXI and reflective oracles frameworks, which provided a good set of tools as starting points for addressing these questions more formally. The resulting ‘monster paper’ is a write-up of the past year of work we did on these topics. Due to our interest in multi-agent learning, a good chunk of the paper is on the game-theoretic behavior of such ‘embedded Bayesian agents’ (Section 4). As Cole mentioned, we arrived independently to some similar results as Cole’s (as we came from a bit outside of the less wrong community), and we are very excited to now start collaborating more closely with Cole on the next questions enabled by both of our theories!
This seems like it’s building on or inspired by work you’ve done? Or was this team interested in embeddedness and reflective oracles for other reasons?
It’s ridiculously long (which is great, I’ll read through it when I get a chance), do you have any pointers to sections that you think have particularly valuable insights?
I believe they were mainly inspired by Demski and Garrabrant, but we were in contact for the last few months and I’m glad to see that some of my recent work was applicable. We arrived at the idea of using a joint distribution with a grain of truth independently, and they introduce a novel “RUI” construction, but also study (what I’ve been calling) AEDT wrt rOSI in section 5.2. The differences are pretty technical, IMO the RUI approach is halfway between rOSI and logical induction.
It’s so long that even I’m still reading it, and I got a copy early. Assuming you’re familiar with Solomonoff induction / AIXI / embedded agency (which it sounds like you are) the core of it is section 3 and section 5 (particularly 5.1-5.3 I think). The appendix is like 100 pages and so far doesn’t seem essential unless you want to extend the results (also some of it will be familiar if you read my GOT paper).
Author here. We were heavily inspired by multiple things, including Demski and Garrabrant, the 1990′s work of Kalai and Lehrer, empirical work in our group inspired by neuroscience pointing towards systems that predict their own actions, and the earlier work on reflective oracles by Leike . We were not aware of @Cole Wyeth et al.’s excellent 2025 paper which puts the reflective oracle work on firmer theoretical footing, as our work was (largely but not entirely) done before this paper appeared.
Hey Jeremy! Our team’s interest is mainly on multi-agent learning, and self-modeling and theory of mind. As properly formalizing a coherent theory for these topics turned out to be quite difficult, we dived deeper and deeper and ultimately arrived at the AIXI and reflective oracles frameworks, which provided a good set of tools as starting points for addressing these questions more formally. The resulting ‘monster paper’ is a write-up of the past year of work we did on these topics. Due to our interest in multi-agent learning, a good chunk of the paper is on the game-theoretic behavior of such ‘embedded Bayesian agents’ (Section 4). As Cole mentioned, we arrived independently to some similar results as Cole’s (as we came from a bit outside of the less wrong community), and we are very excited to now start collaborating more closely with Cole on the next questions enabled by both of our theories!