I’m a mathematician passionate about AI Safety. Currently, I’m doing agent foundations research with Dovetail. Check out my research on this account!
Winter Cross
Karma: 73
The Probability of an Event under a Simplicity Prior
When is Unlimited Optimization Catastrophic?
Not only does this serve as a powerful tool for incentivizing learning — by paying students to do work, we unlock a powerful tool for speeding up education: asynchronous learning.
I don’t really see how paying students can allow for asynchronous learning. It seems to me that would be more related to changes in how daily agendas are structured and how grades are determined rather than the incentive structure
Consider a robot that is manipulating a deck of cards. This can be described with a world model with
Should this be 52! instead of 13! since there are 52 cards in a deck?
Robust Finite Policies are Nontrivially Structured
Thanks for the answer! That confirms what I was thinking.
That second case: and surprised me since I initially thought that the IMP implied that there was an additional isomorphism between the controller and the environment. I guess that isomorphism effectively still exists since it can be created through the use of coarse graining over the controller states like you mentioned.
This article is really approachable for someone like me who’s just getting acquainted with mathematical AI safety research, so I appreciate that! This definitely helped me better understand the IMP.
I have a question about this part in the “How is the controller ‘modelling’ the environment?” section:
If the joint system is represented by environment-controller pairs , then being injective means that no two pairs (within ) will have the same environment value or controller value . This means that with appropriate re-labelling, each joint state can be indexed:
I don’t see how this follows. Couldn’t the shape of the joint states be more complicated such as forming a cycle or having multiple joint states evolve to the same joint state? Both these possibilities would break the indexing. Is there something I’m missing that implies this linear structure?
Thanks for commenting!
One thing to note is that this result does not say anything about the probability of catastrophe. To do that, you’d need to define a prior over proxies resulting from the training process (which we don’t do in this paper/post). Depending on this prior, catastrophe could be arbitrarily likely.
Because of this, I’m not comfortable with the continuous framework inequality alone as an endpoint; I’d also need an understanding of the priors of our learning algorithms or alternate incentive structures that require agents to adjust their values based on feedback from humanity