Their decision theory stuff is perhaps relevant because of how the difficulties encountered seem to be central to alignment: for example, logical uncertainties and logical counterfactuals. Embedded Agency argues that decision theory is merely one of multiple entangled problems that come from trying to have embedded agents.
Encountering the same difficulties provides further evidence that those difficulties are important; especially if the thing is “entangled with” the same core issue. Thus working out some of the theory beforehand helps you figure out what paths to pursue later.
I don’t know if quantilizers will be important. The idea is that you have the AI sample randomly from what the top x% of humans (or whatever “safe” distribution you want) would do, and you can then give some guarantees that essentially the AI will at worst be 1/x (e.g. 10 times) as bad as a human.
I’m skeptical about usefulness because I expect that to be useful you’ll need to push to a high enough part of the human distribution that your safety guarantee is practically useless.
Their decision theory stuff is perhaps relevant because of how the difficulties encountered seem to be central to alignment: for example, logical uncertainties and logical counterfactuals. Embedded Agency argues that decision theory is merely one of multiple entangled problems that come from trying to have embedded agents.
Encountering the same difficulties provides further evidence that those difficulties are important; especially if the thing is “entangled with” the same core issue. Thus working out some of the theory beforehand helps you figure out what paths to pursue later.
I don’t know if quantilizers will be important. The idea is that you have the AI sample randomly from what the top x% of humans (or whatever “safe” distribution you want) would do, and you can then give some guarantees that essentially the AI will at worst be 1/x (e.g. 10 times) as bad as a human.
I’m skeptical about usefulness because I expect that to be useful you’ll need to push to a high enough part of the human distribution that your safety guarantee is practically useless.