Good point; a prior that favors some values over others is going to be a problem, and this is true of the universal prior. The way I’m thinking about it, some of the agents we’re going to be competing with are the malign consequentialists in the universal prior. Of course, figuring out what prior these agents have is going to require more analysis.
Good point; a prior that favors some values over others is going to be a problem, and this is true of the universal prior. The way I’m thinking about it, some of the agents we’re going to be competing with are the malign consequentialists in the universal prior. Of course, figuring out what prior these agents have is going to require more analysis.