I am worried that properties P1...Pk are somehow valuable.
In what sense can they be valuable, if they are not valued by human judgment criteria (even if not consciously most of the time)?
For example, if the genie could predict the results of any action it took, you could simply ask it to select any path that results in you saying “thanks genie, great job” without coercion.
Formalizing “coercion” is itself an exhaustive problem. Saying “don’t manipulate my brain except through my senses” is a big first step, but it doesn’t exclude, e.g., powerful arguments that you don’t really want your mother to live.
I am worried that properties P1...Pk are somehow valuable.
In what sense can they be valuable, if they are not valued by human judgment criteria (even if not consciously most of the time)?
For example, if the genie could predict the results of any action it took, you could simply ask it to select any path that results in you saying “thanks genie, great job” without coercion.
Formalizing “coercion” is itself an exhaustive problem. Saying “don’t manipulate my brain except through my senses” is a big first step, but it doesn’t exclude, e.g., powerful arguments that you don’t really want your mother to live.