>The fact that maintaining goals across ontology shifts and self-modification takes careful effort is basically the core of the orthodox alignment-looks-hard worldview
It’s one argument. The other is “the totality of human value needs to be hardcoded into the AI, you only get one attempt , and if you make the smallest mistake, everyone dies*.
>You must be just making up an opposing worldview here?
>Where are the orthogonalist circles who say this?
It’s long been the case that OT is mostly used as a pro-doom argument, both in the pro- paperclipping and anti moral realism senses.
It’s also true that the OT has some anti-doom implications, and that’s much less.publicised, and therefore worth pointing out.
The other is “the totality of human value needs to be hardcoded into the AI, you only get one attempt , and if you make the smallest mistake, everyone dies*.
I agree something like this is one branch of the argument, but in my mind it’s a relatively small branch. The main branch focuses on bounded corrigible AI and the main difficulty there is instability. There are other branches for non-hardcoding human values, and different targets that aren’t human values.
It’s long been the case that OT is mostly used as a pro-doom argument, both in the pro- paperclipping and anti moral realism senses.
It’s also true that the OT has some anti-doom implications, and that’s much less.publicised, and therefore worth pointing out.
I’m not sure what point you’re making here, what are the implications you’re referring to?
I’m not sure what point you’re making here, what are the implications you’re referring to?
I thought he meant this part:
But I reject (3), and I am much less willing to grant (2). If value means the production of richer cognition, agency, understanding, beauty, and evaluative structure, it is far from obvious that the current human brain is the only physical substrate capable of steering toward it.
>The fact that maintaining goals across ontology shifts and self-modification takes careful effort is basically the core of the orthodox alignment-looks-hard worldview
It’s one argument. The other is “the totality of human value needs to be hardcoded into the AI, you only get one attempt , and if you make the smallest mistake, everyone dies*.
>You must be just making up an opposing worldview here?
>Where are the orthogonalist circles who say this?
It’s long been the case that OT is mostly used as a pro-doom argument, both in the pro- paperclipping and anti moral realism senses.
It’s also true that the OT has some anti-doom implications, and that’s much less.publicised, and therefore worth pointing out.
I agree something like this is one branch of the argument, but in my mind it’s a relatively small branch. The main branch focuses on bounded corrigible AI and the main difficulty there is instability. There are other branches for non-hardcoding human values, and different targets that aren’t human values.
I’m not sure what point you’re making here, what are the implications you’re referring to?
I thought he meant this part: