The other is “the totality of human value needs to be hardcoded into the AI, you only get one attempt , and if you make the smallest mistake, everyone dies*.
I agree something like this is one branch of the argument, but in my mind it’s a relatively small branch. The main branch focuses on bounded corrigible AI and the main difficulty there is instability. There are other branches for non-hardcoding human values, and different targets that aren’t human values.
It’s long been the case that OT is mostly used as a pro-doom argument, both in the pro- paperclipping and anti moral realism senses.
It’s also true that the OT has some anti-doom implications, and that’s much less.publicised, and therefore worth pointing out.
I’m not sure what point you’re making here, what are the implications you’re referring to?
I’m not sure what point you’re making here, what are the implications you’re referring to?
I thought he meant this part:
But I reject (3), and I am much less willing to grant (2). If value means the production of richer cognition, agency, understanding, beauty, and evaluative structure, it is far from obvious that the current human brain is the only physical substrate capable of steering toward it.
I agree something like this is one branch of the argument, but in my mind it’s a relatively small branch. The main branch focuses on bounded corrigible AI and the main difficulty there is instability. There are other branches for non-hardcoding human values, and different targets that aren’t human values.
I’m not sure what point you’re making here, what are the implications you’re referring to?
I thought he meant this part: