The usual argument is that we are likely to be able to build machines that won’t want to modify their goals.
IMO, the more pressing issue with something like OCD is that it might interfere with intelligence tests—in which case you could argue that an OCD superintelligent machine is not really intelligent—since it is using its intelligence to screw itself.
This seems to be a corner case to me. The intended point is more that you could engineer an evil genius, or an autistic mind child.
The usual argument is that we are likely to be able to build machines that won’t want to modify their goals.
IMO, the more pressing issue with something like OCD is that it might interfere with intelligence tests—in which case you could argue that an OCD superintelligent machine is not really intelligent—since it is using its intelligence to screw itself.
This seems to be a corner case to me. The intended point is more that you could engineer an evil genius, or an autistic mind child.