In the default outcome, astronomical amounts of subroutines will be spun up in pursuit of higher-level goals, whether those goals are aligned with the complexity of human value or aligned with paperclips. Without firm protections in place, these subroutines might experience some notion of suffering
Surely, an human goal aligned ASI wouldn’t want to make suffering subroutines.
For paperclip maximizers, there are 2 options, either suffering based algorithms are the most effective way of achieving important real world tasks, or they aren’t. In the latter case, no problem, the paperclip maximizer won’t use them. (Well you still have a big problem, namely the paperclip maximizer)
In the former case, you would need to design a system that intrinsically wanted not to make suffering subroutines, and had that goal stable under self improvement. The level of competence and understanding needed to do this is higher than the amount needed to realize you are making a paperclip maximizer, and not turn it on.
Surely, an human goal aligned ASI wouldn’t want to make suffering subroutines.
For paperclip maximizers, there are 2 options, either suffering based algorithms are the most effective way of achieving important real world tasks, or they aren’t. In the latter case, no problem, the paperclip maximizer won’t use them. (Well you still have a big problem, namely the paperclip maximizer)
In the former case, you would need to design a system that intrinsically wanted not to make suffering subroutines, and had that goal stable under self improvement. The level of competence and understanding needed to do this is higher than the amount needed to realize you are making a paperclip maximizer, and not turn it on.