Undecidability is definitely a related but distinct problem. I felt that there were some objections to undecidability arguments based on bounded rationality solutions, so I decided to focus on the question of uncertainty about the inputs rather than uncertainty about halting (Godel assumed that we knew the inputs and functions perfectly), which I think applies to both unbounded and bounded rationality. You could definitely make an argument to that effect that the open ended nature of super intelligence specifically opens it up to undecidability problems for ranking it’s preferences though.
I haven’t seen those things you cited at the end but will check them out, thanks.
Undecidability is definitely a related but distinct problem. I felt that there were some objections to undecidability arguments based on bounded rationality solutions, so I decided to focus on the question of uncertainty about the inputs rather than uncertainty about halting (Godel assumed that we knew the inputs and functions perfectly), which I think applies to both unbounded and bounded rationality. You could definitely make an argument to that effect that the open ended nature of super intelligence specifically opens it up to undecidability problems for ranking it’s preferences though.
I haven’t seen those things you cited at the end but will check them out, thanks.