Luckily I don’t need to show that sufficiently smart AIs don’t engage in trial and error. All I need to show is that they almost certainly do not engage in the particular kind of trial of running a computer program without already knowing whether the program is satisfactory.
you have thereby defined sufficiently smart as AIs that satisfy this requirement. this is not the case. many likely designs for AIs well above human level will have need to actually run parts of programs to get their results. perhaps usually fairly small ones.
Luckily I don’t need to show that sufficiently smart AIs don’t engage in trial and error. All I need to show is that they almost certainly do not engage in the particular kind of trial of running a computer program without already knowing whether the program is satisfactory.
you have thereby defined sufficiently smart as AIs that satisfy this requirement. this is not the case. many likely designs for AIs well above human level will have need to actually run parts of programs to get their results. perhaps usually fairly small ones.