I feel like once we basically understand how the human predictive algorithm works, it may not be possible to improve on that algorithm (without massive and time-costly experimentation) no matter what the level of intelligence of the entity trying to improve on it. (The reason I gave: The human one has been developed by trial-and-error over millions of years in the real world, a method that won’t be available to the GMAGI. So there’s no guarantee that a greater intelligence could find a way to improve this algorithm without such extended trial-and-error)
...is probably not right. Nobody really knows how tough this problem will prove to be once we stop being able to crib from the human solution—and it is possible that progress will get tougher. However, much of the progress on the problem has not been obviously based on reverse-engineering the human prediction algorithm in the first place. Also machine prediction capabilities already far exceed human ones in some domains—e.g. chess, the weather.
Anyway, this problem makes little difference either way. Machines don’t have the human pelvis to contend with, and won’t be limited to running at 200 Hz.
This bit (from Karnofsky):
...is probably not right. Nobody really knows how tough this problem will prove to be once we stop being able to crib from the human solution—and it is possible that progress will get tougher. However, much of the progress on the problem has not been obviously based on reverse-engineering the human prediction algorithm in the first place. Also machine prediction capabilities already far exceed human ones in some domains—e.g. chess, the weather.
Anyway, this problem makes little difference either way. Machines don’t have the human pelvis to contend with, and won’t be limited to running at 200 Hz.