I agree with a lot of the counter-arguments others have mentioned.
Summary:
I think the basic argument of the post is well summarized in Figure 1, and by Vanessa Kosoy’s comment.
High-level counter-arguments already argued by Vanessa:
This argument proves too much: it suggests that there are not major differences in ability to do long-term planning that matter.
Humans have not reached the limits of predictive ability
You often only need to be one step ahead of your adversary to defeat them.
Prediction accuracy is not the relevant metric: an incremental increase in depth-of-planning could be decisive in conflicts (e.g. if I can predict the weather one day further ahead, this could have a major impact in military strategy).
More generally, the ability to make large / highly leveraged bets on future outcomes means that slight advantages in prediction ability could be decisive.
Low-level counter-arguments:
(RE Claim 1: Why would AI only have an advantage in IQ as opposed to other forms of intelligence / cognitive skill? No argument is provided.
(Argued by Jonathan Uesato) RE Claim 3: Scaling laws provide ~zero evidence that we are at the limit of “what can be achieved with a certain level of resources”.
RE Claim 5: Systems trained with short-term objectives can learn to do long-term planning competently.
This is a great post. Thanks for writing it!
I agree with a lot of the counter-arguments others have mentioned.
Summary:
I think the basic argument of the post is well summarized in Figure 1, and by Vanessa Kosoy’s comment.
High-level counter-arguments already argued by Vanessa:
This argument proves too much: it suggests that there are not major differences in ability to do long-term planning that matter.
Humans have not reached the limits of predictive ability
You often only need to be one step ahead of your adversary to defeat them.
Prediction accuracy is not the relevant metric: an incremental increase in depth-of-planning could be decisive in conflicts (e.g. if I can predict the weather one day further ahead, this could have a major impact in military strategy).
More generally, the ability to make large / highly leveraged bets on future outcomes means that slight advantages in prediction ability could be decisive.
Low-level counter-arguments:
(RE Claim 1: Why would AI only have an advantage in IQ as opposed to other forms of intelligence / cognitive skill? No argument is provided.
(Argued by Jonathan Uesato) RE Claim 3: Scaling laws provide ~zero evidence that we are at the limit of “what can be achieved with a certain level of resources”.
RE Claim 5: Systems trained with short-term objectives can learn to do long-term planning competently.