This seems like a case of generalizing from one example. Chess is a toy scenario. The easiest way to make a chess AI yields an AI that’s good at chess but not much else. And there are any number of toy scenarios we could construct in which backchaining is or isn’t useful. To give an example of a toy scenario where backchaining is clearly useful, consider the case of trying to find the shortest path between two points on a graph. If you do a bidirectional search, searching from both the starting point and the ending point simultaneously, you can cut your algorithm’s exponent in half.
And even for chess, I think you are stating your case too strongly. Suppose I’ve never played chess before and I’m sitting down at a chess board for the first time. My odds of winning will improve if I know what the word “checkmate” means and I have various examples of how checkmate can happen. My odds of winning will further improve if I have tried out various endgame positions and I know whether e.g. it’s possible to win with a king and two knights. Perhaps an experienced player is familiar with chess endgames and knows from memory which piece combinations can win. This might represent a case where the player has reached the point of diminishing returns from backchaining (cf your “you have a reasonable sense of where you’re going” premise), but that doesn’t mean their study of endgames was useless.
For a quick example of a real-world scenario where I suspect backchaining is useful, consider Nick Bostrom’s existential risks paper. I think in general, looking at real world scenarios is a more useful way to address this question. For example, in chess, if all the knights on the board have been captured, the king and two knights scenario is one I can definitively rule out. Knowledge of that sort of endgame is no longer useful. But it’s hard to definitively rule out any of the risks on Bostrom’s list.
A fairly similar statement holds for practically all game playing algorithms; in this respect, chess differs little from go, even though the algorithms used to solve each tend to be quite different. However, the story changes when we move to AI planning algorithms more generally; backchaining is common for planning.
I think studying AI algorithms for these sorts of things is, generally, quite informative with respect to what types of reasoning you can expect to work well. Especially if you then practice the skill yourself and watch for what kinds of reasoning you’re doing, and when they’re effective.
There are many other situations where this sort of thing applies, some with quite more serious consequences than a game of chess.
For instance, during World War II the Imperial Japanese Navy had a strategy called Kantai Kessen (“naval fleet decisive battle”, often referred to simply as the Decisive Battle Doctrine), which was essentially a big backchain of this sort.
Reasoning that a naval war between Japan and the United States would culminate in a decisive battle between the fleets and that winning this battle would win the war (as it had for the Japanese with the Battle of Tsushima against the Russians in 1905), Japanese strategists designed a war plan that focused heavily on putting themselves into a strong position to initate such a decisive battle, chaining back from this all the way to the level of what types of ships to build.
However, this reasoning backfired. The Japanese fixation on concentrating forces for a major battle lead them to ignore elements of the war that could have given them an advantage. For instance, Japan never had a serious anti-commerce raiding strategy on either offense or defense; their submarines were focused on whittling down the enemy fleet in preparation for a final battle and they neglected attacks on US shipping and inadequately defended their own shipping from similar methods.
By contrast, while the United States had begun the war with similar “decisive battle” ideas (these were quite in vogue thanks to Mahan’s influence), they were ironically forced to come up with a new strategy following heavy losses at Pearl Harbor. Their “island hopping” strategy focused on building incremental advantages and didn’t rely on staging a specific battle until circumstances presented that as the best option—and indeed proved far more effective.
Now, there are of course other factors at work here—the US had very relevant industrial and commerce advantages, for instance—but this does seem a non-toy example where focusing on chaining backwards from a desired end point too far in the future led to serious strategic errors.
Hm, I can’t say I find this example very convincing either. In Bostrom’s paper, he identifies many different ways in which the human species could go extinct. If the Japanese thought the same way Bostrom did, they would have brainstormed many different scenarios under which they could lose the war. Their failure to do so represents a lack of lateral thinking, which seems orthogonal to the forward chain vs backward chain thing. Lack of lateral thinking can come up during forward chaining too, if you don’t fully explore your options (e.g. spending all of your time thinking about air power and none of your time thinking about sea power).
Anyway, I suspect a balance of both forward and back chaining is best. Backchaining is good for understanding which factors are actually important. Sometimes it’s not the ones you think would give you “generalized advantage”. For example, during the Vietnam War, the Tet Offensive was a military loss for the North, so a naive notion of “generalized advantage” might have indicated it was a bad idea. But it ended up being what allowed them to win the war in the long run due to its psychological effect on the American public. If the US military had backchained and tried to brainstorm all of the scenarios under which the South could lose the war (“murphyjitsu”), they might have realized at a certain point that demoralization of the American public was one of the few remaining ways for them to lose. Further backchaining, through thinking like the enemy and trying to generate maximally demoralizing attack scenarios, might have suggested the idea of a surprise attack during the Lunar New Year truce period.
I’d expect our intuitions about “generalized advantage” to be least reliable in domains where we have little experience, such as future technologies that haven’t been developed yet. But I think backchaining can be useful in other scenarios as well—e.g. if my goal is to be President, I could look at the resume of every President at the time they were elected, and try to figure out what elements they had in common and how they were positioned right before the start of their successful run.
This seems like a case of generalizing from one example. Chess is a toy scenario. The easiest way to make a chess AI yields an AI that’s good at chess but not much else. And there are any number of toy scenarios we could construct in which backchaining is or isn’t useful. To give an example of a toy scenario where backchaining is clearly useful, consider the case of trying to find the shortest path between two points on a graph. If you do a bidirectional search, searching from both the starting point and the ending point simultaneously, you can cut your algorithm’s exponent in half.
And even for chess, I think you are stating your case too strongly. Suppose I’ve never played chess before and I’m sitting down at a chess board for the first time. My odds of winning will improve if I know what the word “checkmate” means and I have various examples of how checkmate can happen. My odds of winning will further improve if I have tried out various endgame positions and I know whether e.g. it’s possible to win with a king and two knights. Perhaps an experienced player is familiar with chess endgames and knows from memory which piece combinations can win. This might represent a case where the player has reached the point of diminishing returns from backchaining (cf your “you have a reasonable sense of where you’re going” premise), but that doesn’t mean their study of endgames was useless.
For a quick example of a real-world scenario where I suspect backchaining is useful, consider Nick Bostrom’s existential risks paper. I think in general, looking at real world scenarios is a more useful way to address this question. For example, in chess, if all the knights on the board have been captured, the king and two knights scenario is one I can definitively rule out. Knowledge of that sort of endgame is no longer useful. But it’s hard to definitively rule out any of the risks on Bostrom’s list.
A fairly similar statement holds for practically all game playing algorithms; in this respect, chess differs little from go, even though the algorithms used to solve each tend to be quite different. However, the story changes when we move to AI planning algorithms more generally; backchaining is common for planning.
I think studying AI algorithms for these sorts of things is, generally, quite informative with respect to what types of reasoning you can expect to work well. Especially if you then practice the skill yourself and watch for what kinds of reasoning you’re doing, and when they’re effective.
Interesting suggestion! Is there a starting point you would recommend for this sort of study?
There are many other situations where this sort of thing applies, some with quite more serious consequences than a game of chess.
For instance, during World War II the Imperial Japanese Navy had a strategy called Kantai Kessen (“naval fleet decisive battle”, often referred to simply as the Decisive Battle Doctrine), which was essentially a big backchain of this sort.
Reasoning that a naval war between Japan and the United States would culminate in a decisive battle between the fleets and that winning this battle would win the war (as it had for the Japanese with the Battle of Tsushima against the Russians in 1905), Japanese strategists designed a war plan that focused heavily on putting themselves into a strong position to initate such a decisive battle, chaining back from this all the way to the level of what types of ships to build.
However, this reasoning backfired. The Japanese fixation on concentrating forces for a major battle lead them to ignore elements of the war that could have given them an advantage. For instance, Japan never had a serious anti-commerce raiding strategy on either offense or defense; their submarines were focused on whittling down the enemy fleet in preparation for a final battle and they neglected attacks on US shipping and inadequately defended their own shipping from similar methods.
By contrast, while the United States had begun the war with similar “decisive battle” ideas (these were quite in vogue thanks to Mahan’s influence), they were ironically forced to come up with a new strategy following heavy losses at Pearl Harbor. Their “island hopping” strategy focused on building incremental advantages and didn’t rely on staging a specific battle until circumstances presented that as the best option—and indeed proved far more effective.
Now, there are of course other factors at work here—the US had very relevant industrial and commerce advantages, for instance—but this does seem a non-toy example where focusing on chaining backwards from a desired end point too far in the future led to serious strategic errors.
Hm, I can’t say I find this example very convincing either. In Bostrom’s paper, he identifies many different ways in which the human species could go extinct. If the Japanese thought the same way Bostrom did, they would have brainstormed many different scenarios under which they could lose the war. Their failure to do so represents a lack of lateral thinking, which seems orthogonal to the forward chain vs backward chain thing. Lack of lateral thinking can come up during forward chaining too, if you don’t fully explore your options (e.g. spending all of your time thinking about air power and none of your time thinking about sea power).
Anyway, I suspect a balance of both forward and back chaining is best. Backchaining is good for understanding which factors are actually important. Sometimes it’s not the ones you think would give you “generalized advantage”. For example, during the Vietnam War, the Tet Offensive was a military loss for the North, so a naive notion of “generalized advantage” might have indicated it was a bad idea. But it ended up being what allowed them to win the war in the long run due to its psychological effect on the American public. If the US military had backchained and tried to brainstorm all of the scenarios under which the South could lose the war (“murphyjitsu”), they might have realized at a certain point that demoralization of the American public was one of the few remaining ways for them to lose. Further backchaining, through thinking like the enemy and trying to generate maximally demoralizing attack scenarios, might have suggested the idea of a surprise attack during the Lunar New Year truce period.
I’d expect our intuitions about “generalized advantage” to be least reliable in domains where we have little experience, such as future technologies that haven’t been developed yet. But I think backchaining can be useful in other scenarios as well—e.g. if my goal is to be President, I could look at the resume of every President at the time they were elected, and try to figure out what elements they had in common and how they were positioned right before the start of their successful run.