This strikes me as a distribution shift relative to playing actual games. Often the issue is that you don’t consider the correct candidate move, and by the time you’ve narrowed the position down to two candidate moves you’ve already made a bunch of progress. Also, the two moves you’re shown in the app are often not the top two moves you’d consider as a human[1].
ChessTempo has a version of this where you play through master games and guess the move (without any prompting), and you gain points based on your score relative to the engine/master move. I think it still has some issues but seems better than this app to me.
Edit:
Also, it seems your screenshot might have a bug? Neither Bxh7 nor Ne5 are good moves in this position.
I agree with this. I’m tempted to add a mode where you just have to pick Stockfish’s top move and then your score is how much worse your move is than Stockfish’s (Stockfish’s move = perfect score). I focused the initial version on the easiest problems though, since my theory is that the newest players are the ones who will benefit the most from this kind of training anyway.
Would it be possible to use something like Maia, instead of Stockfish? Then you could more plausibly get ‘higher human-strength moves’, where you are not left trying to followup impossibly sharp tactical play you can’t pull off without a chess engine, or other issues with doing direct naive behavior cloning from superhuman chess engines.
I looked into this, and I think it wouldn’t work for what I’m trying to do.
On my item bank (filtered for bad moves that humans thought were plausible), Maia basically agrees with Stockfish around half of the time, but also makes a move as bad or worse than the human around a quarter of the time. There are a small number of cases where Maia picks a good move and thinks Stockfish’s move is completely implausible (1.5%), but either the moves are so close that we’d basically never serve it, or the bad move is obvious even if Stockfish’s move is weird.
Since I want to guarantee that the moves we show people are actually good, I approach this from the other direction and use how hard it is to see Stockfish’s move as a difficulty filter. At low ratings, we show you moves where there’s a large win swing that’s visible with a shallow search, and as you get better we slowly show moves with a smaller win swing or that require a deeper search to see.
Here’s the ratings bands and gaps:
rating band
median win rate gap
0–400
0.441
1200–1600
0.315
2000–2400
0.185
2400–2800
0.065
2800–3200
0.081
So, it’s possible for us to show moves that are very hard to distinguish, but only if you’re insanely good at this task. This item from before would require you to be in our 2400+ band.
This strikes me as a distribution shift relative to playing actual games. Often the issue is that you don’t consider the correct candidate move, and by the time you’ve narrowed the position down to two candidate moves you’ve already made a bunch of progress. Also, the two moves you’re shown in the app are often not the top two moves you’d consider as a human[1].
ChessTempo has a version of this where you play through master games and guess the move (without any prompting), and you gain points based on your score relative to the engine/master move. I think it still has some issues but seems better than this app to me.
Edit:
Also, it seems your screenshot might have a bug? Neither Bxh7 nor Ne5 are good moves in this position.
E.g. in your screenshot, I would consider Ne2 before looking at Bxh7+ or Ne5.
Thanks! I meant for this to be a random move but Claude decided to generate the board by hand and I didn’t catch it.
I updated the preview image to this position (chosen since it’s pretty easy to follow why it’s bad, but the move was played by an Elo 2300 player in a real game): https://chess-pretraining.brendanlong.com/?item=31395
I agree with this. I’m tempted to add a mode where you just have to pick Stockfish’s top move and then your score is how much worse your move is than Stockfish’s (Stockfish’s move = perfect score). I focused the initial version on the easiest problems though, since my theory is that the newest players are the ones who will benefit the most from this kind of training anyway.
Would it be possible to use something like Maia, instead of Stockfish? Then you could more plausibly get ‘higher human-strength moves’, where you are not left trying to followup impossibly sharp tactical play you can’t pull off without a chess engine, or other issues with doing direct naive behavior cloning from superhuman chess engines.
I looked into this, and I think it wouldn’t work for what I’m trying to do.
On my item bank (filtered for bad moves that humans thought were plausible), Maia basically agrees with Stockfish around half of the time, but also makes a move as bad or worse than the human around a quarter of the time. There are a small number of cases where Maia picks a good move and thinks Stockfish’s move is completely implausible (1.5%), but either the moves are so close that we’d basically never serve it, or the bad move is obvious even if Stockfish’s move is weird.
For examples of the last category, Maia plays queen to g7 here, and rook to e8 here.
Since I want to guarantee that the moves we show people are actually good, I approach this from the other direction and use how hard it is to see Stockfish’s move as a difficulty filter. At low ratings, we show you moves where there’s a large win swing that’s visible with a shallow search, and as you get better we slowly show moves with a smaller win swing or that require a deeper search to see.
Here’s the ratings bands and gaps:
rating band
median win rate gap
0–400
0.441
1200–1600
0.315
2000–2400
0.185
2400–2800
0.065
2800–3200
0.081
So, it’s possible for us to show moves that are very hard to distinguish, but only if you’re insanely good at this task. This item from before would require you to be in our 2400+ band.