I think it’d make sense to give C at least as long as B. B doesn’t need to do any explaining. I think giving A significantly longer than B is fine, so long as the players have enough time to stick around for that. I think it’s a more interesting experiment if A has ample time to figure things out to the best of their ability. A failing because they weren’t able to understand quickly seems less interesting.
The best way to handle this seems to be to play A-vs-B and B-vs-C control games with something as close to the final setup as possible.
So e.g. you could have B-vs-C games to check that C really is significantly better, but require C to write explanations for their move and why they didn’t make a couple of other moves. Essentially C imagines they’re playing the final setup, except their move is always picked. And you can do A-vs-B games where A has a significant advantage in time over B (though I think blitz games is still more efficient in gaining the most information in the given time).
This way it doesn’t matter much whether the setup is ‘fair’ to A/B/C, so long as it’s unfair to a similar level in the control 1-v-1 games as in the advisor-based games.
That said, I don’t expect the setup to be particularly sensitive to the control games or time controls.
If you have something like: A: novice B: ~1700 C: ~2200 Then A is going to robustly lose to B and B to C. An extra couple of minutes either way isn’t going to matter. (thinking for longer might get you 100 Elo, but nowhere close to 500)
If this reliably holds—e.g. B beats A 9-0 in blitz games, and the same for C vs B, then it doesn’t seem worth the time to do more careful controls. (or at least the primary reason to do more careful controls at that point would be a worry that the results wouldn’t otherwise be taken seriously by some because you weren’t doing Proper Science)
I think it’d make sense to give C at least as long as B. B doesn’t need to do any explaining.
I think giving A significantly longer than B is fine, so long as the players have enough time to stick around for that.
I think it’s a more interesting experiment if A has ample time to figure things out to the best of their ability. A failing because they weren’t able to understand quickly seems less interesting.
The best way to handle this seems to be to play A-vs-B and B-vs-C control games with something as close to the final setup as possible.
So e.g. you could have B-vs-C games to check that C really is significantly better, but require C to write explanations for their move and why they didn’t make a couple of other moves. Essentially C imagines they’re playing the final setup, except their move is always picked.
And you can do A-vs-B games where A has a significant advantage in time over B (though I think blitz games is still more efficient in gaining the most information in the given time).
This way it doesn’t matter much whether the setup is ‘fair’ to A/B/C, so long as it’s unfair to a similar level in the control 1-v-1 games as in the advisor-based games.
That said, I don’t expect the setup to be particularly sensitive to the control games or time controls.
If you have something like:
A: novice
B: ~1700
C: ~2200
Then A is going to robustly lose to B and B to C.
An extra couple of minutes either way isn’t going to matter. (thinking for longer might get you 100 Elo, but nowhere close to 500)
If this reliably holds—e.g. B beats A 9-0 in blitz games, and the same for C vs B, then it doesn’t seem worth the time to do more careful controls. (or at least the primary reason to do more careful controls at that point would be a worry that the results wouldn’t otherwise be taken seriously by some because you weren’t doing Proper Science)