I did some initial analysis and initially came up with the same team as Measure and Alumium. It’s also what abstractapplic was on the fence on (but chose differently). Remarks on the initial analysis:
the water, fire and earth groups remarked on by abstractapplic are, it seems to me, more easily noticeable via their (anti-)synergies than via their counters, which seem somewhat hit and miss, presumably due to more individual countering effects. The groups are:
Fire: BFIOPV
Water: ACMSTW
Earth: DEGLQR
and Nullifying Nightmare is separate from each.
The enemy team is, recall, DGPQT
My initial pick was BGNRT was based on green side counters (imagining the enemy on the blue team). GNT are generically strong, R counters DQT on the enemy team, and B counters D and P and (weakly) Q, is not as bad against G as most heroes, and is not as badly stomped by T than other fire characters.
The synergies did not seem important enough to affect the team selection particularly much.
Then I saw Maxwell Peterson’s post and downloaded his gradient-boosting R code. I suggest reading his post first. Remarks based on my use of this code:
Maxwell used 350 training steps, based on green and assuming the enemy to be blue. His code outputted AGLNP as the best counter to the target enemy team DGPQT.
My remarks:
The validation is still improving when Maxwell stops training in the code as supplied in that post. So I increased the number of training steps. The risk is that it will just overfit more, but I assume that if that was a big problem it would make the validation worse?
Changing to 1000 training steps results in the code outputting AGLNO as the best counter to the enemy team, with AGLNP as the fourth pick.
The problem as supplied by aphyer does not seem to specify as to whether we are Green or Blue. So we should probably be prepared for both (and this especially applies for PVP). I also wondered if the apparent differences were due to random chance. So, I created a flipped version of the initial csv file (changing all green characters to blue and vice versa, and also the wins). Maxwell’s code run on this flipped version outputted LOPRT as the best counter to DGPQT - quite the change!
I also created a merged csv file from the original and flipped files and 2 cut csv files from the merged file which each contain some entries from the original and some from the flipped file (but no overlap between each other). Validation scores on the cut csv files were worse than on the flipped and original csv files, confirming that side does matter (unless I misimplemented the flip, which would embarrassingly invalidate all my subsequent analysis...). However, even though side does matter, since we don’t know what side we are on, I figure we are mainly interested in averaged data so used the merged file anyway. I also increased the number of steps to 1500 for the merged model and doubled the cutoff indices for the validation (because 2x the data).
The code now outputs BGLPR as the best counter to DGPQT. So, that’s my choice (unless I change it later). It also happens to be one off from GuySrinivasan’s choice (who picked M instead of L).
For PVP, against the average team the code now recommends BGNOT. However, I’d like to modify it to be more specialized against likely opponent teams. This will take some time, possibly more than I can spare, since I have not used R before. If I don’t have time to adjust it, I will go with BGNOT.
edit: now switching to ABGOT as the PVP team. According to the code this counters my above choice, as well as gjm’s pick of BGNPT (sorry gjm). Also seems to do OK against others who have published PVP teams (a good reason not to publish them). Attempts to get a better value estimator for PVP teams and apply it to all possible choices have thus far been thwarted by my unfamiliarity with R.
Post-mortem on my thinking about the sides being asymmetrical:
In order to determine whether there was symmetry, I applied the model to the following datasets and compared the validation scores:
The original data set
A flipped version of the data set
a “cut” version of the data set where some of the data points were flipped and others not (should contains all games in one form or the other, and not have any of the same games)
the complement of the above (everything flipped rather than 3
On finding that 1 and 2 had better validation scores than 3 and 4, and the gap between 1⁄2 and 3⁄4 was larger (but really not all that much larger!) than the gap between 1 and 2 or 3 and 4, I declared that there was asymmetry.
But, really this was totally invalid, because, 1 and 2 are isomorphic to each other under a column swap and bit flip (as pointed out by Maxwell) and while this transformation may affect the results it should not affect validation scores if the algorithm is unbiased, up to random variation if it has random elements (I don’t know if it does, but the validation scores were not actually identical). Likewise, 3 and 4 should have the same validation scores. On the other hand, 1⁄2 are not isomorphic to 3⁄4 up to such a transformation and so have no need to have the same validation scores. So there was a 50% chance of the observed result happening by chance.
Even if my method would have worked the way I had been thinking**, it would be a pretty weak* test. So why was I so willing to believe it? Well, in my previous analysis I had noticed differences in the sides, which might or might not be random, particularly the winrate for games with Greenery Giant on both sides. In such matchups, green wins 1192, (ironically) much less than blue’s 1308. This is not at all unlikely (less than 2 sigma (which I didn’t check), and many other possible hypotheses) but this plus the knowledge that League of Legend’s map is, while almost symmetrical, not perfectly so, led me to have a too-weak prior against asymmetry when going into poking Maxwell’s magic box.
Regardless of this mistake, I do think that my choice to create and use a merged data set including the original data and the flipped data was correct. Given that we either don’t care about asymmetries or don’t believe they exist, the ideal thing to do would be to add some kind of constraint to the learning algorithm to respect the assumed symmetry, but this is an easier alternative.
*Edit: in the sense of providing weak Bayesian evidence due to high false positive rate
**Edit: which I could have made happen by comparing results from disjoint subsets of the data in each of 1 and 2, etc.
Actually, I’ve thought about it more, and I don’t think it’s possible for a flip to change the predictor like this. Flipping like this is equivalent to swapping the first 19 columns of the matrices with the latter 19 columns, and bit-flipping the response. This should end up giving prediction vectors v_flip such that 1 - v_flip = v_original. So my money is currently on something being off with the code that flips.
I don’t understand the details of how your code works very well, but wouldn’t 1-v_flip = v_original be what you would get with just flipping the response without swapping columns?
Also, I spot-checked the flipped csv file and didn’t see any discrepancies.
Yup, it would be—I guess I’m saying that I don’t think swapping the columns has any effect. To the model, during training, it is just 38 unnamed columns. Swapping the first 19 with the last shouldn’t do anything? Weird weird weird
I don’t understand what you mean about swapping the columns not having any effect, it would seem to imply, e.g. that swapping which characters you have on your team would have no effect.
I did some initial analysis and initially came up with the same team as Measure and Alumium. It’s also what abstractapplic was on the fence on (but chose differently). Remarks on the initial analysis:
the water, fire and earth groups remarked on by abstractapplic are, it seems to me, more easily noticeable via their (anti-)synergies than via their counters, which seem somewhat hit and miss, presumably due to more individual countering effects. The groups are:
Fire: BFIOPV
Water: ACMSTW
Earth: DEGLQR
and Nullifying Nightmare is separate from each.
The enemy team is, recall, DGPQT
My initial pick was BGNRT was based on green side counters (imagining the enemy on the blue team). GNT are generically strong, R counters DQT on the enemy team, and B counters D and P and (weakly) Q, is not as bad against G as most heroes, and is not as badly stomped by T than other fire characters.
The synergies did not seem important enough to affect the team selection particularly much.
Then I saw Maxwell Peterson’s post and downloaded his gradient-boosting R code. I suggest reading his post first. Remarks based on my use of this code:
Maxwell used 350 training steps, based on green and assuming the enemy to be blue. His code outputted AGLNP as the best counter to the target enemy team DGPQT.
My remarks:
The validation is still improving when Maxwell stops training in the code as supplied in that post. So I increased the number of training steps. The risk is that it will just overfit more, but I assume that if that was a big problem it would make the validation worse?
Changing to 1000 training steps results in the code outputting AGLNO as the best counter to the enemy team, with AGLNP as the fourth pick.
The problem as supplied by aphyer does not seem to specify as to whether we are Green or Blue. So we should probably be prepared for both (and this especially applies for PVP). I also wondered if the apparent differences were due to random chance. So, I created a flipped version of the initial csv file (changing all green characters to blue and vice versa, and also the wins). Maxwell’s code run on this flipped version outputted LOPRT as the best counter to DGPQT - quite the change!
I also created a merged csv file from the original and flipped files and 2 cut csv files from the merged file which each contain some entries from the original and some from the flipped file (but no overlap between each other). Validation scores on the cut csv files were worse than on the flipped and original csv files, confirming that side does matter (unless I misimplemented the flip, which would embarrassingly invalidate all my subsequent analysis...). However, even though side does matter, since we don’t know what side we are on, I figure we are mainly interested in averaged data so used the merged file anyway. I also increased the number of steps to 1500 for the merged model and doubled the cutoff indices for the validation (because 2x the data).
The code now outputs BGLPR as the best counter to DGPQT. So, that’s my choice (unless I change it later). It also happens to be one off from GuySrinivasan’s choice (who picked M instead of L).
For PVP, against the average team the code now recommends BGNOT. However, I’d like to modify it to be more specialized against likely opponent teams. This will take some time, possibly more than I can spare, since I have not used R before.
If I don’t have time to adjust it, I will go with BGNOT.edit: now switching to ABGOT as the PVP team. According to the code this counters my above choice, as well as gjm’s pick of BGNPT (sorry gjm). Also seems to do OK against others who have published PVP teams (a good reason not to publish them). Attempts to get a better value estimator for PVP teams and apply it to all possible choices have thus far been thwarted by my unfamiliarity with R.
Current choices:
BGLPR for main answer
ABGOT as PVP team
Woah! The effect of flipping is disturbing. Interesting find.
Post-mortem on my thinking about the sides being asymmetrical:
In order to determine whether there was symmetry, I applied the model to the following datasets and compared the validation scores:
The original data set
A flipped version of the data set
a “cut” version of the data set where some of the data points were flipped and others not (should contains all games in one form or the other, and not have any of the same games)
the complement of the above (everything flipped rather than 3
On finding that 1 and 2 had better validation scores than 3 and 4, and the gap between 1⁄2 and 3⁄4 was larger (but really not all that much larger!) than the gap between 1 and 2 or 3 and 4, I declared that there was asymmetry.
But, really this was totally invalid, because, 1 and 2 are isomorphic to each other under a column swap and bit flip (as pointed out by Maxwell) and while this transformation may affect the results it should not affect validation scores if the algorithm is unbiased, up to random variation if it has random elements (I don’t know if it does, but the validation scores were not actually identical). Likewise, 3 and 4 should have the same validation scores. On the other hand, 1⁄2 are not isomorphic to 3⁄4 up to such a transformation and so have no need to have the same validation scores. So there was a 50% chance of the observed result happening by chance.
Even if my method would have worked the way I had been thinking**, it would be a pretty weak* test. So why was I so willing to believe it? Well, in my previous analysis I had noticed differences in the sides, which might or might not be random, particularly the winrate for games with Greenery Giant on both sides. In such matchups, green wins 1192, (ironically) much less than blue’s 1308. This is not at all unlikely (less than 2 sigma (which I didn’t check), and many other possible hypotheses) but this plus the knowledge that League of Legend’s map is, while almost symmetrical, not perfectly so, led me to have a too-weak prior against asymmetry when going into poking Maxwell’s magic box.
Regardless of this mistake, I do think that my choice to create and use a merged data set including the original data and the flipped data was correct. Given that we either don’t care about asymmetries or don’t believe they exist, the ideal thing to do would be to add some kind of constraint to the learning algorithm to respect the assumed symmetry, but this is an easier alternative.
*Edit: in the sense of providing weak Bayesian evidence due to high false positive rate
**Edit: which I could have made happen by comparing results from disjoint subsets of the data in each of 1 and 2, etc.
Actually, I’ve thought about it more, and I don’t think it’s possible for a flip to change the predictor like this. Flipping like this is equivalent to swapping the first 19 columns of the matrices with the latter 19 columns, and bit-flipping the response. This should end up giving prediction vectors v_flip such that 1 - v_flip = v_original. So my money is currently on something being off with the code that flips.
I don’t understand the details of how your code works very well, but wouldn’t 1-v_flip = v_original be what you would get with just flipping the response without swapping columns?
Also, I spot-checked the flipped csv file and didn’t see any discrepancies.
Yup, it would be—I guess I’m saying that I don’t think swapping the columns has any effect. To the model, during training, it is just 38 unnamed columns. Swapping the first 19 with the last shouldn’t do anything? Weird weird weird
I don’t understand what you mean about swapping the columns not having any effect, it would seem to imply, e.g. that swapping which characters you have on your team would have no effect.