With Round 2 over and Round 3 starting later today, here are the quick updated predictions. The model got 3/4 series right in Round 2 (it only got Anaheim-LAK wrong) so most of the predictions are the same as last time:
Revised Picks (Model):
Montreal vs. NYR [92% (Logit), predicted W pct = 0.66 (CI: 0.33, 0.99)]
Chicago vs. LAK [99% (Logit), predicted W pct = 0.80 (CI: 0.47, 1)]
Chicago vs. Montreal [66% (Logit), predicted W pct = 0.53 (CI: 0.20, 0.86)]
Based on the predicted winning percentages, here are the revised predicted series lengths:
Chicago over LAK in 5
Montreal over NYR in 6
Montreal over Chicago in 7
And here are the predicted probabilities for each team of winning the cup (summed over all brackets):
Montreal: 60%
Chicago: 31%
NYR: 8%
LAK: 1%
Obviously, I'm a little disappointed that the model narrowly missed a perfect round, but actually, the model did predict that it would get on an average of 1 series wrong last round given the fact that the predicted margins were very narrow (see my last post for details), so I still definitely see this round as redemption for the mode compared to the first round. One could maybe argue that the Gibson start in Game 7 (which I think was a huge mistake, as much as I think Gibson is a great goalie - you can't start a rookie goalie with less than 10 games NHL experience in a Game 7 against a veteran team like that and not expect him to make some mistakes) and the Gaborik acquisition at the deadline are two key things that the model did not consider when it picked the Ducks, but I also, as I said in my last post, don't think we should read too much into model errors that were in Game 7s, as single-game series come with a lot of stochasticity.
And I'm of course thrilled that the Habs prediction is still intact. The model does strongly pick LA over the Habs though, so I'm also rooting for Chicago this round. Go Habs Go!
Saturday, 17 May 2014
Thursday, 1 May 2014
1st round post-mortem and 2nd round picks
Well, another great round is in the bag, full of drama and some surprises! It was certainly not a good round for the model (4/8 picks right and both of its cup picks lost), but a great round for my Habs!
But back to the model, let's take a look at its successes and failures this round and what it might tell us about how to improve the model for next year:
First, it should be noted that the sole fact that the model got some series wrong doesn't necessarily mean it's a bad model. In fact, logit models make predictions about the failure rates of their own predictions (e.g. a predicted 70% chance of victory for one team winning a series implies a predicted 30% chance of the model being wrong). This round, the model predicted its average failure rate would be:
(5.3% + 24.8% + 2.6% + 15.6% + 11.9% + 3.1% + 23.3% + 17.6%)/8 = 13%.
This means that the model predicted it would get on average (slightly more than) one series wrong this round (1/8 = 12.5%).
In fact, the odds the model predicted for itself to get every series right this round were:
(94.7% x 75.2 x 97.4% x 84.4% x 88.1% x 96.9% x 76.7% x 82.4%) = 31.6%.
This means that the model predicted a greater than 2 in 3 chance it would get at least 1 series wrong this round. In other words, it's hardly surprising that the model missed some picks, though 4 missed picks out of 8 does seem like a particularly bad outing, since that doesn't even beat flipping a coin.
The model can predict the odds of it having an outing this bad, which might shed some light on whether something like this could have happened by chance or whether there is instead a fundamental flaw in the model. Specifically, the odds the model predicted of it getting 4 or more wrong this round were 0.98%, or roughly 1 in 100 (the formula for this is a bit too complicated to write out, but essentially you add up the predicted probabilities of every possible 4 or more mistake outcome, of which there are 163). Thus, if the model's predicted probabilities for each series were equal to the true probabilities, it would only get 4 or more series wrong 1 in every 100 first rounds.
This seems unlikely. Much more likely is that the model is missing something important. In order to facilitate the discussion on this, here are the model effects that are currently included:
1) Average seasons series (SS) goal differential (i.e. (GF - GA)/# games in the SS)
2) Transformed average SS outcome, weighted by the number of games (where shootout wins and losses count as ties) = (SS winning percentage (0-1) - 0.5) x # games in SS
3) Difference in average GF per game in the season between the teams
4) Difference in average GA per game in the season between the teams
5) Difference in overall season winning percentage (0-1) between the teams
The second most supported model included these same 5 effects plus the difference between the teams in average age among the players. I did not show the results of this model in my previous posts, but all of its picks were the same (though it gave weaker odds to Detroit, Dallas, Colorado, and Tampa Bay). Experience certainly could have been a factor in the Dallas and Colorado series, and perhaps also the Boston series. However, I suspect the greatest thing missing from the model currently is something that factors in the effects of trade deadline trades (both good or bad), late season injuries (or season-long injuries that heal), and teams coming into the playoffs hot or cold. For example, I had a feeling Tampa Bay was probably a bad pick by the model last round because of Bishop's injury (combined with Lindback's poor numbers during the season) and Montreal's strong finish since the Thomas Vanek trade. A similar effect of the Matt Duchene injury and Matt Moulson trade could be argued for the Wild (though I won't argue for too much determinism in a Game 7 overtime victory). Boston was hot coming into the playoffs and they also have a forechecking, physical - often dirty - style of play that is advantageous in the playoffs due to the high adrenaline atmosphere and the more relaxed officiating. I one day hope to be able to test these hypotheses with some more in depth player stats than I currently have.
Of course, the effect of chance should also not be discounted. For example, even though the model's pick for the Dallas-Anaheim series (Dallas) was ultimately wrong, I still believe that may have been a good pick (i.e. more often than not, Dallas wins that series). I thought they were the better team in Game 5, but were on the wrong end of some calls, and they should have been able to close out Game 6, though maybe that was one instance where their inexperience was a factor. Tough to say. Similarly, though LA ended up being the right pick, who could have predicted the uncharacteristically poor play of their defense and Quick in Games 1-3? They could have very easily lost that series after being in the 3-0 hole.
One hypothesis that I have is that the increase in the number of teams throughout the modern era, coupled in particular with the rise of the salary cap recently, have made the playoffs generally less predictable over time by making the playing field more level. This can be tested to some degree using the model by comparing observed and predicted model error rates over time to see if they have an increasing trend, which it appears they do, at least in the predicted error rates (Figure 2014.3). This is not a perfect test though, as time trends in error rates could also be caused differences in effect strengths in different time periods.
Anyway, let me know in the comments what you think should be added to the model and hopefully we can build a model that works better next year! (though I do believe that no model or algorithm will ever be able to perfectly predict the playoffs, and there will always be a role of chance)
In the meantime, here are the revised picks of the model, and some thoughts on these picks, in light of known vulnerabilities of the model:
Revised Picks (Model):
Boston vs. Montreal [63% (Logit), predicted W pct = 0.55 (CI: 0.20, 0.86)]
Pittsburgh vs. NYR [72% (Logit), predicted W pct = 0.57 (CI: 0.24, 0.90)]
Montreal vs. NYR [92% (Logit), predicted W pct = 0.66 (CI: 0.33, 0.99)]
Chicago vs. Minnesota [77% (Logit), predicted W pct = 0.58 (CI: 0.25, 0.91)]
Anaheim vs. LAK [91% (Logit), predicted W pct = 0.65 (CI: 0.39, 1)]
Anaheim vs. Chicago [96% (Logit), predicted W pct = 0.72 (CI: 0.32, 0.99)]
Chicago vs. Montreal [66% (Logit), predicted W pct = 0.53 (CI: 0.20, 0.86)]
Obviously, I'm ecstatic about this pick (though I promise it wasn't rigged), and definitely hope the model is right from here on out! And remember, the model did not give the Habs enough credit the last round (though it also did not give Boston enough credit). However, it is also worth noting that the Habs going all the way depends a lot on the outcomes of the other series, which give them matchups the model predicts could be favourable; and that many of the predictions are not strong (i.e do not have large margins of predicted certainty). In particular, the predicted odds of all picks being right this round are: (63% x 72% x 77% x 91%) = 31.8%, and the average certainty is 76%, which means that the model is again predicting it should get on average one of these series wrong. In fact, if you translate the predicted winning percentages form the continuous model, and compare these to the winning percentages obtained from each series length (1 = sweeping, 0.8 = winning in 5, 0.67 = winning in 6, 0.57 = winning in 7), then the model is predicting the following series lengths:
Montreal over Boston in 7
NYR over Pittsburgh in 7
Chicago over Minnesota in 7
Anaheim over LAK in 6
Chicago over Anaheim in 6
Montreal over NYR in 6
Montreal over Chicago in 7
Based on these picks, and considering which teams the model seemed to underestimate last round, I'd guess that Chicago and Montreal may be the most likely of these picks to fall. On the other hand, considering Montreal was also underestimated, and Chicago has lots of playoff experience, which is poorly captured (not to mention the possible injury to Kuemper tonight), these picks may be more spot on than last rounds. Only time will tell!
That's all for this post. Happy picking and Go Habs Go! I will calculate the predicted probabilities of each team winning the cup, integrated over all remaining possible brackets, in my next post!
But back to the model, let's take a look at its successes and failures this round and what it might tell us about how to improve the model for next year:
First, it should be noted that the sole fact that the model got some series wrong doesn't necessarily mean it's a bad model. In fact, logit models make predictions about the failure rates of their own predictions (e.g. a predicted 70% chance of victory for one team winning a series implies a predicted 30% chance of the model being wrong). This round, the model predicted its average failure rate would be:
(5.3% + 24.8% + 2.6% + 15.6% + 11.9% + 3.1% + 23.3% + 17.6%)/8 = 13%.
This means that the model predicted it would get on average (slightly more than) one series wrong this round (1/8 = 12.5%).
In fact, the odds the model predicted for itself to get every series right this round were:
(94.7% x 75.2 x 97.4% x 84.4% x 88.1% x 96.9% x 76.7% x 82.4%) = 31.6%.
This means that the model predicted a greater than 2 in 3 chance it would get at least 1 series wrong this round. In other words, it's hardly surprising that the model missed some picks, though 4 missed picks out of 8 does seem like a particularly bad outing, since that doesn't even beat flipping a coin.
The model can predict the odds of it having an outing this bad, which might shed some light on whether something like this could have happened by chance or whether there is instead a fundamental flaw in the model. Specifically, the odds the model predicted of it getting 4 or more wrong this round were 0.98%, or roughly 1 in 100 (the formula for this is a bit too complicated to write out, but essentially you add up the predicted probabilities of every possible 4 or more mistake outcome, of which there are 163). Thus, if the model's predicted probabilities for each series were equal to the true probabilities, it would only get 4 or more series wrong 1 in every 100 first rounds.
This seems unlikely. Much more likely is that the model is missing something important. In order to facilitate the discussion on this, here are the model effects that are currently included:
1) Average seasons series (SS) goal differential (i.e. (GF - GA)/# games in the SS)
2) Transformed average SS outcome, weighted by the number of games (where shootout wins and losses count as ties) = (SS winning percentage (0-1) - 0.5) x # games in SS
3) Difference in average GF per game in the season between the teams
4) Difference in average GA per game in the season between the teams
5) Difference in overall season winning percentage (0-1) between the teams
The second most supported model included these same 5 effects plus the difference between the teams in average age among the players. I did not show the results of this model in my previous posts, but all of its picks were the same (though it gave weaker odds to Detroit, Dallas, Colorado, and Tampa Bay). Experience certainly could have been a factor in the Dallas and Colorado series, and perhaps also the Boston series. However, I suspect the greatest thing missing from the model currently is something that factors in the effects of trade deadline trades (both good or bad), late season injuries (or season-long injuries that heal), and teams coming into the playoffs hot or cold. For example, I had a feeling Tampa Bay was probably a bad pick by the model last round because of Bishop's injury (combined with Lindback's poor numbers during the season) and Montreal's strong finish since the Thomas Vanek trade. A similar effect of the Matt Duchene injury and Matt Moulson trade could be argued for the Wild (though I won't argue for too much determinism in a Game 7 overtime victory). Boston was hot coming into the playoffs and they also have a forechecking, physical - often dirty - style of play that is advantageous in the playoffs due to the high adrenaline atmosphere and the more relaxed officiating. I one day hope to be able to test these hypotheses with some more in depth player stats than I currently have.
Of course, the effect of chance should also not be discounted. For example, even though the model's pick for the Dallas-Anaheim series (Dallas) was ultimately wrong, I still believe that may have been a good pick (i.e. more often than not, Dallas wins that series). I thought they were the better team in Game 5, but were on the wrong end of some calls, and they should have been able to close out Game 6, though maybe that was one instance where their inexperience was a factor. Tough to say. Similarly, though LA ended up being the right pick, who could have predicted the uncharacteristically poor play of their defense and Quick in Games 1-3? They could have very easily lost that series after being in the 3-0 hole.
One hypothesis that I have is that the increase in the number of teams throughout the modern era, coupled in particular with the rise of the salary cap recently, have made the playoffs generally less predictable over time by making the playing field more level. This can be tested to some degree using the model by comparing observed and predicted model error rates over time to see if they have an increasing trend, which it appears they do, at least in the predicted error rates (Figure 2014.3). This is not a perfect test though, as time trends in error rates could also be caused differences in effect strengths in different time periods.
 |
| Figure 2014.3. Predicted (blue) and observed (red) model error rates. Predicted error rates have been increasing through time at a statistically significant rate (p = 0.002). However, observed error rates do not show a statistically significant increasing trend (p = 0.71). |
Anyway, let me know in the comments what you think should be added to the model and hopefully we can build a model that works better next year! (though I do believe that no model or algorithm will ever be able to perfectly predict the playoffs, and there will always be a role of chance)
In the meantime, here are the revised picks of the model, and some thoughts on these picks, in light of known vulnerabilities of the model:
Revised Picks (Model):
Boston vs. Montreal [63% (Logit), predicted W pct = 0.55 (CI: 0.20, 0.86)]
Pittsburgh vs. NYR [72% (Logit), predicted W pct = 0.57 (CI: 0.24, 0.90)]
Montreal vs. NYR [92% (Logit), predicted W pct = 0.66 (CI: 0.33, 0.99)]
Chicago vs. Minnesota [77% (Logit), predicted W pct = 0.58 (CI: 0.25, 0.91)]
Anaheim vs. LAK [91% (Logit), predicted W pct = 0.65 (CI: 0.39, 1)]
Anaheim vs. Chicago [96% (Logit), predicted W pct = 0.72 (CI: 0.32, 0.99)]
Chicago vs. Montreal [66% (Logit), predicted W pct = 0.53 (CI: 0.20, 0.86)]
Obviously, I'm ecstatic about this pick (though I promise it wasn't rigged), and definitely hope the model is right from here on out! And remember, the model did not give the Habs enough credit the last round (though it also did not give Boston enough credit). However, it is also worth noting that the Habs going all the way depends a lot on the outcomes of the other series, which give them matchups the model predicts could be favourable; and that many of the predictions are not strong (i.e do not have large margins of predicted certainty). In particular, the predicted odds of all picks being right this round are: (63% x 72% x 77% x 91%) = 31.8%, and the average certainty is 76%, which means that the model is again predicting it should get on average one of these series wrong. In fact, if you translate the predicted winning percentages form the continuous model, and compare these to the winning percentages obtained from each series length (1 = sweeping, 0.8 = winning in 5, 0.67 = winning in 6, 0.57 = winning in 7), then the model is predicting the following series lengths:
Montreal over Boston in 7
NYR over Pittsburgh in 7
Chicago over Minnesota in 7
Anaheim over LAK in 6
Chicago over Anaheim in 6
Montreal over NYR in 6
Montreal over Chicago in 7
Based on these picks, and considering which teams the model seemed to underestimate last round, I'd guess that Chicago and Montreal may be the most likely of these picks to fall. On the other hand, considering Montreal was also underestimated, and Chicago has lots of playoff experience, which is poorly captured (not to mention the possible injury to Kuemper tonight), these picks may be more spot on than last rounds. Only time will tell!
That's all for this post. Happy picking and Go Habs Go! I will calculate the predicted probabilities of each team winning the cup, integrated over all remaining possible brackets, in my next post!
Subscribe to:
Comments (Atom)