But back to the model, let's take a look at its successes and failures this round and what it might tell us about how to improve the model for next year:
First, it should be noted that the sole fact that the model got some series wrong doesn't necessarily mean it's a bad model. In fact, logit models make predictions about the failure rates of their own predictions (e.g. a predicted 70% chance of victory for one team winning a series implies a predicted 30% chance of the model being wrong). This round, the model predicted its average failure rate would be:
(5.3% + 24.8% + 2.6% + 15.6% + 11.9% + 3.1% + 23.3% + 17.6%)/8 = 13%.
This means that the model predicted it would get on average (slightly more than) one series wrong this round (1/8 = 12.5%).
In fact, the odds the model predicted for itself to get every series right this round were:
(94.7% x 75.2 x 97.4% x 84.4% x 88.1% x 96.9% x 76.7% x 82.4%) = 31.6%.
This means that the model predicted a greater than 2 in 3 chance it would get at least 1 series wrong this round. In other words, it's hardly surprising that the model missed some picks, though 4 missed picks out of 8 does seem like a particularly bad outing, since that doesn't even beat flipping a coin.
The model can predict the odds of it having an outing this bad, which might shed some light on whether something like this could have happened by chance or whether there is instead a fundamental flaw in the model. Specifically, the odds the model predicted of it getting 4 or more wrong this round were 0.98%, or roughly 1 in 100 (the formula for this is a bit too complicated to write out, but essentially you add up the predicted probabilities of every possible 4 or more mistake outcome, of which there are 163). Thus, if the model's predicted probabilities for each series were equal to the true probabilities, it would only get 4 or more series wrong 1 in every 100 first rounds.
This seems unlikely. Much more likely is that the model is missing something important. In order to facilitate the discussion on this, here are the model effects that are currently included:
1) Average seasons series (SS) goal differential (i.e. (GF - GA)/# games in the SS)
2) Transformed average SS outcome, weighted by the number of games (where shootout wins and losses count as ties) = (SS winning percentage (0-1) - 0.5) x # games in SS
3) Difference in average GF per game in the season between the teams
4) Difference in average GA per game in the season between the teams
5) Difference in overall season winning percentage (0-1) between the teams
The second most supported model included these same 5 effects plus the difference between the teams in average age among the players. I did not show the results of this model in my previous posts, but all of its picks were the same (though it gave weaker odds to Detroit, Dallas, Colorado, and Tampa Bay). Experience certainly could have been a factor in the Dallas and Colorado series, and perhaps also the Boston series. However, I suspect the greatest thing missing from the model currently is something that factors in the effects of trade deadline trades (both good or bad), late season injuries (or season-long injuries that heal), and teams coming into the playoffs hot or cold. For example, I had a feeling Tampa Bay was probably a bad pick by the model last round because of Bishop's injury (combined with Lindback's poor numbers during the season) and Montreal's strong finish since the Thomas Vanek trade. A similar effect of the Matt Duchene injury and Matt Moulson trade could be argued for the Wild (though I won't argue for too much determinism in a Game 7 overtime victory). Boston was hot coming into the playoffs and they also have a forechecking, physical - often dirty - style of play that is advantageous in the playoffs due to the high adrenaline atmosphere and the more relaxed officiating. I one day hope to be able to test these hypotheses with some more in depth player stats than I currently have.
Of course, the effect of chance should also not be discounted. For example, even though the model's pick for the Dallas-Anaheim series (Dallas) was ultimately wrong, I still believe that may have been a good pick (i.e. more often than not, Dallas wins that series). I thought they were the better team in Game 5, but were on the wrong end of some calls, and they should have been able to close out Game 6, though maybe that was one instance where their inexperience was a factor. Tough to say. Similarly, though LA ended up being the right pick, who could have predicted the uncharacteristically poor play of their defense and Quick in Games 1-3? They could have very easily lost that series after being in the 3-0 hole.
One hypothesis that I have is that the increase in the number of teams throughout the modern era, coupled in particular with the rise of the salary cap recently, have made the playoffs generally less predictable over time by making the playing field more level. This can be tested to some degree using the model by comparing observed and predicted model error rates over time to see if they have an increasing trend, which it appears they do, at least in the predicted error rates (Figure 2014.3). This is not a perfect test though, as time trends in error rates could also be caused differences in effect strengths in different time periods.
 |
| Figure 2014.3. Predicted (blue) and observed (red) model error rates. Predicted error rates have been increasing through time at a statistically significant rate (p = 0.002). However, observed error rates do not show a statistically significant increasing trend (p = 0.71). |
Anyway, let me know in the comments what you think should be added to the model and hopefully we can build a model that works better next year! (though I do believe that no model or algorithm will ever be able to perfectly predict the playoffs, and there will always be a role of chance)
In the meantime, here are the revised picks of the model, and some thoughts on these picks, in light of known vulnerabilities of the model:
Revised Picks (Model):
Boston vs. Montreal [63% (Logit), predicted W pct = 0.55 (CI: 0.20, 0.86)]
Pittsburgh vs. NYR [72% (Logit), predicted W pct = 0.57 (CI: 0.24, 0.90)]
Montreal vs. NYR [92% (Logit), predicted W pct = 0.66 (CI: 0.33, 0.99)]
Chicago vs. Minnesota [77% (Logit), predicted W pct = 0.58 (CI: 0.25, 0.91)]
Anaheim vs. LAK [91% (Logit), predicted W pct = 0.65 (CI: 0.39, 1)]
Anaheim vs. Chicago [96% (Logit), predicted W pct = 0.72 (CI: 0.32, 0.99)]
Chicago vs. Montreal [66% (Logit), predicted W pct = 0.53 (CI: 0.20, 0.86)]
Obviously, I'm ecstatic about this pick (though I promise it wasn't rigged), and definitely hope the model is right from here on out! And remember, the model did not give the Habs enough credit the last round (though it also did not give Boston enough credit). However, it is also worth noting that the Habs going all the way depends a lot on the outcomes of the other series, which give them matchups the model predicts could be favourable; and that many of the predictions are not strong (i.e do not have large margins of predicted certainty). In particular, the predicted odds of all picks being right this round are: (63% x 72% x 77% x 91%) = 31.8%, and the average certainty is 76%, which means that the model is again predicting it should get on average one of these series wrong. In fact, if you translate the predicted winning percentages form the continuous model, and compare these to the winning percentages obtained from each series length (1 = sweeping, 0.8 = winning in 5, 0.67 = winning in 6, 0.57 = winning in 7), then the model is predicting the following series lengths:
Montreal over Boston in 7
NYR over Pittsburgh in 7
Chicago over Minnesota in 7
Anaheim over LAK in 6
Chicago over Anaheim in 6
Montreal over NYR in 6
Montreal over Chicago in 7
Based on these picks, and considering which teams the model seemed to underestimate last round, I'd guess that Chicago and Montreal may be the most likely of these picks to fall. On the other hand, considering Montreal was also underestimated, and Chicago has lots of playoff experience, which is poorly captured (not to mention the possible injury to Kuemper tonight), these picks may be more spot on than last rounds. Only time will tell!
That's all for this post. Happy picking and Go Habs Go! I will calculate the predicted probabilities of each team winning the cup, integrated over all remaining possible brackets, in my next post!
No comments:
Post a Comment