Back in the Spring of 2010, Sunny Mehta was looking at the playoff match up between the Flyers and the Devils. In comparing their goalies, he said "No one has conclusively shown a meaningful difference in skill between NHL goaltenders. I'm not saying it doesn't exist, I'm just saying no one's really proved it, and that all signs point to goaltending differences being far less important than everyone thinks." Then he did one of the most amazing things I've seen. Buried down in the comments, he said:
For those interested, I just did the following experiment…
I grabbed the numbers for every goaltender’s ‘09-’10 Even Strength shots faced and saves (minimum 5 games played). I then simulated 10,000 seasons using the assumption that there is no goaltender skill. To do this, I simply gave each of the 68 goaltenders his own "coin" weighted to .919 (which was the league average ES sv%). For each simulated season, every goaltender got to flip his coin the number of times equal to his actual shots faced (for example, since Stephen Valiquette actually faced 99 shots in ‘09-’10, he got to flip his coin 99 times in each simulated season). Then I recorded how many "heads" (i.e. saves) and percentage of heads (i.e. save percentage) each goaltender came up with in a given season. And then I looked at the spread of these "head percentages" by measuring the standard deviation amongst the goaltenders. As mentioned, I ran 10,000 of these simulated seasons, and I recorded the standard deviation every time.
The largest standard deviation in any of the 10,000 simulated seasons was .0187. The smallest was .0073. The average was .0120.
Getting back to real life, the actual standard deviation of even strength save percentages in ‘09-’10 was .0129. That means if I showed you a bunch of graphs of the simulated seasons, you would have absolutely no idea if and when I snuck the actual season in with them, because it doesn’t look any different.
In other words, looking at any goaltender’s save percentage for a single season is completely and utterly meaningless. We absolutely cannot know if it was the result of anything other than pure fucking luck. It is absurd to even give out an award called the Vezina Trophy for Best Goaltender In A Given Season. This is the equivalent of giving out an award called Best Coinflipper.
Note that if we run this type of study for certain other sport skills, we do not see this effect. For example, instead of using goaltender save percentage, if I ran the same study for NHL forwards’ ability to direct shots at net, or MLB batters’ ability to hit home runs, the simulated seasons look nothing like the actual seasons. If I were showing you graphs of simulated seasons of pitcher strikeout percentages, as soon as I snuck in the actual season you would know immediately, because of how different it looks.
Holy Shit! "It is absurd to even give out an award called the Vezina Trophy for Best Goaltender In A Given Season. This is the equivalent of giving out an award called Best Coinflipper." Still one of my favorite quotes of all time.
The original article is lost. I had to go to a web archive to find it. I'm going to re-post it so that people can see it.