Whether you like it or not, one of the main storylines of this offseason will be Monty Williams. More specifically, should Monty Williams retain his job as the Pelicans head coach? It seems like most Pels fans think that Monty has had enough time with the team and has shown too little. Some have even taken to using #FireMonty to voice their opinions on Twitter and other social media sites.

Now, I want to start off by saying that this article will not be an argument for keeping or firing Monty Williams. I think it is perfectly fine for fans to have their opinions and to voice them loudly about their team. However, it is not my place, nor it is productive for me to argue that a man should lose his job and his ability to provide for his family. After all, the coaches and players we lampoon and lament are, by all accounts and metrics, humans.

Instead, I want to provide some new insights to this coaching discussion. Some of these will come straight from my own head and others will come from some published academic work. This is a different type of Show me the Data piece. It will be a sort of literature review with my own commentary. I won’t be adding original data. Most of it will be taken from already published work. As always, comment and let me know what you think.

 

Do Coaches Really Matter?

I imagine that most of you would say yes, but renowned sports economist David Berri argues that economic theory and data tells us they don’t. Berri published his research on this topic in an article titled The Role of Mangers in Team Performance in the academic journal International Journal of Sports Finance (May 2009). A pdf can be found here.

Before I outline Dr. Berri’s argument, and I do want to make one quick point. There is often a disconnect between an academic’s published work in a peer reviewed journal and the description of that work in more popular media (i.e. blogs, interviews, etc.) Frankly, there is a tendency to overstate findings and to ignore the nuances of academia. I’m not taking a shot at Berri or anyone else. In fact, I’m not saying that it is even wrong. It is just a fact of the world. My point is that you can easily find a tweet or quote from someone referencing Berri’s that says something along the lines of, “Coaches don’t matter to team performance”. However, the whole truth is a bit more complicated than that.

Let’s start with the paper’s conclusion and work backwards. Here is a selection taken from the paper:

Our most surprising finding was that most of the coaches in our data set did not have a statistically significant impact on player performance relative to a generic coach. Even the most successful coaches by our metric—Jackson, Popovich, and Fitzsimmons— were statistically discernable only from the very worst-rated coaches. We therefore find little evidence that most coaches in the NBA are more than the “principal clerks” that Adam Smith claimed managers were more than 200 years ago (pg. 92).

As much as I love talking about the history of economic theory, I don’t want to focus on that Adam Smith reference too much. Instead, I’m going to focus on the bomb that was dropped in the preceding paragraph.  That is the idea that even the best NBA coaches have about the same effect as average NBA coaches. This conclusion isn’t quite as clear as you may think.

First of all, this paper examines how player production changes when a player begins to play for another coach. This can happen when a coach is fired, a player is traded, or another team signs a player. This can be a problem, because like all studies in the social sciences, this one is limited by the availability of data. Economists don’t have a laboratory to generate data. That means they have to take what they can get and limit their scope. That means the measure the researchers use for impact of coaching might not contain everything a coach does. It is kind of like statistically measuring defense. It is just difficult to quantify every part of coaching. For now, just realize that the data set has limitations and isn’t ideal.

If you want to get into the really nitty gritty, you’ll have to read the paper. I’ll just provide a brief summary of what comes next here. Basically, Berri and his coauthors build a model, which includes several variables they consider to have a relationship with player production. They then ran multiple linear regressions. Again, I’m not going to dive in to the details of MLR analysis, but basically this tells you the relative impact each variable in the model has on player production. The value in this method is that we can see how much a coach affects a player’s production when things like age, position, teammate quality, and roster stability are held constant.

Anyway, Berri and co. ran these regressions, and they got a bunch of numbers for each coach. These numbers told us of the relative impact of each coach in their data set. The two coaches with the biggest impact were Phil Jackson and Greg Popovich, which seems entirely reasonable. In fact, the model estimated that Pop and Jackson would add over 15 wins to their teams per season, all else being equal.

The problem came when the researchers looked at the coaches’ confidence intervals. For those of you who don’t know, a confidence interval is a way of saying that an estimate falls somewhere in a certain range. I’ll explain through a metaphor. Let’s say you and I are at a bar, and I ask you to estimate the height of a guy standing across the room’s. You may say, “I think he is 6 feet tall”. That is your estimate. Then I say, “Okay, tell me the range of heights you’re 95 percent sure he falls between.” The clever thing to do would be to say, “He is somewhere between 4 feet and 8 feet tall. “ Of course, I’d push you to give me the smallest range you’re 95% confident he falls between. You may respond, “He is taller than 5 feet 8 inches, but shorter than 6 feet 4 inches.” That is a confidence interval. That is the range of heights you’re 95% sure that guy’s height falls between. In the case of this paper, the first round of statistics estimated just a single number (they guessed the guy’s height), but when they looked at the confidence interval of the effect of each coach, the low end of the best coaches overlapped with the top end of the worst coaches. To return to my bar example, this would be like I asked you to rank 20 guys in a bar by height, but when I asked you to provide a range of their heights, the tallest guys minimum height was lower than the shortest guys tallest height.

This confidence interval issue seems to justify a piece of basic economic theory, which is that managers don’t have a huge effect on individual employee production. Now, if we go back and look at the quote from the article above, do I think that overstates things a bit? Of course. First of all, this paper just examines the way coaches impact individual player production. It doesn’t and can’t measure the more intangible sides of coaching like managing egos and personalities. Maybe, that is the most important part of a manger’s job, which is pretty hard to represent numerically.

If we ignore the confidence interval stuff for just a bit, we do see some solid estimates that coaches can have a positive and negative impact on player performance. Still, that impact overall isn’t quite as great as most fans may have thought it would be. In my opinion, a more appropriate conclusion would be that coaches do have an impact on player production and team performance. However, that impact may be lower than other variable and lower than the general public perception may have expected.

 

The Counter Factual Problem 

Let’s step back from this research and turn to something I think of as the counter factual problem. The counter factual problem is basically way of saying, we can never know the outcome of a situation if there had been a different coach on the team or if that coach had made a different choice in an in game situation. In short, we can’t produce a perfect counter example to the fact of what happened.

Pretend you and I are playing tic-tac-toe. Tic-tac-toe isn’t an interesting game, because it is completely determined, which means I know the optimal strategy and move at every point in the game.  If you lose to me, we can go back in the game and determine where you made the wrong choice. Then, we can change that decision and play the game out to see if the outcome changes.

We can’t do the same for coaches. We can’t start the Pelicans season over, and have George Karl be the head coach to see if the outcome changes. In fact, it wouldn’t even tell us much, because luck or chance play such a large role in season outcomes. Furthermore, we can’t even say that we know a coach made a bad decision, because we can’t provide the counter factual. We can’t replay the game and make a different choice with lineups or strategy to see what happens.

Now, I know some of you have become frustrated. You probably think I’m telling you that you can’t critique a coach. I’m not saying that at all. You can question coach’s decisions, but you can’t know they made the wrong choice. This is probably particularly relevant for our team. The tone surrounding Monty Williams’ performance has grown particularly knowing, in my opinion. We could all benefit from remembering that we can’t know how well things would have gone if Monty played the strategies we wished he would have.

 

Conclusion

The title of this post is, Do Coaches Really Matter? That has been the driving question here, and my answer is not as much as you think. Sure, a good coach makes a difference, but you’re going to need some other things to be successful. I know someone will post in the comments that two coaches, Jackson and Red Auerbach, have won about 30% of all the NBA titles in history. Fair enough, but how many of those teams had just average talent? How many of them had only 1 all time great player? How many had 2? Also, what franchises did they coach for? Were these teams with a history of success?

In my opinion, a team’s odds of winning a championship depends on four variables.

1. Player Talent
2. Front Office Quality
3. Coaching Talent
4. Luck

 It is a combination of all of those variables that leads to championships. You can’t just isolate one thing. So yes, I agree that coaching matters, but it isn’t the only thing. Frankly, I don’t even think it is the most important thing. You’re going to need the talent, and you’re going to need a stable front office. Can you remember a time in the Hornets’ history when the talent was there, but front office instability closed a championship window? Can you remember a time when a stable front office with an excellent coach won a championship with below average talent?

The one other Monty related comment I will make before closing is realize that the Pelicans were a below average team with respect to talent, especially after the injuries. Some of our expectations for the team, as fans, may have been unreasonably inflated due to all the offseason moves. I saw at least one site that predicted our win total would be in the mid 20’s, and they assumed everyone would be healthy. Again, you can tweet #FireMonty or not, but I promise he isn’t the only issue nor is he the most important issue with this team. At least, that is what the data says.

 

Note: Mason Ginsberg is posting a fair and detailed piece on more of a micro level about Monty Williams’ performance and future with the team on Monday. Be sure to check that out as well!

 

References

Berri, David J., Michael Leeds, Eva Leeds, and Michael Mondello. “The Role of Managers in Team Performance.” International Journal of Sports Finance. 4.2 (2009): 75-93. Web. 3 Apr. 2014.