Do Coaches Really Matter?

By:
Published: April 4, 2014
Show me the Data

Whether you like it or not, one of the main storylines of this offseason will be Monty Williams. More specifically, should Monty Williams retain his job as the Pelicans head coach? It seems like most Pels fans think that Monty has had enough time with the team and has shown too little. Some have even taken to using #FireMonty to voice their opinions on Twitter and other social media sites.

Now, I want to start off by saying that this article will not be an argument for keeping or firing Monty Williams. I think it is perfectly fine for fans to have their opinions and to voice them loudly about their team. However, it is not my place, nor it is productive for me to argue that a man should lose his job and his ability to provide for his family. After all, the coaches and players we lampoon and lament are, by all accounts and metrics, humans.

Instead, I want to provide some new insights to this coaching discussion. Some of these will come straight from my own head and others will come from some published academic work. This is a different type of Show me the Data piece. It will be a sort of literature review with my own commentary. I won’t be adding original data. Most of it will be taken from already published work. As always, comment and let me know what you think.

 

Do Coaches Really Matter?

I imagine that most of you would say yes, but renowned sports economist David Berri argues that economic theory and data tells us they don’t. Berri published his research on this topic in an article titled The Role of Mangers in Team Performance in the academic journal International Journal of Sports Finance (May 2009). A pdf can be found here.

Before I outline Dr. Berri’s argument, and I do want to make one quick point. There is often a disconnect between an academic’s published work in a peer reviewed journal and the description of that work in more popular media (i.e. blogs, interviews, etc.) Frankly, there is a tendency to overstate findings and to ignore the nuances of academia. I’m not taking a shot at Berri or anyone else. In fact, I’m not saying that it is even wrong. It is just a fact of the world. My point is that you can easily find a tweet or quote from someone referencing Berri’s that says something along the lines of, “Coaches don’t matter to team performance”. However, the whole truth is a bit more complicated than that.

Let’s start with the paper’s conclusion and work backwards. Here is a selection taken from the paper:

Our most surprising finding was that most of the coaches in our data set did not have a statistically significant impact on player performance relative to a generic coach. Even the most successful coaches by our metric—Jackson, Popovich, and Fitzsimmons— were statistically discernable only from the very worst-rated coaches. We therefore find little evidence that most coaches in the NBA are more than the “principal clerks” that Adam Smith claimed managers were more than 200 years ago (pg. 92).

As much as I love talking about the history of economic theory, I don’t want to focus on that Adam Smith reference too much. Instead, I’m going to focus on the bomb that was dropped in the preceding paragraph.  That is the idea that even the best NBA coaches have about the same effect as average NBA coaches. This conclusion isn’t quite as clear as you may think.

First of all, this paper examines how player production changes when a player begins to play for another coach. This can happen when a coach is fired, a player is traded, or another team signs a player. This can be a problem, because like all studies in the social sciences, this one is limited by the availability of data. Economists don’t have a laboratory to generate data. That means they have to take what they can get and limit their scope. That means the measure the researchers use for impact of coaching might not contain everything a coach does. It is kind of like statistically measuring defense. It is just difficult to quantify every part of coaching. For now, just realize that the data set has limitations and isn’t ideal.

If you want to get into the really nitty gritty, you’ll have to read the paper. I’ll just provide a brief summary of what comes next here. Basically, Berri and his coauthors build a model, which includes several variables they consider to have a relationship with player production. They then ran multiple linear regressions. Again, I’m not going to dive in to the details of MLR analysis, but basically this tells you the relative impact each variable in the model has on player production. The value in this method is that we can see how much a coach affects a player’s production when things like age, position, teammate quality, and roster stability are held constant.

Anyway, Berri and co. ran these regressions, and they got a bunch of numbers for each coach. These numbers told us of the relative impact of each coach in their data set. The two coaches with the biggest impact were Phil Jackson and Greg Popovich, which seems entirely reasonable. In fact, the model estimated that Pop and Jackson would add over 15 wins to their teams per season, all else being equal.

The problem came when the researchers looked at the coaches’ confidence intervals. For those of you who don’t know, a confidence interval is a way of saying that an estimate falls somewhere in a certain range. I’ll explain through a metaphor. Let’s say you and I are at a bar, and I ask you to estimate the height of a guy standing across the room’s. You may say, “I think he is 6 feet tall”. That is your estimate. Then I say, “Okay, tell me the range of heights you’re 95 percent sure he falls between.” The clever thing to do would be to say, “He is somewhere between 4 feet and 8 feet tall. “ Of course, I’d push you to give me the smallest range you’re 95% confident he falls between. You may respond, “He is taller than 5 feet 8 inches, but shorter than 6 feet 4 inches.” That is a confidence interval. That is the range of heights you’re 95% sure that guy’s height falls between. In the case of this paper, the first round of statistics estimated just a single number (they guessed the guy’s height), but when they looked at the confidence interval of the effect of each coach, the low end of the best coaches overlapped with the top end of the worst coaches. To return to my bar example, this would be like I asked you to rank 20 guys in a bar by height, but when I asked you to provide a range of their heights, the tallest guys minimum height was lower than the shortest guys tallest height.

This confidence interval issue seems to justify a piece of basic economic theory, which is that managers don’t have a huge effect on individual employee production. Now, if we go back and look at the quote from the article above, do I think that overstates things a bit? Of course. First of all, this paper just examines the way coaches impact individual player production. It doesn’t and can’t measure the more intangible sides of coaching like managing egos and personalities. Maybe, that is the most important part of a manger’s job, which is pretty hard to represent numerically.

If we ignore the confidence interval stuff for just a bit, we do see some solid estimates that coaches can have a positive and negative impact on player performance. Still, that impact overall isn’t quite as great as most fans may have thought it would be. In my opinion, a more appropriate conclusion would be that coaches do have an impact on player production and team performance. However, that impact may be lower than other variable and lower than the general public perception may have expected.

 

The Counter Factual Problem 

Let’s step back from this research and turn to something I think of as the counter factual problem. The counter factual problem is basically way of saying, we can never know the outcome of a situation if there had been a different coach on the team or if that coach had made a different choice in an in game situation. In short, we can’t produce a perfect counter example to the fact of what happened.

Pretend you and I are playing tic-tac-toe. Tic-tac-toe isn’t an interesting game, because it is completely determined, which means I know the optimal strategy and move at every point in the game.  If you lose to me, we can go back in the game and determine where you made the wrong choice. Then, we can change that decision and play the game out to see if the outcome changes.

We can’t do the same for coaches. We can’t start the Pelicans season over, and have George Karl be the head coach to see if the outcome changes. In fact, it wouldn’t even tell us much, because luck or chance play such a large role in season outcomes. Furthermore, we can’t even say that we know a coach made a bad decision, because we can’t provide the counter factual. We can’t replay the game and make a different choice with lineups or strategy to see what happens.

Now, I know some of you have become frustrated. You probably think I’m telling you that you can’t critique a coach. I’m not saying that at all. You can question coach’s decisions, but you can’t know they made the wrong choice. This is probably particularly relevant for our team. The tone surrounding Monty Williams’ performance has grown particularly knowing, in my opinion. We could all benefit from remembering that we can’t know how well things would have gone if Monty played the strategies we wished he would have.

 

Conclusion

The title of this post is, Do Coaches Really Matter? That has been the driving question here, and my answer is not as much as you think. Sure, a good coach makes a difference, but you’re going to need some other things to be successful. I know someone will post in the comments that two coaches, Jackson and Red Auerbach, have won about 30% of all the NBA titles in history. Fair enough, but how many of those teams had just average talent? How many of them had only 1 all time great player? How many had 2? Also, what franchises did they coach for? Were these teams with a history of success?

In my opinion, a team’s odds of winning a championship depends on four variables.

  1. Player Talent
  2. Front Office Quality
  3. Coaching Talent
  4. Luck

 It is a combination of all of those variables that leads to championships. You can’t just isolate one thing. So yes, I agree that coaching matters, but it isn’t the only thing. Frankly, I don’t even think it is the most important thing. You’re going to need the talent, and you’re going to need a stable front office. Can you remember a time in the Hornets’ history when the talent was there, but front office instability closed a championship window? Can you remember a time when a stable front office with an excellent coach won a championship with below average talent?

The one other Monty related comment I will make before closing is realize that the Pelicans were a below average team with respect to talent, especially after the injuries. Some of our expectations for the team, as fans, may have been unreasonably inflated due to all the offseason moves. I saw at least one site that predicted our win total would be in the mid 20’s, and they assumed everyone would be healthy. Again, you can tweet #FireMonty or not, but I promise he isn’t the only issue nor is he the most important issue with this team. At least, that is what the data says.

 

Note: Mason Ginsberg is posting a fair and detailed piece on more of a micro level about Monty Williams’ performance and future with the team on Monday. Be sure to check that out as well!

 

References

Berri, David J., Michael Leeds, Eva Leeds, and Michael Mondello. “The Role of Managers in Team Performance.” International Journal of Sports Finance. 4.2 (2009): 75-93. Web. 3 Apr. 2014.

16 comments
xman20002000
xman20002000

So Pop and Jackson are examples of coaches that matter... Let' them coach a team like the Pelicans at the beginning of the season...  Personally this article is only relevant where CP3 and DWest remained with the team...  Didn't Coach have a more of a chance going to post season with a veteran team... There are far too many variables to both analyze this question or prove an hypothesis.. or prove this research question..  


Rather than hate on Monty Williams... try analyzing Jason Kidd as a Coach with a veteran team like the Nets...

PelicanSaints
PelicanSaints

It definitely matters...good coaches have to manage talent,as well as personalities....

lsucpolk
lsucpolk

Here is a story that I read once.  It's paraphrased, not quoted:

Clair Bee was a head coach when basketball was just gaining steam in popularity, and when conventional wisdom determined that the best way to shoot the ball into the hoop was with two hands.  One evening, his team's opponent was a team that shot the ball in a new way.  They used a one handed method that involved shooting the ball with the dominant hand, while supporting the ball with their non-dominant one.  The team full of one-handed shooters totally dominated Coach Bee's team that night.  Instead of stubbornly insisting that the two-handed shot was superior, Coach Bee decided to immediately teach his players the one-handed shot method, and his team continued to win games.  

This was from Coach Bob Knight's autobiography.  I believe a quality head coach can be rather impactful, and a bad one can be rather destructive.  

mateor
mateor

Specious claims in the abstract of this paper. There are no metrics that have conclusively proven Pat Riley eats, breathes or has the capacity to love/mourn. He cannot be fired, he will not show remorse, etc, xyz.

jsgrayson
jsgrayson

I think this is a fantastic article. Really good job Nick. 

The difference between winning a championship and being a middling playoff team isn't rested on the coach, but rather the players. However the difference between winning a championship and losing one? Maybe incremental differences may matter? 

Coaches do matter. I think all you have to do is watch a Pelicans game to quickly realize that any Greg Stiemsma based lineup is a flawed one. I do think we need to remember that with the research you've highlighted that the "confidence interval" is a measure in place to account for error, no? 

Ideally we'd like the interval to be larger between the average coaches best and the best coaches worst, but it still is not permanent.

I'm not well versed in economic theory especially linear regression models so I can freely admit that I had a tough time agreeing with the premise of the article. After reading it though I am at least more aware that coaching may not have the effect that sports analysts think it does. 

jmbell7
jmbell7

In all my years of playing, watching, and coaching basketball, I have found that great players make great coaches-especially in the NBA. 

If we're talking about little dribblers, sure, coaches matter the most because players haven't learned the fundamentals yet. But in the NBA, the best coach in the world won't win without great players. 

Certain coaches are better at finding the best ways to utilize those players (Popovich), but for the most part, having great players will benefit you a whole lot more than a great coach. If we get AD some help, and our young players continue to develop as individual players, you will see the team improve a whole lot more than if we just fire Monty and replace him with someone else.

Michael McNamara
Michael McNamara moderator

My simple thoughts: The question really isn't 'Do Coaches Matter', It is, "Does the 12th best coach matter that much compared to the 48th best in America?" 

When you think of other professions that are dependent on mental skills over physical ones, do we assume much seperation amongst the Top 200 or so, or don't we assume that the separation at that level will be minimal? Is the 114th best accountant going to miss a ton of things that the 18th best accountant will find? 

Once you get to a level like that in any profession (again, a mentally dependent one not physical), I don't think there is a big separation at the top. 

thouse
thouse

Great, great article.

I tend to agree that in game coaching is overstated a bit. It's a players' league. This ties in nicely with the Morey SSAC panel anecdote on random offense being the most efficient

To me, it's much more important to create a framework for talent to thrive than individual tactics; to understand broad strengths and weaknesses.

GerryV
GerryV

Yes Coaches matter! They organize and make sure development programs are taken care of.Players want to know what the plan is during the pressure late game moments of the game...Coaches provide the voice of direction and open the lines up for better communication among the players...you think a player what the burden of having to plan everything?...the data report must have been authored by an alien..i look at it as toal jive tossed like a propaganda leaflet from a plane!...Players matter..leaders matter..front office matters and COACHES are huge factors in many ways.....end of story.

504ever
504ever

Really like this piece.  

The only thing I disagree with is this: "It seems like most Pels fans think that Monty has had enough time with the team and has shown too little."  

I think it is a vocal (especially after a loss) minority that wants to fire Monty.  I think the vast majority of fans realize that talent and front office player moves are much more significant factors than coaching and, this year, luck (injuries) was a much more significant factor than coaching.  All of that seems very intuitive and supported by this article.

Nicks65
Nicks65

@xman20002000  Try reading the research that is cited. Then tell me which variables weren't included in the model. There are obviously some there, but guess what? Age (used as a proxy for playing experience) is included. That means the model accounts for veteran players on teams. Just saying, "there are far too many variables" to consider, then citing a variable that is included as an example doesn't make your point very convincing. 

Also, I'm not sure if you're accusing me of hating on Monty Williams, but if you are go back and read my second paragraph. 

Nicks65
Nicks65

@jsgrayson  I think you're right, but I'll make a couple of quick points to see it clears anything up. 

First of all, this research basically starts off with Adam Smith's whole deal about managers being interchangeable and suggests this may also be true for NBA coaches (i.e. swapping team's coaches will not impact outcomes). 

The first results show that this is not entirely true. Some coaches affected players more than others, and coaches weren't perfectly interchangeable, especially the top few coaches. Now, it did show that the difference between average coaches wasn't that great. I was fine with this result, and I don't think it was all that out of line with core economic theory, which is, quite literally, only theoretical.

The confidence interval thing was a comment made at the end of the paper. Honestly, I felt like it was thrown in there to say, Hey, it may be the case that something in the model is inflating the significance of coaches. That could be true, but I'd find it just as likely that something was left out of the model and that maybe causing a downward bias on coaches significance. Issues with data quality and omitted variables can throw wrenches in to things. I do think it can be used to check for "error", but I felt like it was a throw in comment here. Not sure I think it is really the most important thing. 

So yes, if you are in the camp of "there is a big difference between the 6th and 15th coach in the NBA" you'd like to see a wider margin in the confidence intervals and in the coefficients. I think if you accept there are a few top coaches, everybody else, and maybe once in a blue moon a terrible guy, I think you're okay with these results. 

Nicks65
Nicks65

@Michael McNamara  Take a bow Mr. McNamara. That is entirely correct. The reason I choose that particular title and that main question was because so many people (including those close to the author) have taken this research and similar research to say, "Coaches don't matter". So I wanted to play off that cursory perspective. 


Now, what you just said calls back to the mentions of Adam Smith in the paper. One of Smith's many contributions to economics was that mangers are odd to justify/understand. His reasoning was much like your accountant example. They all have pretty much the same education, experience, and tools at their disposal. There may be slight variations in each of these, but in a perfect theoretical market where all else is equal, it won't matter. That means that the difference in how a manger is increases production (or profits for a factory) between the best and the rest isn't all that great. Like you just said. 


Now, this creates a bit of a crisis for some basic economic theory, but it isn't worth mentioning here.  I think what is interesting is if we assume the same is true for NBA coaches. Most new coaches are around the same age, have the same experiences, and have the same relevant education. So we aren't likely to see a HUGE difference between the 12th or 48th best NBA coach. However, with so much competition and parity in the NBA (relative to the real business world not other professional sports leagues) could having the 8th best coach instead of the 12th best coach be enough create a big change? In other words, even if the margins of coaching success are so incredibly small, couldn't they be the difference when the outcome of a game, season, championship when the race is tight? Think about a game that your team has a 50% chance of winning. It might help to have that small increase in effectiveness. 

Nicks65
Nicks65

@GerryV  I don't think the post disagrees with you. In fact, I know I agree with you. 

The article I'm referencing says that coaches are largely interchangeable, and they don't vastly improve veterans players production on the court. This author has said other places, I don't think coaches matter much. 


My overall point was, like you said, of course they matter, but it only show up minimally in individual player production. I think they dictate a team's philosophy, the manage individual players psychology, and like you said they game plan. Furthermore, coaches aren't the only thing that matter, which has been lost on a few Pelicans fans this season. Again, we both said that. I also think development is the biggest part of a coaches job. I.e. how they develop young player. That is NOT and really CANNOT be shown in this data set. So again, we agree. 


On the other hand, the argument being made by some sports economists isn't, "Teams would be just as good without a coach at all." It is that there isn't much of a difference between the impact of great coaches and average coaches in how player actually produce on the court. So it isn't like they are suggesting teams replace coaches with an empty chair on the sidelines. 


Having said all of those things, we reached the same conclusion or, "Yes coaches matter". 

Nicks65
Nicks65

@504ever  I think you're right, but I only interact with a small portion of Pelicans fans. I put the "seems like" in there as a nod to those people and their opinions. Most people just want to be heard. 


Thanks for your comment. 

GerryV
GerryV

Fans have no idea what its like coaching an NBA team.....all that goes into it would exhaust a mere mortal....( ok,its nap time fo rme)