Monday, October 17, 2011

Epistemology is a Bitch

There’s an interesting article over on Backhand Shelf about choice and statistics, and for some reason Kierkegaard. It’s quite well-done, and clearly much thought-over, and although I agree with it in principle, I’m not entirely comfortable with the project it seems to be endorsing.

I’ve made a similar point about choice before, in a much longer and unnecessarily florid way, although I wouldn’t argue that statisticians neglect choice so much as they fold it into the category of ‘luck’. The decision to use the word ‘luck’ in sports stats is one of the great terminological disasters of the 21st century, the cause of numerous unnecessary semantic battles, the casualties of which still litter the fields of Hockey Blogistan whimpering about how Jaroslav Halak wasn’t just lucky, he really wasn’t. The trouble is that statistical luck isn’t the same thing as layperson luck. Layperson luck is the random, uncontrollable, unpredictable acts of the universe. Hockey stats luck is a fine hash of that randomness, choices made in response to that randomness, and myriad biological and psychological factors so case specific as to be innumerable and immeasurable. Personally, if I were a probability wonk, I would have called that category something like ‘miscellaneous’ or ‘various minor and nonpersistant factors’ or even just ‘Olga’- anything but ‘luck’, really, would have saved a lot of bloodshed.

Anyway, I’m pretty sure that the statisticians know that some choices do have some degree of impact on some outcomes. They’re big boys (and girls? Are there any girls? Please let there be some girls by now), and they’ve had this argument far more times than we have. They know that there are probably elements of choice and hustle and heart and choking and whatever else in the mix, influencing a bit of this here and that there. It’s not that they don’t believe in it exactly (although some of them really don’t), it’s that they don’t care. It’s not their project. They’re not anthropologists or psychologists (okay, that one dude is an anthropologist I think, but only in his spare time), they’re not trying to figure out why one person did one particular thing at one particular time, they’re trying to figure out the generally applicable principles according to which hockey always works, across teams and seasons. Things that happen once and only once, like the choices made by a particular player on a particular day, are not relevant to the scale of their work. So they get folded in with all the other small things, under the category of Olga.

Now, it is possible that there could be a study of the choices of hockey players that would demonstrate the long-term, large-scale impacts of certain kinds of choice-making. There have been longitudinal studies of other human populations before, looking at groups who made one choice and groups who made another and seeing, over a period of years, how that choice impacted their lives. There are a thousand wonderful hypothetical studies of hockey players that could answer a lot of questions about the role of choice vs. natural talent, what parts of the player are man and which are machine. You could study factors of disposition and temperament, training regimens, family situations, the efficacy of different methods of psychological self-discipline. You could look at correlations of IQ, EQ, executive function. And if you did those studies, and you took those findings to the statisticians, I bet you cash money they would be extremely interested in your findings.

Unfortunately, where professional hockey is concerned, those studies will never happen. There is too much money, too much power, too much hockey on the line for teams (or families, or communities) to surrender their precious precious raw materials up for double-blind studies, ethnographic observation, or psychological experiments. Any effort to use the tools of social science to really understand what kinds of choices and intangible characteristics make for better players would risk damaging them or holding back their development. The act of observing would disturb the observed, and so most of the people who make hockey would prefer to muddle through the process according to what they think is best, without knowing for certain if it is.

The problem with valuing choice, or any intangible, unquantifiable personal quality in hockey is that we do not know about these things. We know, from individual cases, that sometimes some intangible qualities can be important, but we don’t really know anything more than that. For example, we know that some players have an extraordinary work ethic, insofar as they spend far more time training and practicing than they are required to, and that some of those players are also extraordinarily good. But we don’t know how many players are putting in extra training to no avail, and how many actually damage themselves in the course of overtraining, and so we cannot really say how valuable work ethic is. We don’t know if more heart is necessarily better- as I learned in the post on choking, in other areas sometimes laissez-faire attitude can actually correlate with better performance. We have very little evidence on which to weigh these qualities against each other, and almost no idea how to weigh them against objective performance, so when we ask that they be valued, what exactly are we asking for? That they be given full explanatory power whenever we feel they ought to apply?

But then again, we don’t really know when they apply, do we? Hockey players are celebrities, we see only tiny slices of them, and most of those slices are edited and packaged by journalists and PR goons in service of their own agendas. Twenty minutes of play per game, some generic post-game interviews, the occasional cutesy get-to-know-you featurette, maybe a Twitter feed, and every now and then a scandalous escaped cell phone pic is all we actually see of a professional hockey player. When we derive our assessments of a guy’s work ethic, or his courage, or his choices, we are making an interpretation based on an artificial image of a person filtered through multiple intervening media. We do not see the man himself, we see only a fragmentary representation of him.

So while it is quite valid and probably quite true to say that player choices impact the outcomes of their seasons and careers, it is more important to remember that there are very few people close enough to the players themselves to really understand that process. Honestly, the player himself may have only a partial and confused notion of where his own choices ended and luck began, of which parts of his life he determined and which were forced upon him. Most people are not wholly aware of how their character and decisions impact their work performance, and I would assume hockey players are no more rich in intrapersonal intelligence than the average person. Someone who is truly intimate with a locker room- a trainer, a doctor- might be able to tell you a lot about the kinds of choices that have influenced the outcomes of games and careers, but that information would be the result of months and ideally years of close contact. It takes that kind of time to truly know the sort of intangible qualities a person has.

Fans cannot have that knowledge, and I call a priori bullshit on anyone who says they can see heart in 16:38 TOI and a ten-second postgame comment.

I don’t know this dude, the writer of this article, from around, but I’ve seen enough of these types of essays and comment threads- hell, I’ve written enough articles and comments in that vein myself- to smell the fear behind it. I feel it too. I live in dread of the coming of the Great Hockey Equation, because I know that the day the final formula is complete and proven is the day I will no longer be able to say anything about the game. This is my fundamental problem: I am no mathematician, but I love talking about hockey, and from time to time that conflict of interest does impel me to try to jam a shoe in the gears of the Great Calculator, in the hopes of slowing it down a little.

But the statisticians are damn near the only ones in this game who maintain any strict standards of epistemological honesty, and I’m afraid more and more that the assertion of faith in ‘intangibles’ is just an excuse for unfounded speculation on the character of people we do not know. It is a serious thing to sit in judgment of a stranger’s habits, his courage, his intelligence. His heart! My God, we go around thinking we are qualified, based on a few hours of game footage and a few bland interviews, to determine the measure of a man’s love for the game. Has it ever occurred to you, reader- and I ask this sincerely- what unforgivable hubris that is? What complete and utter fantasy most fan opinions of that kind must be? How often people see some tidbits of something that is vaguely suggestive in some way of something they don’t like and immediately connect it to an intangible flaw and the repeat it to each other so many times that it becomes the accepted wisdom? These days I cannot so much as hear of Alexei Kovalev without thinking I need to send the man a Hallmark apology and a dozen roses in whatever color is most suggestive of regret, but they don’t have a card for “I’m sorry I called you lazy and disinterested and enigmatic when in reality you’re probably just not quite as good of a player as I wanted you to be, and didn’t express yourself in the cultural idiom to which I am accustomed,” and I haven’t got the budget for roses.

Go ahead, get out the existentialist weed-whacker and clear as much ground as you like for the principle that choice matters in NHL hockey; but if you care about intellectual honesty, then that ground you clear will have to remain empty. The second you try to plant any specific ideas about which players making what choice caused what outcome, you’re sacrificing epistemological rigor for narrative satisfaction, making something up to satisfy your need to feel as though you know something you do not and cannot know.


Clare said...

Wait. Why do I have to account for everything in hockey? All this talk about the predictive value of statistics and applying Moneyball principles and how to factor in "luck" or "choice" or whatever the hell we call it these days has gotten me thinking that more and more we're marginalizing our own experiences of the game. Hockey isn't a spreadsheet. It's a narrative.

I don't watch hockey to find out about shooting percentage or Corsi. I watch hockey to see potential in action. I watch hockey because its the fucking butterfly effect on ice three times a week for nine months. Hockey is about potentiality, not predictability. All of this trying to corral it and count it and tame it is really starting to piss me off.

Give me the unpredictable, the subjective, the momentary. Stop trying to take the hope and joy and passion out of my game in the name of epistemology and predictability. Stop breaking it down into a set of data. Let me just find out each night, as it unfolds, how the story ends.

E said...

well, i don't think anyone is trying to take away your ability to enjoy the game however it comes to you. i mean, nobody else can really control that, right? the people who are interested in data and equations do it because that's the view of the game that they get off on, but if it's not what does it for you, than nobody's making you care. i never meant to say that you had to account for anything.

the conflict comes in because the sit-back-and-enjoy-the-unfolding-of-the-moment is rarely just that. people who watch the games for nothing more than the visceral pleasure of watching still, after a time, form very strong opinions about underlying truths in the game- what skills are most useful, which players are the best, what off-season moves the team ought to make, etc. the thing is, those questions are not entirely subjective, there is also objective data that comes to bear on them, and if one cares about finding better answers, then one has to contend with that data and what it finds.

although i'm no statistician, this blog has been unabashedly analytical from the beginning, and as such i feel a duty to consider all perspectives on the game, and i've found that the contentions coming out of the advanced stats crowd right now are far more original, challenging, and thought-provoking than the standard-issue narratives. that's not to say that i always agree with them, but i honestly feel as though the narrative side of hockey has gotten very rote indeed, formulaic even, as though it feels there is nothing left to learn. there's a hell of a lot left to learn.

as i said in the paragraph about fear, i know exactly what you mean about 'taking the hope and joy and passion' out of the game. the folks over at arctic insist many times over that their lens doesn't take away any of the thrills for them, but for me it does. it's a problem i'm still wrestling with.

Clare said...

I think that perhaps I ask questions about hockey that hockey statistics as they currently stand just can't answer. I want to know how a player learns to "think" the game. Can that be taught or is it inherent? If it's inherent, how do you recognize and develop it in a kid that has it? I want to know what "leadership" is and how it brings a team together. Why is hockey so central to Canadian self-image? Exactly how did that happen and what does that mean as the sport gets increasingly international? I spend very little of my time figuring out whether a player shoots enough or takes more offensive zone starts, and a lot more time watching to see if they talk to each other on the ice or at what point they've figured out what their teammates are likely to try to do. It's not that there's no statistical component to those things. It's just that the stats illuminate only one part of them.

E said...

true, there are plenty of questions that the stats folk can't answer and, generally, i don't think they try to- i haven't seen a whole lot of discussion of hockey and canadian identity over at behindthenet. so i think there's plenty of room for separate spheres and mutual respect between them. every discipline has some questions it is uniquely equipped to answer and others it cannot hope to address, no?

to take one of your examples, though, i do think that when considering an intangible like leadership, one benefits from considering the questions stats people would pose: how do you measure it? if you can't measure it, where do you see evidence of it? how do you separate that evidence out from confounding factors? is it enough for people to feel it or does it have to have concrete effects on the ice? etc, etc. you don't need to believe what the statisticians say, but considering those type of questions (which are different from the kind of questions about leadership that narrative often poses) is going to enrich your thinking.

Clare said...

I really don't mean to say that statistics have zero place in the way I look at the game. They do illuminate issues and point out new questions to ask. They are important and valuable tools. But they're still just tools.

My real problem is that it seems like right now the _only_ thing we are talking about is statistics, and not just player stats or game evaluation stats (such as shots against or whatever). We've gotten to the point that in some places in the hockey world, every experience gets a stat to define it. We are so bound up in trying to predict accurately that we have stopped actually looking at the game.

Eliotte Friedman wrote a piece the other day about the "10-game rule" in which he said that "If your team is two wins out of the playoffs at the beginning of November, the three-point games give it a seven per cent chance of recovery." Wow. Just wow, I thought. Eliotte Friedman just essentially said that for roughly one half to one third of the teams in the league only the first ten games of the season matter. Why even bother? I happen to be a fan of two teams that are not doing very well this month, the Predators and the Lightning. This 7% statistic has been endlessly repeated within these fan bases, always with the edge of panic and doom. If we don't win RIGHT NOW, it's all over. The season is completely finished. It has a number now, so it must be true.

But I have real questions. For one, he looked at 6 seasons. With that small a sample size, one outlier will move that mean. Second, if 3 teams that are out of playoff contention on Nov 1 make the playoffs every year, who do they replace? Third, what about the standings on Dec 1, or Jan 1? Are there any other periods of the year that correlate like that? If so, can't we say that all games are critical? Do divisions matter? Do schedules matter?

But more than that, why do we let a number dictate how we feel emotionally about our team? Sports fans have always been hopers and dreamers. That's why we love the Cinderella stories. We've cheered in the past, before statistics, when our teams lose. Why does 7% change that? Why do people allow one new number to kill off their hope? When did we get so wrapped up in the predictive value of data that we forgot how to cheer for a team that may still not achieve its goals. It's never been guaranteed before. Why are we trying to guarantee it now?

And at this point I'm just babbling. Let me simply repeat that I don't disregard statistics and how they can inform our understanding of the sport. I just believe that a corrective is necessary to remind people that numbers aren't the whole of our experience, because a lot of people seem to be forgetting that.

Clare said...

Oooh. I jsut figured out what I'm trying to say. The same way that an over-reliance on technique can fail a goaltender, an over-reliance on statistics can fail the fan. Use them but know when to set them asiee.

E said...

do you have a link to this freidman article? i'd love to see how he derived that figure. if it's just based on averaging out results since the lockout, than it's basically got as much credibility as the suggestion that it's better to come in 2nd last rather than last, because the 2nd last team has won the draft lottery more frequently so far.

it's rather peculiar the way fans react to statistical ideas, emotionally speaking. i mean, whatever stats prove or disprove, they don't tell anyone how to feel about anything. a few of their more overzealous partisans have been know to do so, but that's mostly due to individual contrariness rather than what the data shows. but many people- myself included- really feel like data-based conclusions have a radically difficult 'meaning' than conclusions derived from other types of reasoning, and react to them in a different emotional key. really, the whole culture of the advanced-stats/anti-advanced-stats debate needs a lot of unpacking, doesn't it?

Clare said...

And for funsies, the verification word for this post was "outch." That made me laugh for some reason.