The Tyranny of Metrics

One of the great themes of Aristotle’s work on ethics and politics is the need for human judgment. In the Politics, when he describes the virtues that the “master craftsman” (architekton) of the state must have, chief among them is practical wisdom (phronesis). In the Ethics he points out that no matter how carefully laws are written, they will always be incomplete by virtue of their generality — their relevance to a given case will always have to be determined by judges, judges who therefore need to possess the virtue of equity (epieikeia) in their decision-making: that is, the ability to decide, with tact and shrewdness, just how the law should be applied in a given case.

There are few things that the Modern Moral Order despises more than human judgment. One could argue that the chief energies of the MMO have been devoted to the elimination of such judgment, to render phronesis and epieikeia wholly unnecessary. What drives the MMO is what Taylor calls “code fetishism” or “normolatry.” In our time, one of the primary manifestations of code fetishism is, in the title of Jerry Z. Muller’s important new book, The Tyranny of Metrics. From the Introduction:

Schemes of measured performance are deceptively attractive because they often “prove” themselves by spotting the most egregious cases of error or neglect, but are then applied to all cases. Tools appropriate for discovering real misconduct become tools for measuring all performance. The initial findings of performance measurement may lead poor performers to improve, or to drop out of the market. But in many cases, the extension of standardized measurement may be of diminishing utility, or even counterproductive — sliding from sensible solutions to metric madness. Above all, measurement may become counterproductive when it tries to measure the unmeasurable and quantify the unquantifiable.

Concrete interests of power, money, and status are at stake. Metric fixation leads to a diversion of resources away from frontline producers toward managers, administrators, and those who gather and manipulate data.

When metrics are used by managers as a tool to control professionals, it often creates a tension between the managers who seek to measure and reward performance, and the ethos of the professionals (doctors, nurses, policemen, teachers, professors, etc.). The professional ethos is based on mastery of a body of specialized knowledge acquired through an extended process of education and training; autonomy and control over work; an identification with one’s professional group and a sense of responsibility toward colleagues; a high valuation of intrinsic rewards; and a commitment to the interests of clients above considerations of cost. 

It is noteworthy — and from where I sit very interesting — that Muller came to write this book because of his experience as the chair of an academic department. Much of a department chair’s job in the American academy today involves manipulating the metrics of assessing “learning outcomes” — as described in this essay by Molly Worthen. (There are advocates for more nuanced and humane models of assessment — Kate Drezek McConnell, for instance — but if you’re a professor and you get to deal with someone who thinks the way McConnell does, you’re very lucky.)

Of course, the reign of metrics extends far beyond the academy. Muller shows it at work in law enforcement — How many arrests is a police department making in relation to what the metrics say the number should be? Is the DA’s office meeting its expected conviction rate? — and in medicine — Hey surgeons, don’t take on difficult cases that might lower your success rate. And I vividly recall the moment several years ago when the gifted designer Douglas Bowman left Google because he wasn’t allowed to design, only oversee A/B testing.

Where does metrics succeed? Among other places, in sports. The analytics revolution has affected almost all sports, and has been wonderfully illuminating. Sometimes advanced analytics tells you that what you believed all along is indeed correct — there are no analytical models of basketball success that don’t put Michael Jordan at the top of the heap — and sometimes you discover that your observations of the game have led you to dramatically overrate some players and underrate others. (The latter discoveries are especially fun.) But all sports are, in one way or another, counting games: you count wins and losses, and count the actions that lead to wins and losses: made and missed shots, strikeouts, completed passes, unforced errors, and so on.

You can sort much of the rest of life that way if you want, I suppose. For instance, in evaluating the design of a website you can ignore such fuzzy notions as “beauty” and simply count the number of clicks associated with various shades of blue. (That’s why Bowman left Google.) You can “teach to the test,” ignoring every aspect of education except the ones that produce higher test scores — and if your job depends on your students’ test scores, teaching to the test is what you’d damn well better do.

And wherever it’s possible to make the metrics better, we should. Something that is not measurable now may become at least partially measurable in the future. The problem is not the use of metrics, it’s the tyranny of metrics. And perhaps the worst consequence of that tyranny is its tendency to make us give up altogether on the cultivation of judgment — of phronesis and epieikeia. Mistrusting judgment, believing that it can never be accurate, our technocracy figures that using whatever metrics we have — and torquing our questions and thoughts and concerns in the direction of existing techniques of measurement and assessment — is the best available option. The fear is that human judgment will never be anything more than emotionally-driven opinion. And you know what? Untrained judgment always will be emotionally-driven opinion. This is what we call self-fulfilling prophecy.

Text Patterns

March 16, 2018