There is a scene in Moneyball, the 2011 film based on Michael Lewis’ novel, that captures a critical tipping point in how we understand the world around us.
Moneyball tells the story of Billy Beane, general manager of the Oakland Athletics. Beane pushed aside decades of conventional baseball wisdom in favor of analytics, found a competitive edge, and turned his low-budget baseball team into a championship contender. Relying on data analysis to make decisions might sound commonplace today, but the scene below shows just how revolutionary Beane’s approach was:
How ridiculous does Grady, the scout airing his grievances, sound? You don’t put a team together with a computer … It’s not science … You and ‘Google boy’ … There are intangibles that only baseball people understand!
Grady sounds like he is from the stone age – I can barely tell if I am listening to a MLB scout or a character from the Flintstones. In today’s age of “big data,” it’s mind-boggling to consider that multi-million dollar decisions were shaped by this kind of thinking just 15 years ago.
Moneyball marked the beginning of a seismic shift in how we make decisions that has stretched well beyond the world of sports. The so-called Moneyball approach – relying on data and analytics over intuition and experience – has transformed criminal justice, healthcare, music, government, and business. You name an industry, there’s a “Moneyball” for that. I can’t even pick a taco stand without crunching the numbers on Yelp. We define truth not just by what we see – we define it by what we can analyze and support with statistics. Moneyball is a symbol for the age of data, and a fundamental shift in how humans “know” what they know.
But what if we have gone too far?
I recently came across an editorial from 1990 titled Many Ways of Knowing, by Ann Hartman. Hartman, a leading practitioner, educator, and researcher in the field of social work, explains how no one method of research, of learning, can capture all there is to know:
We need large-scale studies in which variables can be reduced to measurable units and the results translated into the language of statistical significance. We need in-depth “thick descriptions,” grounded in context, of a single case, a single instance, or even a brief exchange. For example, large-scale studies of trends in marriage today furnish helpful information about a rapidly changing social institution. But getting inside one marriage, as in “Who’s Afraid of Virginia Wolfe?” richly displays the complexities of one marriage, leading us to new insights about the pain, the joys, the expectations, the disappointments, the intimacy, and the ultimate aloneness in relationships. Both the scientific and the artistic methods provide us with ways of knowing… There are indeed many ways of knowing and many kinds of knowers…
Some seekers of truth may take a path that demands distance and objectivity, whereas others rely on personal and empathic knowledge. Some will find the validation of their findings through statistical analysis and probability tests. Others will find it through the intensity and authenticity of “being there” … We must not turn our backs on any opportunities to enhance our knowledge… No one way of knowing can explore this vast and varied territory.
The Moneyball revolution was born from an over-reliance on intuition and first-hand experience over objective analysis and data. Billy Beane carefully reviewed all the information at his disposal, saw this imbalance, and exploited it. The world has followed suit in emphasizing data and analytics at the expense of more traditionally qualitative evidence. We need to be careful, however, not to abandon intuition and first-hand experience all together. The data is not always right – just ask the Atlanta Falcons (too soon?) or Hillary Clinton (yep, definitely too soon).
The best outcomes are often reached by finding that difficult balance between data and experience, analytics and instinct. Top venture capitalists like hideous-cowboy-shirt-wearing Chris Sacca don’t pick investments just based on balance sheets and financial statements; they need to meet the team and trust the company’s founder before they make a decision. High-stakes investors like Sacca realize that the numbers don’t tell the full story, and they are not alone. A comprehensive review of academic research on intuition and decision making, published in the Academy of Management Review, stated that:
…effective managers do not have the luxury of choosing between analysis and intuition—real expertise involves the use of both types of decision making… the ability to switch between “habits of mind” and “active thinking” is the ultimate skill…
While we may be able to debate what the “ultimate skill” really is (have you seen this? Or this?), the takeaway is clear. Analytics is an immensely valuable tool for making meaning of the world, but this tool is not a silver bullet.
This is not an attempt to discredit data or dismiss science. Throwing analytics out the window for being imperfect is, well, fucking stupid. This is a reminder to think critically and consider all the evidence available, whether that evidence is a statistic or something more intangible. The true lesson of Moneyball was not the importance of data or on base percentage. In reality, the Moneyball approach is to value all available information, even if it is not the accepted truth, and be bold enough to reach your own conclusions. No one way of knowing is enough.