Source: Edinburgh Business School, Heriot-Watt University. Big Data… Making better data-driven decisions!1
‘Big data’ is a term that’s been in common usage for nearly 10 years, yet we still talk about it as though it’s something other organisations are doing. There is a growing body of case studies of (mainly large) organisations that are managing to show how they are sifting through, analysing and acting upon large datasets. In many cases the examples of cutting-edge use are coming from tech-savvy data-centric organisations such as Google and Facebook. But what about the vast majority of organisations that do not consider themselves technology companies but that use and depend on technology to support their primary business function? These organisations exist across all sectors of industry.
On the one hand, organizations such as the OECD predict a significant growth in demand for data handling skills, which is in the region of a 25–30% increase in jobs for data specialists over the next five years – more than double the expected increase in IT related jobs alone. On the other hand, many organizations struggle with the basics of data analysis. While technology advances have created the need for data specialists, this focus on role specialisation does not address the fundamental question: ‘What data do I need to analyse in order to be competitive?’ This is at the heart of a potential opportunity, and threat, for organizations. Those that quickly understand the data landscape that defines their competitive environment will be able to identify and target specific data sets for analysis. Having someone within your organisation who can work with Hadoop, R or Cassandra will not be enough… from a business perspective you will need to understand the type and source of data in order to understand what’s happening and shaping the competitive environment.
So, what ‘new’ skills or insights does the key decision-maker need in order to keep in step with a dynamic, fast-moving market? The good news is that the skills have been around for a long time; the bad news, for many, is that they are numeracy-based. A decline in numeracy skills over time has left many professional managers nervous around numbers. However, you don’t need to get your old maths books down from the attic just yet. In order to start making some sense of the data flowing across your desk, and which you are increasingly being expected to use to make a decision, there are some basic questions you can ask that will improve the chances of making the right decision. Thomas Davenport (HBR Insight Centre, 2014) articulates these as follows:
- What can you tell about the source of data you used in your analysis?
- Are you sure the sample data are representative of the population?
- Are there any outliers in your data distribution?
- How did they affect the results?
- What assumptions are behind your analysis?
- Are there any conditions that would make your assumptions invalid?
As you can see, the questions focus on understanding the origin, context and relevance of any data presented to you. Before analysing or acting on these data, you must first have some idea as to their relevance and the biases that shape them. Once you’re sure the data are representative of the population you are looking to understand, you can start to think about the right way to analyse them. If you don’t do this, you could be basing your data-driven decisions on ‘rubbish’ data. So, if you consider your data against the questions posed above before rushing to analyse them, you could avoid a costly error.
You don’t need to be a maths guru to become a better data-driven decision maker – just keep asking these questions. When you’re happy with the answers, then go ahead and get the data analysed.