+44 (0)1892 512348


Dark data: You have the data, but do you need it?

  |   INDUSTRY   |   No comment

Big Data equals big opportunities for organisations of all sizes. In fact, organisations that utilise data analytics in their decision-making process have a 5-6% better output, on average, than those who rely on experience and intuition.

Lack of awareness has a part to play for this figure not being higher. The common misconception is that Data analytics is only suitable and effective at the data testing phase only (and has no role to play during any decision making process). The reality is that the technology has evolved, and those who are effectively embedding Data analytics are using the technology to look at data during the audit planning phase, prior to delivering the audit, to help make a more informed (or to help validate) a decision.

That being said, utilising data analytics doesn’t automatically equate to an improved company performance and not all companies are managing to translate their data into actionable insight. This is further compounded by the fact that the rate of data growth is exponential; it’s doubling every two years, and 90% of all existing data was produced in the last two-year window. So what is a company to do with all this data they’re accumulating?

Sophisticated Data analytics technology such as IDEA will help to analyse “Big Data” by delivering 100% data population testing on entire data sets. But how does data analytics overcome the problems caused by “Dark Data”?

Dark Data

Did you know that more data has been collected over the past two years than in the entire history of the human race (George Nice, Savannah Group), but that only 20% of the data collected by businesses in those two years is actually being used? That’s right, just 20% of the data collected, on average, is utilised by organisations in their decision-making processes. The remaining 80% remains unused; this unused or “dark data” – defined by Gartner as the “information assets organisations collect, process and store during regular business activities, but generally fail to use for other purposes” – does little more than sit in an organisation computer banks, taking up costly space; and with the IDC predicting that by 2020 we’ll be producing 1.7MB of new data every second (40ZB). That’s a lot of data going to waste!

Turning the rest into Actionable Insights

So now we’ve identified the dark data in your system, you’ll want to turn the rest, your “light data”, into actionable insights, but how exactly do you go about doing that.

Data analytics has been widely accepted as the way forward. Many however argue that one of the fundamental problems with data analytics is the fact that, by nature, it is an enterprise technology: whilst it can very often become instrumental in the day-to-day operations of running a business very few, if any, will ever get to see it in action. Because of this, it is easy for senior management to take the opinion that existing technologies ‘do it well enough’, and whilst most senior management will agree that their systems need to be modernised sooner rather than later, it’s something that almost always gets pushed back in favour of client-facing priorities. The reality is Data analytics is suitable for and being used by small one man band organisations, both junior and senior users who have very little (perhaps zero) technical knowledge.

Data analytics is also more than just capturing and storing data; it involves querying, analysing, and eventually visualising said data. A more formal definition is the art of discovering and analysing patterns, deviations and inconsistencies, and extracting other useful information in the data underlying or related to the subject matter of an audit through analysis, modelling and visualisation.

When dealing with a deluge of unstructured data from multiple sources it can become extremely challenging to deliver meaningful results through manual analysis. Even more so if you’re not sure exactly what it is you’re looking for. By utilising powerful purpose-built data analytics software, like CaseWare’s IDEA Data Analysis software, organisations are able to easily view, dissect, and visualise data from a variety of different sources into an easy-to-read format.

No Comments

Post A Comment