Drowning in Data

The IT world is all a-flutter about the potential of big data. But, beware of losing sight of the real value to your business, writes Daniel Senter.

The rate at which we are creating data is increasing at almost exponentially: mobile phone GPS, social media posts and sensors on objects collect information about almost everything while decreases in storage costs mean that the volume, variety and velocity of data we collect and store will remain on its current rapidly increasing trajectory.

A study published in 2011 by Discovery News, for instance, claimed that up to 94% of data worldwide is stored digitally, compared to just 0.8% 25 years ago!

However the question is ‘What are we doing with all this data?’

A 2011 Gartner survey highlighted corporate data is growing by around 60% each year, however 73% of those interviewed felt their competitors could use the data better. More interesting - or perhaps shocking - is that only 17% of the respondents claimed to be using more than 75% of the data they collect. So companies are capturing and storing data they are not even using!

The question to answer is WHY?

· Do they fully understand what data they are collecting?

· Do they know how to process the data - structured or unstructured?

· Is the data actually useful?

Many authors have taken the view that whilst the increase in data offers the potential to drive better decision making through business insights, there is a note of caution around this. As the volume and variety of data increases, so does the uncertainty, ambiguity and complexity, i.e. data noise. The removal or correction of data noise requires considerable thought and appropriate tools to ensure that true insights are extracted from the data.

To complicate this further another shift we are seeing within the data growth is the increase in unstructured data. By unstructured data I mean data that does not sit with in a database and exists as textual (text in documents, blogs, social media, etc) or non textual data (images, video and music). Analysing this type of data requires different tools to those traditionally used and asks us the same challenging questions above in finding the nuggets of information amongst the noise...

The effort required in filtering and processing the wealth of data we collect requires time and money, which should decrease over time.

However, the high start up investments may deter many businesses.

Business that are able to take on the challenge in digesting and processing the data will become more efficient in doing so and gain quick wins along the way. Although as seen with other specialisms, technologies will soon become available to automate and improve this activity thus closing the gap and making it more affordable to the masses.

Therefore one of the biggest challenges in the short term to stay ahead is to really understand and focus on the data that is meaningful to you. Define what you or your business value in the data being produced and then look at ways in which to capture, process and understand it.

[inlinead]