Aron Smetana: 5 things you need to know about big data

TechRadar: All latest feeds 5 things you need to know about big data

Introduction

Big Data SAP

Big in name and big in nature, big data is a technology buzzword that isn't going away any time soon. ABI Research thinks that big data spending on a global basis exceeded $31 billion (around £20.8 billion, or AU$40.2 billion) in 2013 and will motor on to hit $114 billion (around £76.6 billion, or AU$148 billion) in 2018, and yet there are many who are still befuddled as to what big data actually is and why it might be worth that eye-watering amount in just three years time.

1. What is it?

Big Data server

Purely and simply it's a term used to describe the huge data sets that are being produced by the digital processes and social media exchanges that are currently increasing by the bucket-load every minute of every day. It's a mish-mash of both structured, semi-structured and unstructured data that cannot be handled by regular databases or software, and instead has to be funnelled through specific analytical programs.

Processing big data is already big business and the lack of skills out there to properly use the analytics to decipher actionable insights is something that is still a very real problem, even though we are reasonably far along when it comes to understanding what big data is.

2. Solid analytics are the key

Cloud analytics

Drawing any sort of business advantage from big data means having solid analytics in place as well as the skills to use them. Possessing a successful analytics model means that your business will be able to find new correlations to solve problems, identify trends and basically make more money.

A succinct analysis of big data analytics was given by Michael Watson of the Supply Chain Quarterly, where he wrote that there are three different types of analytics: descriptive, predictive and prescriptive. Descriptive presents data in a way that lets you know what is going on in the place the data is drawn from. Predictive describes the way you can take data and make better predictions using it. Prescriptive, meanwhile, concerns using data combined with the subsequent predictions to take action that will improve business.

Marrying these together and asking the right questions at the beginning of the whole analytical process are key to making sure your model works and delivers the results you desire.

3. Internet of Things contributes a lot

Plant

Believe it or not, Gartner has a Hype Cycle for Emerging Technologies and after topping the table in 2013, big data has slipped behind another of the technology buzzwords of the moment - the Internet of Things. Even though this is the case, big data has a huge amount to benefit from when it comes to the IoT.

Cisco's conservative estimate on the IoT mentions that the number of connected devices will hit 21 billion by 2018 and the bad news for any unprepared firms is that there is going to be zettabyte upon zettabyte of data to keep a handle on. Making sense of all this data is something that is still proving to be a huge challenge and the skills gap, which we will come to in the next slide, is something that has the potential to make or break the success of the data drawn from all these "things".

4. There is still a big skills gap

Hadron Collider

Data scientists are tasked with handling the data sets on offer and producing insights from them, and for anyone looking to kick off a career the news is good: the skills gap for data scientists is currently huge. Research from Gartner found that 85% of companies on the Fortune 500 list will fail to exploit data in an effective way in 2015 and deciding how to bridge this gap is critical for enterprises to realise the benefits of big data.

Accenture, meanwhile, carried out a year-long research project on data scientists and found that the United States will create around 39,000 new jobs for analytical experts through 2015, but will only actually be able to assign candidates to 23% of those roles.

An easy way to fill in the gaps is, obviously, to train and promote from within although this depends on the right training being in place as well as having the "right" people within the organisation in the first place. The other way is education.

One such educational institution trying to help is the European Data Science Academy, which is an online platform for training data scientists across Europe. It claims that the demand for professionals with the skills to manage big data will grow by 160% by 2020 and has already secured a €2.9 million (£2.2 million, or $3.25 million) investment from the EU to run a range of courses, which get underway in late 2015. More schemes like this are also in the pipeline and they can only help to address the shortfall.

5. Rich data could be even more important

Rich data

Dr. Rado Kotorov, chief innovation officer at Information Builders, put it well when he described the difference between big data and rich data as the way that crude and refined oil differ. Rich data, in layman's terms, is what comes out when data from different systems is combined and given a context so that it becomes a practical proposition for businesses and individuals.

It solves one of the main complaints about big data in that the unstructured data produced doesn't provided a detailed enough insight compared to the level of contextualised data that is possible with rich data. The fact that rich data needs big data in order to exist means that big data will never go away, yet for the higher level of insights and benefits, rich data has to be a part of the picture.

One of the big concerns about rich data, however, will ultimately be that it is more detailed and as such could present a privacy risk to consumers that unstructured big data may not end up doing. What this could well present is a situation where consumers become even more guarded over their personal data and choose to give less up to the companies asking for it, and perhaps even use it as a bargaining tool in the future.

"In the future, people may choose to control information that they are creating and then monetise this back to companies – this may be in the form of lowering their bills or getting better service quality from one provider," admitted Matt Pfeil, Chief Customer Officer at DataStax.