When Big Data and the Internet of Things Collide

Hans Heerooms's picture
 By | December 09, 2013
in Business Intelligence, when big data and the internet of things collide, Business Intelligence
December 09, 2013

There is already a lot of discussion around Big Data, and increasingly the term 'Internet of Things' (IoT) is becoming common parlance in data management circles.

This is being driven by a massive growth in the number of sensors embedded in everything from our devices right through to the infrastructure around us and even our clothing. We are moving into a future where smart objects can sense the environment they are in, and can interact not only with their owners, but also with each other. This machine-generated data will be, in the short term, the bulk of data flowing through the Internet, and hence the leading provider for Big Data systems.

The pace of change to this new digital landscape is unprecedented: Gartner predicts that in 2020 more than 30 billion devices will be connected to the internet – compared to around 2.5 billion devices in 2009. Three years ago, the devices were mainly PC’s, tablets and phones, but in 2020 we will see far more variety, from sensors to RFID tag-based technology. The availability of this continuous information will allow us to monitor everything from energy consumption to traffic and everything in between.

This is not just a new uprising technology for technology sake: in a worldwide survey of The Economist 95 per cent of the C-level respondents expected that they would use IoT in three years’ time, while 63 per cent of them believed that companies slow in integrating IoT would fall behind.

There is also some soft evidence to suggest that Europe is leading the charge in this area. This brings a high potential to positively impact the European economy and society. As such, the European Commission is taking a vested interest in this topic, having run a public consultation and released several reports in order to inform a consistent policy stance.

As the IoT creates ever-increasing amounts of data, it also generates the opportunity to capitalise on information from this data – to the point that new businesses will rise based upon trading this information.

But this is not a trivial step to take: businesses have to handle the incoming data stream from potentially millions of distributed sources that pop up and disappear, and this incoming Big Data has to be stored and organised so that it can be used for Business Analytics to transform this data into valuable information.

I am using here the term 'Business Analytics' - rather than just the more neutral 'Analytics' - as it's only possible to derive real information from data if it's placed it in the context of business processes and models. Harnessing the IoT will provide an additional source of data for optimising the business, but it has to be related to all other information sources as data warehouses and operational applications. Accomplishing that means having the master of all master data strategies, one that incorporates data from all sources, whether machine or application or human generated.

Business Analytics is not only computing the right information, but is also about enabling the information consumer to find this data easily. Similarly, Big Data consists of both structured and unstructured information and the information consumer expects that business analytics has transparent access to both. Enterprise Search is the most common paradigm for exploring unstructured data, while more traditional Business Intelligence tools are better suited for structured data. Only those who use an information platform that is capable of merging BI and Enterprise Search across Big Data and existing sources will optimally address the needs of the future information consumer.