Maximizing Potential of IoT Data for Enriched Analytics

In most IoT solutions, device data is collected continuously as streams and persisted into a data lake (or a data storage for simplicity of discussion); what happens next? Usually, a data scientist picks up that data, analyzes for patterns, discovers insights for actions, and builds models to predict future behavior of the devices.

What happens if the data is not rich enough for analysis? It leads to sparse patterns, less meaningful insights, and weak predictions. That exactly is often the problem being experienced by the data scientists in IoT space. To address the sparsity challenge, data scientists have to coordinate/collaborate closely with domain experts and/or learn the domain themselves. While the former adds considerable cycles of time and human efforts across disparate organizational functions, the latter can make data science experts lose valuable time and add frustration (besides, domain knowledge is something that is earned by field experts over many years).

So, is there a better way to solve the problem?

Yes, and that is by deploying the real-time streaming analytics that can:

• Contextualize the stream data with the device’s or system’s meta data
• Correlate with ecosystem data; example: 2P and 3P data
• Annotate the data with thresholds and other meta data
• Label the raw data with outcome metrics based on domain expertise
• Compute derived metrics from raw metrics, leading to richer set of features

And, all this is possible in real-time during processing and analytics of data streams (either at the Edge or on the Cloud). That is, allow the domain expert to interact with live data in motion via simple-to-use interactive technologies (complicated interfaces and complex programmer environments limit the domain user’s intent and interactions).

The domain expertise based enrichment of live data in motion adds enormous value, and when persisted into storage, it can further be used by data scientists for greater impact.