The business postures and related underneath architectures of data platforms have been evolving over the years: from data warehouses to data lakes to data lakehouses. This evolution is driven by the increasing demand for data-driven decision making and the need to effectively manage and analyze large volumes of structured and unstructured data in real-time. As data systems will continue to evolve to meet the ever changing needs of varied stakeholders, and as possibilities and choices can be overwhelming, there are a few simple principles / guiding factors to build a data platform and drive its successful adoption and usage.

1. CUTD (Come Use Transformed Data) is a critical principle to start with. Provide readily useful and usable data right out of the box to users. Useful as in relevant to the user’s business needs and use cases. Usable that is already processed and transformed into consumable format(s) as expected by users; this would also require providing the right tools and interfaces for users to gather insights (via queries, analytics, dashboards, ML, etc.) with minimal friction for consumption. With increasingly low-cost storage and on-demand compute oncloud, enterprises are collecting anything and everything which is already leading to a data deluge. A data platform’s investment in offering transformed data for readily harnessing insights will help build data gravity and even convert early majority customers as quickly as early adopters.
2. DYOT (Do Your Own Transforms): Provide data warehousing tools for power users to perform their own data operations & transformations and so allow creating new datasets suitable for their business use cases. Again, ensuring the tools and interfaces offer minimal UX friction for data processing would augment the stickiness for early users and further help attract critical mass of new users.
3. BYOD (Bring Your Own Data): Provide tools, connectors, and interfaces for users to easily and securely bring in their own data, of course within the governance and compliance guardrails of the platform. Combined BYOD and DYOT would become a significantly attractive functionality for majority users who prefer leveraging benefits from enriching their data with data shared by other participants; the platform can experience rapid growth due to network effects, shared value among participants via market place, and opportunities for participants to capture higher value via monetization of data products.

The fourth principle, BYOT (Bring Your Own Tools), depends on the business purpose of the platform. Is it a closed platform, meant for non-commercial purposes, operated as a walled-garden to be used by internal users with secured access to mostly confidential and restricted data? Or is it an open-platform, meant for Customers and Partners and other stakeholders allowed to build monetized data products on top?

A closed platform’s approach of integration, consolidation, and standardization of experience for all users within the company makes sense for internal purposes of enterprise productivity and efficiencies. Also, a closed platform’s posture usually is to discourage users from taking data out into their local systems to avoid governance and compliance nightmare.

In contrast, an open platform can take an approach of diversification and BYOT enablement (via connectors and open standards for customers and partners to bring and connect and port data into their own choice of warehouse instances, ML tools, visual layers, etc.). This makes sense for a commercial market for a few reasons:

• The open platform approach to tools/interfaces is particularly helpful to inducing external complementary innovation; the stakeholder (Customers/Partners) developers can use the data and metadata that they can access through the open interface to build compatible and complementary innovation; such opening of the platform for innovation would draw on a wider set of accessible external capabilities as well as independent experimentation;
  • This open approach to interfaces/tools would be a good way for the platform to make the journey from a transactional platform (i.e., an analyst discovers a specific business insight) to a hybrid platform by adding innovation capabilities (i.e., developers discover models independently to persist back and foster more engagement and transactions);
  • This also paves the path for the platform to attract more adoption, higher engagement, and so better monetization, forming a virtuous cycle.
• And to meet that open platform approach, there cannot be only one tool for one functionality that fits the needs of all Customers, dictated by the market factors such as:
  • Customers/Partners come in different sizes, shapes, and maturity of their digital transformation journey; their diverse economic investments/constraints and spectrum of talent / skill sets would invariably lead to the need to support a wide menu of tools;
  • Each stakeholder’s own multi-cloud strategy would also invariably require the platform to support same functionality but with a variety of equivalent tools across different clouds.

Overall, an open platform has to make it frictionless for the stakeholders to take their own data into their own managed environments for further analysis, discovery, and innovation.

Besides the above principles, any platform at scale should focus simultaneously on some critical functional and non-functional elements for its sustained success. We shall cover those elements in the next part.

In most IoT solutions, device data is collected continuously as streams and persisted into a data lake (or a data storage for simplicity of discussion); what happens next? Usually, a data scientist picks up that data, analyzes for patterns, discovers insights for actions, and builds models to predict future behavior of the devices.

What happens if the data is not rich enough for analysis? It leads to sparse patterns, less meaningful insights, and weak predictions. That exactly is often the problem being experienced by the data scientists in IoT space. To address the sparsity challenge, data scientists have to coordinate/collaborate closely with domain experts and/or learn the domain themselves. While the former adds considerable cycles of time and human efforts across disparate organizational functions, the latter can make data science experts lose valuable time and add frustration (besides, domain knowledge is something that is earned by field experts over many years).

So, is there a better way to solve the problem?

Yes, and that is by deploying the real-time streaming analytics that can:

• Contextualize the stream data with the device’s or system’s meta data
• Correlate with ecosystem data; example: 2P and 3P data
• Annotate the data with thresholds and other meta data
• Label the raw data with outcome metrics based on domain expertise
• Compute derived metrics from raw metrics, leading to richer set of features

And, all this is possible in real-time during processing and analytics of data streams (either at the Edge or on the Cloud). That is, allow the domain expert to interact with live data in motion via simple-to-use interactive technologies (complicated interfaces and complex programmer environments limit the domain user’s intent and interactions).

The domain expertise based enrichment of live data in motion adds enormous value, and when persisted into storage, it can further be used by data scientists for greater impact.

We talked about IoT being the connection of the physical and the digital worlds. That is, connecting those things that were physical in nature hitherto and now find a need to be connected to the digital world. This phenomenon about things/objects/entities is also influencing the enterprises in how they are transforming and shaping themselves to survive and thrive in the fast evolving world.

The enterprises across consumer, commercial, public, and industrial sectors that were born in the pre-Internet era (Honeywell, ABB, GE, Philips, Siemens, and so on) are making moves to position themselves as digitally transformed companies. More subtle are the moves being made by the Internet era companies (Google, Amazon, etc.) to integrate themselves with the physical world. Just as the enterprises from physical world have come to realize that they cannot compete unless digital technologies are leveraged to deliver value added products and services on top of physical assets (sometimes even in a freemium model with large physical assets being given free), the digital companies too realize that they cannot continue to enrich the end consumer’s life only through pure software products/services. They recognize the need to blur the line by playing in the field of physical objects that humans touch every day (for instance, besides the much publicized Nest and self-driving car, it can be interesting for you to observe how many of the Alphabet companies are associated with the physical world). Even more interesting are the approaches of enterprises that are born to build physical assets in the digital era (Tesla, for example).

In tandem with the above phenomenon, there are a number of plays across IoT market that numerous enterprises see the opportunity to position themselves. (First, a disclaimer: the below is by no means a comprehensive list of areas or players and neither an endorsement of any enterprise; large dedicated teams of expert market analysts exist across organizations that spend full-time analyzing these areas and enterprises to enable M&As or partnerships or commercial relationships).

Semiconductor Chips: the physical assets have to be added intelligence to perform one or more of sense, connect, store, transmit, and compute functions. Such functionality can be added with semiconductor hardware, and so it became imperative for chip manufacturers (Intel, Nvidia, ARM, Qualcomm, Broadcom, etc.) to partner with various OEMs of the install base. Another angle for them also is the increase of complex compute needs in data centers and on clouds as billions of more devices pump data.

Install base of Physical Things: enterprises across consumer, commercial, public, and industrial sectors with their traditional install base are already in the game, and are now looking to wrest control of the majority pie with deeper integration across the value chain (for example by building the platforms and the value added software services).

Connectivity: Network service providers (AT&T, Verizon, Sprint, etc.); larger the install base better for them with more subscriptions, and so prefer as many physical assets as possible connected natively to the Internet, rather than to a local Edge device. Companies like Cisco (with Jasper offering) also have partnerships with network service providers in cellular connectivity play.

Platform: most misrepresented piece in the IoT space. An Edge device on premise (home, building, plant, etc.), with ability to receive data from sensors/devices, to store and process the data, and then to transmit command and control signals to those end points, is often positioned as a platform; sometimes referred as Edge Platform (Cisco, Dell, etc. play in this space). And, even a smart thermostat is referred occasionally as an IoT platform. And then there are large scale PaaS platforms on the cloud that enable end-to-end (connectivity to Commercialization with all things data in between). GE Predix, Honeywell Sentience, etc. fall under this category (more on the evolution of such cloud platforms in later months). So, your definition of an IoT platform can vary depending on where you are standing.

Cloud: service providers (Azure, AWS, Google Cloud, etc.) offer the infrastructure (IaaS), and they also often offer some out-of-box IoT specific services (PaaS) along with IaaS. To enable IoT platforms on the cloud, there is also a range of technology and commercial plays (Docker Swarm, Kubernetes, Pivotal Cloud Foundry, etc.) to make the platforms compatible or portable to multiple cloud service providers in order to serve expanded markets and geographies.

Security: cuts across all aspects of IoT, starting from securing the local devices to protecting the data and services on cloud, while also ensuring access to authorized people and systems. The security play has different implications, both from technology and commercial perspectives, for consumer IoT vs. industrial IoT solutions with the latter demanding far more stringent cybersecurity needs. Numerous enterprises are making mark in Identity management, device management/authentication, data encryption, secured data access, etc., both on the Edge and the cloud.

Data: its lifecycle management including ingestion, engineering, exploration, quality/integrity assurance & management, governance & compliance, storage, access at scale, and so on, within itself offers an area of unlimited opportunities.

Analytics: plethora of technologies for data discovery, machine learning, deep learning, AI, NLP, NLU, NLG, Multimedia (audio, video, image) analytics, real-time streaming analytics at industrial scale, augmented reality, virtual reality, self-service, visualization, and so on play a critical role in IoT. Numerous startups are emerging in these areas.

Beyond all these, a wide range of technologies that are significantly impacting the IoT include Block Chain, Drones, Robotics, and Biometrics, The list keeps increasing.

And, many commercial areas that influence and shape the IoT world include, System Integrators (SIs), Value-added Resellers (VARs), Commercial Service Providers, Data Aggregators (2nd party and 3rd party data providers), etc. Also include the enterprises that focus on building and delivering pointed IoT solutions embedding processes, methodologies, and practices for a specific domain (buildings, factories, electric grids, agriculture, infrastructure, transportation, oil & gas, etc.).

Global IoT spending is expected to reach a total nearly $1.4 trillion by 2021, per IDC spending guide, as enterprises continue to invest in the hardware, software, services, and connectivity that enable the IoT.

If so, what areas and which players do you think would derive or capture the most value?

In conversations with family and friends, I have come to realize the benefit for broader audience in explaining some foundational concepts on “Internet of Things” (IoT), before delving any deeper. So, this appetizer is for the IoT starters. IoT Ninjas: while this one may not appeal to your taste, upcoming servings will do. Stay tuned.

Internet of Things, as you may have heard, is about connecting “things” to a central entity to gather data and extract insights for actions. There are several concepts, some simple and some complex, hidden in that definition.

First, the connection of things in IoT implies the connecting of two worlds, physical and digital, that have been traditionally disconnected. Ha…sounds like a cliché again. Right? Well, in simple words, it means connecting tangible physical things that were hitherto not instrumented to generate data and/or to receive feedback insights. That is, the “things” in IoT are primarily not the computers, laptops, tablets, smart phones, or other items that were born digital, but rather are items that were first born physical and now have use cases to turn them into digital: daily items we see at home such as thermostats, A/C machines, washers, dryers, toasters and other kitchenware, watches and other wearables, water fixtures, toilets, etc. The list is endless (of course, it is a separate discussion as to the value of connecting a physical entity to the digital world and the willingness of consumer to pay for insights and recommended actions). More than the consumer world, the real impact with IoT is already being realized in the industrial and commercial worlds. Most of the physical things in the manufacturing, mining, oil & gas, and other heavy industries are mechanical or electrical components or a combination of both; same with physical things in city and transportation infrastructure, utilities, and other public/commercial sectors. Rarely were these items invented and instrumented for data. Now being made possible with emerging technologies, harnessing data from these things will elevate one or more of safety, reliability, security, productivity, efficiency, and optimization for an individual and/or enterprise and/or the society as a whole. More on that later.

Now, what makes an IoT solution complete? Connect things, Collect data, Compute analytics, Consume insights and/or Command systems, Control things, and Commercialize solution. The first mile (Connect) is paramount for IoT and the last Mile (control) augments IoT. Without these two, it becomes a traditional data & analytics play. On the other hand, without the data & analytics pieces, connecting things does not hold much value. That is, for IoT data is the blood, connection and collection is the heart, and the analytics is the brain. All pieces have to come together to make the IoT solution meaningful.

So, what does connecting things to a central entity mean? For simplicity sake, there are two primary variations:

1. Connect multiple sensor devices deployed at a facility to a central computing node (referred to as Edge in IoT parlance) that is still local to the facility. For example, an instrumented light bulb most likely will not have computing infrastructure and so will transmit data to an Edge node in the building. The Edge can collect data from all such light bulbs, compute analytics for insights, and send commands back to light bulb (or its controller).
2. Connect more complex things to a much larger central computing entity such as a cloud platform or a data center. In this case, the “complex thing” can be a combination of sensor, edge, and gateway software, all three embedded into one device such as smart thermostat. Or it can be a combination of edge and gateway software in one device, while several sensors are deployed separately across a large facility. In that case, on one side the edge talks to the multiple sensors that do not have native connectivity to the cloud directly (as in above # 1), and the gateway talks to the cloud on the other side.

Another distinction to note in IoT is the approach in computing analytics for insights. It is an alloy of operations technology (OT) computing paradigm supported by the information technology (IT) systems. IoT is primarily about extracting insights from thing data and acting on those insights before the value diminishes rapidly in time. So, primary focus of IoT is (near) real-time operations in manufacturing plants, buildings, utilities, physical infrastructure, etc. The analytics requires the compute footprint and configuration to handle in runtime (hundreds of) millions of data events per second generated by millions of connected things.

Stay tuned for more discussion about the IoT Connect and Compute Platforms, and their on-going evolution for greater business value.

A lot of knowledge has been shared on this topic on various forums. And, a significant body of literature has been created by world class experts and researchers on leadership and product management. And yet, I believe there is always something new to share, especially insights gained from my own struggles and failures in product management (summarized through recent reflections as I transition into a product management leadership role at a much larger global company).

And so, here we go…..I have come to realize the benefit of remembering and practicing the following key principles to succeed as a product manager aka product owner:

There is no such thing as a perfect product: First and foremost to remember is the hard truth that, no matter how well you define the functionality, build the product with a great UI, and deliver a great user experience, you will always fall short of 100% customer expectations. No matter how best you do, you will receive some negative feedback. First, customers’ product assessment can be influenced by various biases, including the Negativity Bias. Second, a single product can never solve hundred use cases from hundred different customers. So, the key is to take the feedback in stride, evaluate the requirements objectively, prioritize ruthlessly, and keep iterating to continuously enhance the product.

You need not and will not have all the answers by yourself: The one misconception I used to primarily suffer from in the early stages of my PM career was the need to go find all the answers by myself. And, that created a lot of self-induced pressure. The truth is, there is a reason the stakeholder ecosystem around you will have subject matter experts for each piece of the puzzle, called the product, that you eventually have to put together. It is not your job to always find the answer by yourself, but rather it is your job to gather the relevant experts into the room, huddle with them, foster-mediate-guide the debate among them to a conclusion. Be it the best technology component to use, or best UI design, or best pricing/subscription model, or best marketing/messaging for the product, always leverage the relevant experts.

Situational leadership: You will be tempted to drive all the meetings all the time. You want to drive every discussion on the technology architecture; you want to be front driving the UI discussions; and you want to be in control of everything. Yes, the PM owns the product, and in some organizations the PM is expected to drive and control everything related to the product. I disagree. My dictum is simple: “the more you try to control, the more you lose control”. You have to understand well your own as well as others’ core competencies, and realize how to leverage those differently depending on the context. So, as a PM you should sometimes let the other expert drive, while you can be the front seat passenger navigating the route. And, when you are in that front seat, relax somewhat, focus more on the navigation and speed and less on whether the driver is using one foot or two feet to control the brake and the accelerator!

If you have read this, then you must be a PM or an aspiring PM candidate. Share your experiences and arguments!

More often than not, the analytics users and developers focus on answering “what” questions: “what happened?” or “what is happening?”, depending on the nature of data being handled, be it historical or real-time streaming. Regardless of the temporal aspect, the “what” analysis provides only a small part of a bigger story. Don’t get me wrong; the “what” analysis is a much required foundation for further deeper analytics. But as a business analytics professional, if you provide insights to your key stakeholders (executives, operations managers, etc.) from only the “what” analysis, you are shooting yourself in the foot!

So, what would complete the analytics story?

A primary question that has to be addressed by the analytics for creating greater business value is: “How much impact is there on my key business outcomes?”. The business outcomes for the impact analysis can be financial metrics such as revenue and costs, and qualitative yet valuable metrics such as the customer experience and brand value.

For example, if your job is in product analytics, it is not enough to measure the “what has been happening” or “what happened” trends of product awareness, trial rate, conversion, purchase rate (first or repeat), etc. It is also not sufficient to perform quantitative analytics on various stages of consider-evaluate-buy-experience-advocate-bond-buy(repeat) consumer journey. It is a must to mine deeper insights by measuring “how much impact” those trends have on the business itself. If your job is in web analytics or any other domain monitoring and analyzing various KPIs, it is not enough to measure “what is happening (in real-time)” or “what has been happening” or “what happened”. The value of such “what analysis” is akin to the skin deep beauty.

The same applies even if your job is to develop analytics tools/platforms that help users perform the above variety of analytics. If the product (tool or platform) you are building enables the users to conduct only the “”what” analytics, you are doing a huge disservice to your customers (assuming the customers are still buying that stuff!). And, if you are the customer exploring the analytics tool to buy, you have the right to demand more than the “what analysis” capabilities from your potential vendor. Remember, the right to remain silent does not apply here!

Now, if it is the degree of impact on the business outcomes that is of higher importance, how can one connect those business outcomes to the metrics being actually measured and analyzed? That’s where the business domain knowledge comes into play. For example, see the DuPont model for the Retail industry. The DuPont model establishes a relationship among various value drivers with the financial levers which in turn are connected to the business outcomes. On the other end, the measurable KPIs (for example: page visits, visitors, sessions, duration, bounce rate, etc. in web analytics) can be (and dare I also say “must be”!) quantitatively connected to the value drivers. That can be done through various statistical and/or machine learning techniques applied on the historical data.

Such “How much?” analytics with established relationships to assess the degree of impact being made on the business outcomes, are the ones that truly generate the business value.

In fact, the process of establishing the relationships among various metrics and outcomes through “How much?” analytics will become a valuable foundation for “Advanced What Analytics” such as predictive (“What might happen?”) and prescriptive (“What should happen for optimal outcome?”) for more actionable insights.

That is a nice circle of “What – How Much – What” analytics to complete.

Your thoughts please!

Acknowledgment: DuPont model for Retail developed with inputs from my ex-colleagues (Bernadette, Kim, Robyn, and Victoria), the true industry experts! Any errors are mine.

Not to be confused with Real-Time Analytics. It is different.

As a product manager in the domain of predictive analytics, I own the responsibility to build predictive analytics capabilities for consumer facing and/or enterprise platforms; the business applications vary among item recommendations for consumers, prediction of event outcomes based on classification models, demand forecasting for supply optimization, and so on. We usually see the applications where the predictive model built using machine learning technique(s) is leveraged to score the new set of data, and that new set of data is most often fed to the model on-demand as a batch.

However, the more exciting aspect of my recent work has been in the realm of real-time predictive analytics, where each single observation (raw data point) has to be used to compute the predicted outcome; note that this is a continuous process as the stream of new observations continuously arrive and the business decisions based on the predicted outcomes have to be made in real-time. A classic use case for such a scenario is the credit card fraud detection: when a credit card swipe occurs, all the data relevant to the nature of the transaction is fed to a pre-built predictive model in order to classify if the transaction is fraudulent, and if so deny it; all this has to happen in a split second at scale (millions of transactions each second) in real-time. Another exciting use case is the preventive maintenance in Internet of Things (IoT), where continuous streaming data from thousands/millions of smart devices have to be leveraged to predict any possible failure in advance to prevent/reduce downtime.

Let me address some of the common questions that I often receive in the context of real-time predictive analytics.

What exactly is real-time predictive analytics – does that mean we can build the predictive model in real-time? A data scientist requires an aggregated mass of data which forms the historical basis over which the predictive model can be built. The model building exercise is a deep subject by itself and we can have a separate discussion about that; however, the main point to note is that model building for better predictive performance involves rigorous experimentation, requires sufficient historical data, and is a time consuming process. So, a predictive model cannot be built in “real-time” in its true sense.

Can the predictive model be updated in real-time? Again, model building is an iterative process with rigorous experimentation. So, if the premise is to update the model on each new observation arriving in real-time, it is not practical to do so from multiple perspectives. One, the retraining of the model involves feeding the base data set including the new observation data point (choosing either to drop older data points in order to keep the data set size the same or not drop and keep growing the data set size) and so requires rebuilding of the model. There is no practical way of “incrementally updating the model” with each new observation; unless, the model is a simple rule based; for example: predict as “fail” if the observation falls outside the two standard deviations from the sample mean; in such a simple model, it is possible to recompute and update the mean and standard deviation values of the sample data by including the new observation even while the outcome for the current observation is being predicted. But for our discussion on predictive analytics here, we are considering more complex machine learning or statistical techniques.

Second, even if technologies make it possible to feed large volume of data including the new observation each time to rebuild the model in a split second, there is no tangible benefit in doing so. The model does not much with just one more data point. Drawing an analogy, if one wants to measure by how much the weight has reduced from an intensive running program, it is common sense that the needle does not move much if measured after every mile run. One has to accumulate a considerable number of miles before experiencing any tangible change in the weight! Same is true in Data Science. Rebuild the model only after aggregating a considerable volume of data to experience a tangible difference in the model.

(Even the recent developments, such as Cloudera Oryx, that are making efforts to move forward from Apache Mahout and similar tools (limited to only batch processing for both model building and prediction) are focused on real-time prediction and yet rightly so on batch-based model building. For example, Oryx has a computational layer and a serving layer, where the former performs a model building/update periodically on an aggregated data at a batch level in the back-end, and the latter serves queries to the model in real-time via an HTTP REST API)

Then, what is real-time predictive analytics? It is when a predictive model (built/fitted on a set of aggregated data) is deployed to perform run-time prediction on a continuous stream of event data to enable decision making in real-time. In order to achieve this, there are two aspects involved. One, the predictive model built by a Data Scientist via a stand-alone tool (R, SAS, SPSS, etc.) has to be exported in a consumable format (PMML is a preferred method across machine learning environments these days; we have done this and also via other formats). Second, a streaming operational analytics platform has to consume the model (PMML or other format) and translate it into the necessary predictive function (via open-source jPMML or Cascading Pattern or Zementis’ commercial licensed UPPI or other interfaces), and also feed the processed streaming event data (via a stream processing component in CEP or similar) to compute the predicted outcome.

This deployment of a complex predictive model, from its parent machine learning environment to an operational analytics environment, is one possible route in order to successfully achieve a continuous run-time prediction on streaming event data in real-time.