A Look Back At 2016 and Predictions for the New Year
N.B., Dr. Rado Kotorov will discuss 5 Hot Trends for Data and Analytics in 2017 in a January 10 webcast.
Welcome to the last work day of 2016! Traditionally, at the end of a year, we try to glean some wisdom from the past and some insight into the future.
I’ll distinguish between knowledge and wisdom here: Knowledge is the recognition that we're probably in a pendulum swing – I think that’s pretty clearly true, because it’s always true in some way or other – while wisdom is knowing what pendulum we’re on and where in its swing we currently sit.
The past few years have been disruptive. Business users demanded tools that enable them to 1) find data points that they need 2) using data that they source, which resulted in IT -- and its mantras of reliability, repeatability, and efficiency -- being relegated to a back seat for analytics.
Some organizations (vendors, mostly, along with one very prominent IT analyst firm) have even described this tool architecture as the definition of "modern BI".
I'm pretty sure that we've gone as far as we can on this swing of the pendulum. Several things have led me to this belief:
- We're seeing people hitting the wall with "modern" BI tools. The disruptive darlings of the industry continue to be well-known and to sell product. But when we go to trade shows, talk to prospects, and listen to customers, we consistently find that they're running into the same things that they used to see with Excel: Untrusted data, different results from different people, and answers that open up more questions. It may be relevant that the stock prices for the companies that make these tools have declined, and that their profitability is in question. Don't get me wrong, I don't expect them to go away -- just like I don't expect Excel to go away, and for many of the same reasons. But their current position has the feel of “peak pendulum” to me.
- Many of these problems are related, at least in part, to a lack of governance. Sometimes they lose trust because people source their own data, and it's hard to govern data that can come from anywhere. Sometimes the analytical processes themselves are too open-ended, unpredictable, and unreproducible.
- More data prep capabilities are being demanded in the tools. This sounds like it implies greater flexibility in the tools, but it actually implies two other things as well: First, IT-centric "data as a service" notions aren't giving business users the level of flexibility that they want, which implies that it will continue to be difficult to get the same answers among different groups when they walk into boardrooms. Second, business users are putting more custom data cleansing, integration, and preparation into their analytics, which means those same rules aren't being applied to enterprise data -- meaning the rules themselves are lost, as far as the enterprise is concerned.
- All of these things have led to the rise of the Chief Data Officer. This position is becoming more popular -- but that wouldn't be likely if self-service tooling were truly the be-all and end-all of analytics. Chief Data Officers are taking a broader look at how data can help everywhere within their organizations. Yes, that will include self-service tools, but by definition a C-level officer don't just focus on individuals. Trust in data, trust in analytical processes, and trust in people all will factor into a CDO's value to the enterprise.
- There's some pushback to the Gartner "bi-modal IT" model that led to this "modern" approach in the first place. In 2014, Gartner described bi-modal IT as an approach that allows IT to handle the things that require high quality and reproducibility, and allows the business to handle rapid-fire, agile requirements when they think IT is taking too long. But if you read Forbes, CIO Magazine, and others, you find senior technologists and analysts rejecting the idea that IT effectively means "not agile".
These are things that lead me to believe that the self-service BI tooling pendulum has swung about as far toward individual business user tools as it can. But what do they imply?
Here are some ideas of things that will take place over the course of this year (and the next) as a result of the pendulum swing.
IT will start automating the choices for data management and analysis, leading to standardized data prep, quality, and governance.
BI tools have been making more decisions for people and automating more processes. The knowledge for doing this -- e.g., choosing one chart type over another -- was embedded into the tools themselves. Data prep and management tends to be different, because the required rules are specific to the business requirements rather than being inherent in the data.
Rule-based data management will enable IT to define rules that the business uses in its analytics processes, making business analysts more productive while still ensuring reliability and reproducibility.
For a use case, consider a data scientist who sources data externally, and lets the data tools automatically choose which enterprise data prep and cleansing processes need to be applied.
The decisions made by business users will make their way back into core enterprise IT processes.
This is the flip side of the coin above. There is a stock of cleansing rules, integration processes, and so on that business users employ to get data into the shape they want. These rules are frequently used once, or used only by a specific person, and then thrown away. Those are intellectual resources that shouldn't be wasted.
Think of a data scientist creating data cleansing rules for an analytical process that leverages a data lake. The knowledge that goes into her processes can be captured, stored, and applied to downstream feeds, data governance processes that correct source systems, and integrated views of the data for others.
The Chief Data Officer position will pick up steam significantly.
This is a sure sign of the pendulum swinging back: A company officer centrally managing the value of data.
And a CDO's job isn't to empower analysts per se, although that will often be part of what they do. If that were all it was, companies could save a lot of money by handing out tools and not creating the CDO position.
The CDO's job is to extract maximum value from data. That can be done in many ways, including customer-facing portals, large-scale analytical apps, data feeds that stem from unified views of business entities, embedded BI inside other enterprise applications, and so on.
So as the CDO position picks up steam, we can expect to see larger data-focused projects where information is managed and shared across divisional and even company boundaries, leading to better data monetization, lower per-user cost of data, and higher business value per unit of data.
Those are three major predictions that I'm willing to stand by. What do you think about them? If you disagree with these ideas, what sorts of things do you think come next? Let us know in the comments.
Want more ideas? Don't miss our webcast with Dr. Rado Kotorov on January 10: 5 Hot Trends for Data and Analytics in 2017. Topics include:
- Industry 4.0: building the new digital enterprise using automation.
- Analytics in the digital culture revolution: democratizing BI for fact-based decision-making.
- Report consolidation: saving costs, improving BI adoption, and moving from reports to InfoApps.
- The rise of customer-facing analytics: monetizing data and consumerizing analytics.
- Data management in the digital enterprise: master data management (MDM), data quality, and governance.