Overcoming Data Stewardship Challenges in Complex and Big Data Environments: Six Mission-Critical Tasks

Dan Carotenuto's picture
 By | October 01, 2013
in Big Data, Data Quality, gartner magic quadrant for data quality tools, master data management, strategy, voice of the customer
October 01, 2013

Have you considered how to manage your data, your information assets, effectively no matter how complex the environment--big or small? 

Let me guess, 1) your organization’s stakeholders, customers and/or partners are not getting what they need on time or in the form they need it, or 2) you and/or your partners spend too much time continually fixing and manually preparing reports and data, or 3) your organization is in the process of implementing data stewardship or a more formal data governance program to get your "house" (data) in order.  If you’re not in group 3, you should be.  If you are in group 3, have you considered how to manage your data--your information assets—effectively no matter how complex the environment--big or small?  There are six mission-critical functions highlighted in a recent white paper from Information Builders, "Data Stewardship in Complex and Big Data Environments," that you should review to make your approach as future proof as possible.

The paper examines data stewardship and the value and role technology plays in automating its interoperable aspects that enable companies and agencies to become data-driven organizations. It reviews six functions that data management technologies should perform “in an integrated fashion to promote data integrity, collaboration, workflow management and accountability among participants:" 

  • Data Integration
  • Data Profiling
  • Data Quality
  • Remediation for Exceptions
  • Document Access
  • Collaboration/Interoperability

It is imperative that an organization’s data management technologies not only support all of these functions, but that all of these functions interoperate with each other and facilitate collaboration among subject matter experts (SME) and data contributors.  "Remediation for Exceptions" and "Collaboration/Interoperability" are two functions that facilitate the interoperability among stakeholders to improve operations. 

For example, as data comes into the enterprise from a partner, customer or social network, a fully integrated and interoperable data management solution should be able to enrich and transform that data with information from other systems and sources (e.g., master data for Voice of Customer, inventory management system, etc.), cleanse it automatically based on rules or notify contributors or SMEs for manual remediation and then send it for further processing or storage for corporate reporting and analysis.  One of the key points here is that data management technologies must support manual remediation and collaborative workflows when data falls outside of automated cleansing rules as part of the integration process. 

The paper walks through the operational challenges a retail organization faces and illustrates how these six functions can be used as part of any organization’s "...corporate data governance practice to help drive measurable improvements in sales and operations."  Some of the challenges the paper notes span store operations having POS data inconsistencies and lack of quality checks, marketing operations having to work with duplicate, inconsistent, inaccurate and missing customer loyalty data, corporate reporting and analytic teams having to constantly reconcile reports at the store, regional and national level due to inconsistencies between systems, stock keeping units, supplier product codes, areal overlap, and duplicate customer information. 

A collaboration-based improvement noted in the paper includes formulating a data stewardship program between in-store merchandising teams, in-store sales associate teams, corporate reporting teams, sales administration reporting teams, marketing campaign team and customer service representatives.  "The collaborative tools enabled the company to proactively catch and fix data problems up front, identify opportunities and risk, address and correct lingering  inconsistencies, and share and fix common problems that impact multiple business operations."

Data quality as part of the integration process "...allowed the organization to validate customer data in real time, as a transaction is entered at the store. It was also used to cleanse product data by cross-referencing, validating, and standardizing product information in both batch and real-time environments. This standardized data between inventory systems and significantly raised the level of consistency and accuracy of the information for downstream business use."

Integrating manual remediation into the integration process helped the retail organization to, "...identify unique data problems at every step in the data lifecycle, enter the issue into an interactive workflow, and have the appropriate work group address the problem for resolution."

I strongly recommend you read the paper for more in depth analysis of how these six functions can help overcome data stewardship challenges in complex and big data environment.

Does your organization use these six functions to support your data stewardship program?  Sound off below.