Data Integration in Government – 5 Lessons Learned

Dave Kelly
September 09, 2019

"Why?" is a great question; it will lead you down paths you never imagined traveling. Often, your pursuit of the answer will bring you knowledge and experiences you would have missed had you not taken the time to understand.

In my previous career, I asked a simple "why" question: “Why can’t our personnel have easy access to all the data relevant to their areas of responsibility?” In pursuit of the answer, I found myself leading a large-scale, public sector data integration project.

Find the people who support your efforts, and remember to continuously ask why. You might be surprised how far that one simple question will take you.

The answer to my question proved to be complicated. But the ultimate rewards were well worth the effort. We achieved a one-stop-shop for data insights, gained significant workflow efficiencies with a single query to all data approaches, and, most importantly, we showed the agency what was possible with a thoughtful approach to data. Along our path to success, we learned valuable lessons, which I share with you in the hopes that your path may be a bit less rocky:

Start Small

People are afraid of change and this fear can be a significant impediment if you begin by announcing your grand vision to integrate every dataset inside and outside of your organization. A better approach is to find a specific problem to solve, find out who owns that problem, and tell them you want to help them solve it. We found that small successes were a force multiplier to gaining buy-in across our agency and ultimately led to interest from outside the agency.

Focus on Outcomes

There is a tendency to allow the technology to wag the dog. I mean that while we may have acquired a deep understanding of the technology, we have a shallow understanding of the business problems. This leads to an expectation mismatch between the developers of a technology and the end-users, which leads to a lack of adoption. If the end-users expect it to do X and the developers built it to do X-Y, it shouldn't be a surprise when no one uses it. If you're leading a data integration project, spend your time listening to the subject matter experts. It is your job to gain a solid understanding of their business problems and desired outcomes.

Let the Use Cases Drive the Integration

A use case is simply an example of how an end-user will use your integration project to achieve their goals. The data you ultimately select to integrate should be specific to achieving user goals. Here is the use case-driven approach that we used: identify the problem, spend time with the subject matter experts to ensure you understand the problem, identify the data needed to solve the problem, build use cases around the problem, incessantly repeat the use cases to your team until there is no question about the problem they have been tasked with solving, and simultaneously integrate the data while building the necessary data visualization.

Build It and the DSA Will Come

The purpose of a data integration project should be to create an efficient means for sharing data. That said, resist the temptation to start your project by attempting to secure Data Sharing Agreements. Rather, understand that other parts of government have problems similar to the ones you are trying to solve. Prove the concept internally before seeking outside participation. We found that it was much easier to sell the integration idea when we had a working prototype with which we could demonstrate success.

Communicate, Communicate, Communicate

This will prove difficult. There will be all sorts of people who are inherently suspicious of what you are doing, but there are innovators out there.

Find them and recruit them to help you explain the value of your integration project. The two communication points that we found most critical were:

  1. Ensuring the decision-makers in each area knew we were following the same access controls as their source system
  2. Making sure we were engaged with their subject matter experts to ensure we understood the data classification and complied with all data sharing regulations (CJIS, HIPAA, etc.)

Once we communicated these two points to the right people, the friction was greatly reduced. These projects are difficult. To provide your team with the best chance to succeed, start by understanding how you will bring value to the organization, find the people who support your efforts, and remember to continuously ask why. You might be surprised how far that one simple question will take you.

Want to know the six worst practices in data governance?