Data Integration: Making the Complex Simple
Enterprise application integration (EAI) created a buzz surrounding the way it changes the way "traditional" Information Technology (IT) organizations process their data. EAI effectively addressed and addresses the need for applications to integrate with one another. Tools have been on the market now for several years giving IT organizations the ability to perform EAI functions. What does EAI have to do with data integration?
While organizations were rapidly addressing the need for their applications to interact in more effective ways they also allowed data integration to become tooled. The process of extracting, transforming, and loading data in bulk mode to serve data warehouses designed by those IT organizations became the de facto data integration process. ETL tools arose to serve the data warehousing need and an entire discipline, entirely separate from the parallel EAI discipline, was created.
Enterprise data integration became a function of the enterprise ETL tool. This functionality belongs, rightly so, in the traditional IT organization. It takes an efficient and effective organization to accurately and actively manage data warehousing initiatives in any organization. ETL tools over the years have grown in their footprints and reach within IT organizations to the point where they are unusable by people outside of that organization. They are managed as another IT application, with the traditional IT disciplines associated with that type of an application.
Within those same organizations, it's likely that there are technical folks doing ETL processing. They are using tools like FOCUS, SAS, and Easytrieve to create extract files, transfer those files to servers, and load those files to databases like Microsoft Access or SQL Server. This has been going on for years, providing a "departmental data mart" for reporting and analysis. Rather than working with IT organizations and the behemoth ETL tools, these technical business analysts retain their processes simply because they work and they can be run within the business unit with no interference from the IT organization.
Consider the process: A business analyst creates a data mart for her department's reporting needs every Monday morning. She comes in, submits a job, and waits for the job to finish. The job finishes and she transfers that file to a server. She then runs a load script on the server to load the data mart. All of the processing involved in the creation of the data mart, along with the maintenance of the extract job, the file transfer, and load script, exist with her – and changes required to the data mart are entirely up to her and her availability. Once she gets the data mart loaded, she moves on to do her analyses, the thing she is actually paid to do! It is not uncommon for these types of functions to take between five and 10 hours per week (at a minimum) to execute and maintain.
This type of processing is nothing new to large organizations. Business analysts have used it for years and IT organizations have peripherally supported their development and use. In order for the business unit to continue to maintain those data marts, an ETL tool must be easy to use and able to support the myriad data sources (from legacy FOCUS databases to incoming XML documents) that are a part of those data marts. An ETL tool must exist for the business analyst to use to reclaim the time she takes loading the data mart for her department. Traditional ETL tools and processes are not the answer for departmental data marts.
IT organizations supporting enterprise level ETL tools should welcome this type of departmental data mart creation, even if the creation of that data mart is done with a tool outside the "standard" IT supported ETL tool. The business analyst who does her job every Monday morning has a tendency to usurp expensive computing resources to complete that job. With an easy-to-use ETL tool in the hands of the business analyst, one that is complete with a scheduling component, the departmental data mart's creation can be moved to non-peak hours, freeing up those expensive resources.
Information Builders' DataMigrator certainly can help that business analyst reclaim her time. The extraction, transformation, and loading of the departmental data mart is wrapped into a graphical user interface, making use of industry-leading adapters from iWay Software to access any information asset in the organization. Process flows allow the ability to schedule ETL processes early on a Monday morning before the department's staff arrives so that the data mart is available immediately.
Putting information in the hands of people that use it should be the goal of any data integration or ETL initiative. That is our job here at Information Builders and our toolset is committed to doing just that. We invite you to take a look at the next generation of ETL processing.
 
|
