April 2017, IDC #US41715317
TAXONOMY
IDC's Worldwide Data Integration and Integrity Software
Taxonomy, 2017
Stewart Bond
IDC'S WORLDWIDE DATA INTEGRATION AND INTEGRITY SOFTWARE TAXONOMY
FIGURE 1
Data Integration and Integrity Software Primary Segments
Source: IDC, 2017
©2017 IDC
#US41715317
2
DATA INTEGRATION AND INTEGRITY SOFTWARE TAXONOMY CHANGES FOR 2017
Three primary changes have occurred since the last published taxonomy for the data integration and
access (DIA) software functional market (see IDC's Worldwide Data Integration and Access Software
Taxonomy, 2015, IDC #255856, June 2015):
 Name change from data integration and access software to data integration and integrity
software
 Consolidation of the general data quality and domain-based matching and cleansing market
segments into one data quality software segment
 Addition of the self-service data preparation software segment
Table 1 maps these changes as they were made in 2016, and how they relate to the 2015 market
definition.
TABLE 1
Summary of Data Integration and Integrity Software Functional Market Changes
for 2016
2015 Market
2016 Market
Comments
Application development
and deployment
Data integration and
access software
Data integration and
integrity software
Renamed and expanded the scope of
this market (Self-service data
preparation is added as a submarket.
Overall market name changed [to data
integration and integrity] to emphasize
the growing importance of data quality.)
Data integration and
integrity software
General data quality
segment consolidated
with domain-based
matching and cleansing
segment
Data quality segment
Market consolidation – software vendors
with general data quality software are
investing in domain-based matching and
cleansing and vice versa
Self-service data
preparation segment
New segment describing data access
and preparation tools suitable for
business users
Source: IDC, 2017
Data is core to digital transformation, and data without integrity will not be able to support digital
transformation initiatives. Data with integrity is data that is trusted, available, secure, and compliant
across the enterprise IT environment, regardless of whether data is persisted on-premises or in the
cloud. Data integration software vendors have the tools, intellectual property, and products that can
improve the integrity of data. Integrity has become the secondary focus of solutions in this market, and
for this reason, IDC changed the name of the software market from data integration and access to data
integration and integrity (DII).
©2017 IDC
#US41715317
3
Until 2016, IDC tracked data quality software spend within the data integration and access software
functional market as two subsegments: general data quality, and domain-based matching and
cleansing software. These two segments are consolidating as software vendors in the general data
quality segment add domain-based matching and cleansing software, and conversely, domain-based
matching and cleansing software vendors are adding general data quality software to their portfolio.
While in some cases the products continue to be separate, it is becoming increasingly difficult to track
spend as the lines are blurring. As the market consolidates, so must the segments that IDC is tracking.
A lot happened on the way to big data insights, including a need to put data preparation into the hands
of business users. 2015 saw a surge of activity in a self-service data preparation software category,
and therefore, IDC started tracking spend in this category in 2016. In many cases, the purpose of
software in this segment is to prepare data for analysis, and because functionality is similar to extract,
transform, and load (ETL), IDC decided to record and track revenue within the DII software market
instead of the business analytics market.
TAXONOMY OVERVIEW
Data integration and integrity (DII) software enables the access, blending, movement, and integrity of
data among multiple data sources. The purpose of data integration and integrity software is to ensure
the consistency of information where there is a logical overlap of the information contents of two or
more discrete systems. Data integration is increasingly being used to capture, prepare, and curate
data for analytics. It is also the conduit through which new data types, structures, and content
transformation can occur in modern IT environments that are inclusive of relational, nonrelational, and
semistructured data repositories.
DII software employs a wide range of technologies, including, but not limited to, extract, transform, and
load; change data capture (CDC); federated access; format and semantic mediation; data quality and
profiling; and associated metadata management. Data access is increasingly becoming open with
standards-based APIs; however, there is still a segment of this market focused on data connectivity
software, which includes data connectors and connectivity drivers for legacy technologies and
orchestration of API calls.
DII software may be used in a wide variety of functions. The most common is that of building and
maintaining a data warehouse, but other uses include enterprise information integration, data
migration, database consolidation, master data management (MDM), reference data management,
metadata management, and database synchronization, to name a few. Data integration may be
deployed and executed as batch processes, typical for data warehouses, or in near-real-time modes
for data synchronization or dedicated operational data stores. A data migration can also be considered
a one-time data integration. Although most applications of DII software are centered on structured data
in databases, they may also include integrating data from disparate, distributed unstructured or
semistructured data sources, including flat files, XML files, JSON, and legacy applications, big data
persistence technologies, and the proprietary data sources associated with commercial applications
from vendors such as SAP and Oracle. More recently, an interactive form of data integration has
emerged in the application of self-service data preparation.
The data integration and integrity software market includes the submarkets as illustrated in Figure 2
and discussed in the Definitions section that follows.
©2017 IDC
#US41715317
4
FIGURE 2
IDC Data Integration and Integrity Software Taxonomy Model
Source: IDC, 2017
DEFINITIONS
Bulk Data Movement Software
This software, commonly referred to as extract, transform, and load software, selectively draws data
from source databases, transforms it into a common format, merges it according to rules governing
possible collisions, and loads it into a target. A derivative of this process, extract, load, and transform
(ELT), can be considered synonymous with ETL for the purpose of market definition and analysis. ELT
is an alternative commonly used by database software vendors to leverage the inherent data
processing performance of database platforms. ETL and/or ELT software normally runs in batch but
may also be invoked dynamically by command. It could be characterized as providing the ability to
move many things (data records) in one process.
©2017 IDC
#US41715317
5
The following are representative companies in this submarket:

IBM (InfoSphere DataStage)

Informatica (PowerCenter)
 Oracle (Oracle Data Integrator)
 SAP (SAP Data Integrator, SAP Data Services, HANA Smart Data Integration)
 SAS (SAS Data Management)
 Talend (Open Studio for Data Integration)
Dynamic Data Movement Software
Vendors often refer to this functionality as providing "real time" data integration; however, it can only
ever be "near" real time because of the latency inherent in sensing and responding to a data change
event occurring dynamically inside a database. Data changes are either captured through a stored
procedure associated with a data table or field trigger or a change data capture (CDC) facility that is
either log or agent based.
These solutions actively move the data associated with the change among correspondent databases,
driven by metadata that defines interrelationships among the data managed by those databases. The
software is able to perform transformation and routing of the data, inserting or updating the target,
depending on the rules associated with the event that triggered the change. It normally either features
a runtime environment or operates by generating the program code that does the extracting,
transforming, and routing of the data and updating of the target. It could be characterized as software
that moves one thing (data record or change), many times.
The following are representative companies in this segment:

IBM (InfoSphere Data Replication)

Informatica (PowerCenter Real Time Edition)
 Oracle (Oracle GoldenGate)
 SAP (SAP Replication Server, SAP Data Services)
Data Quality Software
This submarket includes products used to identify errors or inconsistencies in data, normalize data
formats, infer update rules from changes in data, validate against data catalog and schema definitions,
and match data entries with known values. Data quality activities are normally associated with data
integration tasks such as match/merge and federated joins but may also be used to monitor the quality
of data in the database, either in near real time or on a scheduled basis.
The data quality software submarket also includes purpose-built software that can interpret,
deduplicate, and correct data in specific domains such as customers, locations, and products. Vendors
with domain-specific capabilities typically offer products that manage mailing lists and feed data into
customer relationship management and marketing systems. The following are representative
companies in this submarket:
 Experian Data Quality

IBM (InfoSphere Discovery, BigInsights BigQuality)

Informatica (Informatica Data Quality and Data-as-a-Service)
©2017 IDC
#US41715317
6
 Melissa Data
 Pitney Bowes (Code-1 Plus and Finalist)
 SAP (SAP Data Services, SAP HANA Smart Data Quality)
 SAS Data Quality
 Talend (Talend Open Studio for Data Quality)
 Syncsort (acquired Trillium Software)
Data Access Infrastructure
This software is used to establish connections between users or applications and data sources without
requiring a direct connection to the data source's API or hard-coded database interface. It includes
cross-platform connectivity software. It also includes ODBC and JDBC drivers and application and
database adapters. The following are representative companies in this submarket:
 Micro Focus (Attachmate Databridge)

Information Builders (iWay Integration Solutions)
 Progress Software (DataDirect Connect and DataDirect Cloud)
 Rocket Software (Rocket Data)
Composite Data Framework
This market segment includes data federation and data virtualization software, enabling multiple
clients (usually applications, application components, or databases running on the same or different
servers in a network) to share data in a way that reduces latency. Usually involving a cache, this
software sometimes also provides transaction support, including maintaining a recovery log.
Data federation permits the access to multiple databases as if they were one database. Most are read
only, but some provide update capabilities. Data virtualization products are similar but offer full schema
management coordinated with the source database schemas to create a complete database
environment that sits atop multiple disparate physical databases. Originally created to provide the data
services layer of a service-oriented architecture (SOA), data virtualization has been applied to multiple
use cases including data abstraction, rapid prototyping of data marts and reports, and providing
abilities for data distribution without replication.
The following are representative companies in this submarket:
 Cisco Data Virtualization Platform
 Denodo

IBM (InfoSphere Federation Server)

Informatica Data Virtualization
 Red Hat (JBoss Data Virtualization)
 SAS (SAS Federation Server)
Master Data Definition and Control Software
The master data definition and control software market segment includes products that help
organizations define and maintain master data, which is of significance to the enterprise and multiple
systems. Master data is usually categorized by one of four domains: party (people, organizations),
©2017 IDC
#US41715317
7
product (including services), financial assets (accounts, investments), and location (physical property
assets).
Master data definition and control software manages metadata regarding entity relationships,
attributes, hierarchies, and processing rules to ensure the integrity of master data for where it is to be
used. Key functionalities includes master data modeling, import and export, and the definition and
configuration of rules for master data handling such as versioning, access, synchronization, and
reconciliation.
This market segment also includes reference data management software. Reference data is used to
categorize or classify other data and is typically limited to a finite set of allowable values aligned with a
value domain. Domains can be public such as time zones, countries and subdivisions, currencies,
units of measurement, or proprietary codes used within an organization. Domains can also be industry
specific such as medical codes, SWIFT BIC codes, and ACORD ICD codes. As with master data,
reference data is referenced in multiple places within business systems, and in some cases, can
deviate across disparate systems. Reference data management software could be considered a
subset of master data management (MDM) software and, in many cases, may share the same code
base as master data management software sold by the same vendor.
Master data definition and control products can include operational orchestration facilities to coordinate
master data management processes across multiple information sources in either a batch, a service,
or an event-based context. Master data definition and control software provides capabilities to facilitate
single or multiple master data entity domain definition and processing and usually serves as the core
technical component to a broader master data management solution (a competitive market in the IDC
taxonomy).
Representative vendors and products include:

IBM (IBM InfoSphere Master Data Management)

Informatica (Multi-Domain MDM, PIM)
 Oracle (Oracle Product Hub, Oracle Customer Hub, Oracle Site Hub, Oracle Higher Education
Constituent Hub, and Oracle Data Relationship Management)
 SAP (SAP Master Data Governance)
 TIBCO (Master Data Management Platform)
Metadata Management Software
Metadata is a data about data. It is commonly associated with unstructured data, as a structure in
which to describe the content. However, it is increasingly being used in the structured data world, as
the data becomes more complex in its definition, usage, and distribution across an enterprise. At a
basic level, metadata includes definitions of the data, when, how, and by whom the data was created
and last modified. More advanced metadata adds context to the data, traces lineage of the data, cross-
references where and how the data is used, and improves interoperability of the data. Metadata has
become critical in highly regulated industries as a useful source of compliance information.
The metadata management submarket has grown out of the database management, data integration,
and data modeling markets. Metadata management solutions provide the functionality to define
metadata schema, automated and/or manual population of the metadata values associated with
structured data, and basic analytic capabilities for quick reporting. These solutions also offer
©2017 IDC
#US41715317
8
application program interfaces (APIs) for programmatic access to metadata for data integration and
preparation for analytics.
Representative vendors and products include:
 ASG (Enterprise Data Intelligence)

IBM (InfoSphere Information Governance Catalog)

Informatica (Business Glossary, Enterprise Information Catalog, Live Data Map)
 Oracle (Enterprise Metadata Management)
 SAP (SAP Information Steward Metadata Management and Metapedia)
Self-Service Data Preparation Software
Self-service data preparation is an emerging segment in the DII software functional market. Demand is
coming from today's tech-savvy business users wanting access to their data in increasingly complex
environments, without IT getting in the way. IT has also been looking for technology that can put data
preparation capabilities into the hands of business users, where the requirements and desires of
analytic outcomes are best understood.
These solutions are targeted at business analysts, but some offer a range of user interfaces by
persona. Some require deeper levels of data analytics knowledge, and others require deeper levels of
data integration knowledge. Functionality ranges from cataloging data sets within repositories,
providing interactive capabilities for cleansing and standardizing data sets, joining disparate data sets,
performing calculations, and varying degrees of analytics for validation prior to visualization.
Software in this segment represents vendors that explicitly sell self-service data preparation
components. Business intelligence and analytics vendors that offer self-service data preparation
capability within a suite as part of the analytics process they support are excluded from this segment.
Representative vendors and products include:
 Alteryx (Designer)
 Datawatch (Monarch)
 Paxata
 SAP (Agile Data Preparation)
 Talend (Data Preparation)
 Trifacta (Wrangler)
 UniFi
RELATED MARKETS
Table 2 provides data integration and integrity software related markets.
©2017 IDC
#US41715317
9
TABLE 2
Data Integration and Integrity Software Related Markets
Competitive Market
Relationship
Master data management
(MDM) software
The master data definition and control segment of the DII market, combined with portions of
data access, movement, quality and metadata segments make up much of the MDM platform
segment of the MDM competitive market.
Business analytics
software
A portion of the data integration and integrity software market is used to derive the business
analytics market, specifically within the data warehouse generation segment.
Source: IDC, 2017
LEARN MORE
Appendix: Patterns of Use
A pattern of use is a commonly observed assembly of software components, meeting the needs of
commonly observed business and technical requirements, within specific organizational and technical
constraints. The DII software market segments are aligned with data integration patterns of use. Some
patterns that leverage components from across the segments, and conversely some patterns support
solutions in other segments and competitive markets (e.g., master data management).
Vendors could consider patterns in the packaging and licensing of software products. Vendors are in a
unique position to help identify new patterns that are emerging in the install base and respond with
new products, packaging, and licensing options to gain a competitive advantage.
The patterns discussed in the sections that follow are typical of many (but not all) data integration
solution implementations, applied to meet business requirements within technical and organizational
constraints. They are presented in a way that vendors could help customers identify the pattern and
justify motivation for, and implications of, using the pattern.
Scheduled Bulk Data Integration
Scheduled bulk data integration is used to move large sets of data from a source to a destination on a
frequent schedule. This pattern primarily leverages software in the bulk data movement segment of the
market, namely ETL or the derivative ELT. Figure 3 illustrates the two types of implementation within
this pattern.
©2017 IDC
#US41715317
10
FIGURE 3
Scheduled Bulk Data Integration Patterns
Source: IDC, 2017
The extract process happens on a scheduled basis, depending on the data latency requirements in the
target. This pattern is often used to populate data warehouses, data marts, or operational data stores
(ODSs) on a regularly scheduled basis. It can also be used as a one-time data migration.
This pattern is efficient for processing large data sets, as bulk data movement technology is built to
handle large volumes of data. Some data transformation requirements can only be achieved with
ETL/ELT, such as match/merge between multiple source data sets, complex de-normalization, and
summarizing and sorting of transactional data.
Use of this pattern introduces latency between the source and target and is constrained by the
physical data stores and processing capabilities of the underlying infrastructure.
Although the bulk data movement market segment primarily serves this pattern, vendors could bundle
components from the data access segment into solutions for this pattern. Vendors could also consider
metadata management add-on components for traceability of data lineage through the transformation
process, and data quality components for cleansing the data.
Near-Real-Time Data Integration
Near-real-time data integration is used to move smaller sets of data more frequently. This pattern
primarily leverages components in the dynamic data movement market segment. It is most often used
when changes in source data need to be captured and distributed to other systems. Figure 4 illustrates
the pattern and the multiple methods used to capture changes.
©2017 IDC
#US41715317
11
FIGURE 4
Near-Real-Time Data Integration Pattern
Source: IDC, 2017
Changes in source data can be captured using a stored procedure or database trigger. The trigger
may populate a change log inside the source database or send to an external persistence service.
CDC is a technology that runs outside of the source database and monitors database log files for
changes. Once a change is recognized, "before" and "after" images of the data in scope are sent to a
persistence service. A micro-batch is another alternative for picking up changes. These run every few
minutes and pull only the data that has changed since the last time the micro-batch was executed. It
may pull data out of a change log or directly out of source tables. The persistence service could be a
message queue, file system, external data store, or other types of containers. Mediation services pick
up the changed data from the persistence service and perform the necessary transformation,
formatting, and distribution to the target system(s) that need a copy of the change.
This pattern is efficient for processing data changes when latency between source and target needs to
be minimized. Change data capture is an efficient, noninvasive method for recognizing changes in
application databases when API-level access is not available, such as packaged applications with
proprietary APIs, or legacy systems that are costly to modify. Database triggers, change tables, or
other source data schema modifications are invasive and can nullify application vendor support
agreements.
Dynamic data movement is the primary market segment that serves this pattern. Vendors could also
bundle data access and metadata management components for solutions in this pattern. Vendors
could also bundle this pattern as the underlying middleware in master data management solutions,
distributing changes to master data from the systems of entry to systems of record and reference.
Real-Time Data Integration
Real-time data integration provides blending and transformation of data across multiple disparate
sources in a virtualized view. Also known as data virtualization, it could be considered data federation
without persistence. Figure 5 illustrates this pattern assuming two disparate data sources.
©2017 IDC
#US41715317
12
FIGURE 5
Real-Time Data Integration Pattern
Source: IDC, 2017
In this pattern, the composite data framework accesses the disparate data sources, querying for the
data in scope from both, joins the results in memory and provides a virtual view of the integrated data,
isolating the data access and integration logic from the consumer of the view. The virtual view is made
available to consumers via APIs.
This pattern is efficient when zero latency is required between the source data and consumers. It is
also useful in situations where flexibility and agility of reporting schemas are important. A virtual view
can be changed without any underlying impacts.
With real-time access comes impact. Querying against live systems can impact performance of the
source databases and in turn can impact the response time of the virtual views. Availability of sufficient
memory on the infrastructure running the composite data framework software, in addition to sufficient
network bandwidth, is important when considering this pattern. It is also important that all underlying
data security policies are replicated in the virtual views, to prevent unauthorized access.
Packaging of composite data framework, metadata management, and data access software
components is required for end-to-end solutions in this pattern. Vendors that have composite data
frameworks, but not data quality solutions, could encourage the application of this pattern to identify
quality inconsistencies across disparate data sources.
Data Federation
Data federation pulls data from multiple disparate sources, blends and transforms it into one data set,
and persists the results into another physical data store. It could be used to populate an operational
data store, reporting database, or data mart that needs a federated view of disparate data. Figure 6
illustrates the data federation pattern.
©2017 IDC
#US41715317
13
FIGURE 6
Data Federation Pattern
Source: IDC, 2017
Data federation can be used when latency is allowed between source systems and the federated
views. Latency can be reduced with near-real-time population of the target data store. It is also
appropriate when memory and network bandwidth constraints preclude data virtualization.
Federated data should be used in read-only mode for reporting and analysis purposes. Allowing write
access to the federated repository can cause a data governance issue as it creates an alternate
version of the source data, leading to inconsistencies.
The data federation use case leverages software in the composite data framework segment and can
use any combination of dynamic and bulk data movement components to move data from source to
target. Vendors could consider bundling these components together and adding metadata
management components for a comprehensive solution that will also provide context and lineage of
data in the federated data store.
Data in Motion Pattern
Data in motion represents the application of data integration technology against streaming data. Many
3rd Platform technologies constantly emit data, available to organizations that have an interest or need
for the data. Data is constantly in motion, coming from social networks, the Internet of Things, stock
market tickers, and so forth. Figure 7 illustrates an application of data integration technology to data in
motion.
FIGURE 7
Streaming Integration Pattern
Source: IDC, 2017
©2017 IDC
#US41715317
14
This is an emerging pattern and is applicable to those organizations that are working with, or have a
desire to leverage, streaming data. Any foray into the use of data streams needs to be made taking
into consideration the abilities of the organization to capture and respond to business-relevant events
in the stream. Available system resources also need to be understood in the context of the velocity and
volume of the streaming data.
This pattern leverages several components in the data integration software market, and vendors are
encouraged to consider bundling alternatives to meet the requirements of this pattern, as applicable to
customer requirements. Data access software provides the capability to interface with social media,
Internet of Things, and other data stream sources. Mediation capabilities available in bulk and dynamic
data movement components assist with identifying and filtering data in the streams that are relevant to
the organization. Data quality components can be used to validate the data prior to allowing into the
persistent storage of the organization.
Related Research

IDC's Worldwide Services Taxonomy, 2017 (IDC #US42356617, March 2017)
 Market Analysis Perspective: Worldwide Data Integration and Integrity Software, 2016 (IDC
#US40799016, September 2016)
 Worldwide Data Integration and Integrity Software Market Shares, 2015: Year of Big Data and
the Emergence of Self-Service (IDC #US40696216, June 2016)
 Worldwide Data Integration and Integrity Software Forecast, 2016–2020 (IDC #US40696116,
June 2016)
Synopsis
This IDC study provides taxonomical definitions for the data integration and integrity software market.
The study segments the market into data access, movement, integration, and governance functional
categories. Market segments include bulk data movement, dynamic data movement, composite data
frameworks, data quality, master data definition and control, metadata management, and self-service
data preparation.
"Data is core to digital transformation, and data without integrity will not be able to support digital
transformation initiatives," says Stewart Bond, research director for Data Integration and Integrity
Software at IDC. "Data integration software provides organizations with the ability to catalog, move,
transform, match, cleanse, and prepare data improving the level of trust and integrity of data,
improving the outcomes of digital transformation initiatives on the 3rd Platform."
About IDC
International Data Corporation (IDC) is the premier global provider of market intelligence, advisory
services, and events for the information technology, telecommunications and consumer technology
markets. IDC helps IT professionals, business executives, and the investment community make fact-
based decisions on technology purchases and business strategy. More than 1,100 IDC analysts
provide global, regional, and local expertise on technology and industry opportunities and trends in
over 110 countries worldwide. For 50 years, IDC has provided strategic insights to help our clients
achieve their key business objectives. IDC is a subsidiary of IDG, the world's leading technology
media, research, and events company.
Global Headquarters
5 Speen Street
Framingham, MA 01701
USA
508.872.8200
Twitter: @IDC
idc-community.com
www.idc.com
Copyright Notice
This IDC research document was published as part of an IDC continuous intelligence service, providing written
research, analyst interactions, telebriefings, and conferences. Visit www.idc.com to learn more about IDC
subscription and consulting services. To view a list of IDC offices worldwide, visit www.idc.com/offices. Please
contact the IDC Hotline at 800.343.4952, ext. 7988 (or +1.508.988.7988) or sales@idc.com for information on
applying the price of this document toward the purchase of an IDC service or for information on additional copies
or web rights.
Copyright 2017 IDC. Reproduction is forbidden unless authorized. All rights reserved.