How Change Data Capture Can Fuel Dynamic Data Movement

Your enterprise refuses to stand still. Progress in meeting strategic goals sees new projects, services and applications arriving all the time with data spreading between them, often on an ad hoc basis. Even in a relatively static business, data is moving from local servers or data centers to the clouds and warehouse as application trends change and businesses focus on predictive analytics and other AI-enhanced operations.

As data movement accelerates across this continuously shifting infrastructure and application landscape, database administrators need to keep on top of the trends and patterns, as well as the applications that support the movement of data in more efficient and useful ways.

What is dynamic data movement?

Dynamic data movement is a term defined by IDC as part of their effort to link data integration and intelligence software (and streaming data pipeline software) to business outcomes. Dynamic data movement is increasingly used to replace bulk data movement between databases as a more efficient (read non-batch) way to deliver data to applications and those users that need it across the enterprise.

In a world of increasingly constant data flows, codeless platforms, next-gen data warehouses, data lakes and other innovations, the need to move the latest and right data to the correct place dynamically is more efficient than shuffling huge databases or batches of data between data centers, warehouses and clouds.

With growing workloads across multiple services or vendors’ products, dynamic data movement supports data replication to heterogenous databases. This enables one-to-many vs Oracle-to-Oracle operations without losing integrity.

With many different types of data movement across various platforms and products, including ETL (extract, transformation, load), ELT (extract, load, transform), data replication and change data capture, dynamic data movement represents a leading way to deliver cost savings, cut down on multiple systems and shadow IT repositories and help get rid of legacy IT systems.

What is change data capture?

Change data capture (CDC) enables enterprises to gain visibility into the changes that happen to their databases in real-time or as quickly as required. This information can be sent downstream to other data sources, helping them keep current, ideal for analytics and data science tasks across clouds. And for enterprises migrating from data centers to clouds, CDC helps keep the new clouds up to date until the originals are switched off.

Consider the typical database, where only 1% to 2% of records change over an active period. These records and their changes are the ones that the business and teams are interested in, and CDC can identify the records, update those in the data warehouse version of the database for analysis and crunch the numbers to deliver the latest insights. CDC services can use triggers such as logs, audit columns, table deltas and other methods to identify change, triggering an update.

CDC is a more modern take on ELT which uses slower, bulk efforts to maintain raw data retention by building an archive that can be used to generate business intelligence.

The benefits of CDC include:

Reduction or elimination of the need for batches of bulk uploads, replaced by live or incremental loading as required to the destination

• Support real-time analytics and data science efforts
• Support the synchronization of data across geographies
• Log-based CDC limits the workload on the source database
• Supports downtime-free database migrations
• Efficiently help move data across WANs and clouds
• Supports stream processing database efforts
• Maintains synchronization across multiple databases and services

How CDC helps achieve dynamic data movement

CDC can play a key role in data integration, but the applications are still relatively new in a market dominated by ETL/ELT products. Using CDC, especially modern solutions can easily integrate with most databases including Oracle, Azure or AWS, and work with ETL tools to reduce the workload.

With the power of high-performance ingestion and update features, a single CDC tool can deliver on the enterprise’s real-time data integration and data warehousing needs.

The faster data moves and the more current all data stores are the better for business operations and strategic decision making. In cases such as fraud detection, live operations are critical to preventing banking, retail or commodities frauds.

And, as enterprises explore the boundaries of data science, AI-based analytics or moving data across edge networks in IoT facilities, the need for always on-time data is vital in keeping those recommendations flowing and accurate.

Final thoughts

In the modern business environment, data wants to be free and more users want access to it, even if those users are bots or analytics tools. Using data warehouses and services that don’t interrupt the flow of the source databases is key to these efforts. CDC represents the latest approach to ensure your enterprise clouds and analytics apps can flow at full speed without waiting for a batch upload to trigger.

Supporting the move to high speed and low fault tolerance as you replicate between databases, SharePlex by Quest supports data operations to upgrade, migrate and improve system performance, while improving availability, scalability and analytics across the enterprise.