The CDO's Toolkit for Overcoming Data Migration Challenges

Today's enterprises hold more data than ever. With these assets continuing to grow at an exponential rate, it's vital you have the right processes and technologies to effectively manage them. And, sooner or later, the existing solutions you have in place for handling this data aren't going to be adequate.

Conducting a data migration to more modern and suitable solutions is therefore something every firm will have to be ready for. Often in today's environment, this means a cloud migration, but these projects don't necessarily involve moving to the cloud. Shifting data from a legacy application to a new system or transferring databases from outdated arrays to more up-to-date hardware within your data center also fall under this category.

However, no matter what type of migration it is you're planning, this can be a process fraught with difficulty. According to research by Experian, for instance, just 46% of these projects are completed on time, while only around a third (36%) are achieved within budget. In some cases, total failures of data migration plans can cause major disruption if they leave employees unable to access vital resources easily.

With this in mind, here are some of the most common data migration mistakes you're likely to face, and how you can overcome these challenges.

1. Insufficient planning

Many of the issues firms may encounter throughout the process can be traced back to a failure to draw up a clear and comprehensive data migration strategy long before the first byte of data is transferred.

Not having a plan is the root cause of many time and budget overruns as firms fail to appreciate the scale of the task in front of them. For instance, one critical step must be to identify exactly where all your relevant data is and in what format. Today's businesses often have huge amounts of data spread around the world that, if missed, can add time and expense to the project.

Having the right people and expertise in place is also vital. Deloitte reported that a lack of understanding of technology is one of the leading obstacles to successful data migration, with 44% of professionals citing this as an issue. Therefore, it's essential you take the time to bring in the right skilled personnel, who understand both the existing and target systems and see where any bottlenecks may lie and what changes, if any, will be required to data formats.

2. Failure to engage with business units

Treating a data migration as purely an IT project is another common mistake. Your IT pros may understand the reasons behind shifting to a new platform or system, but if the people actually using the data day-to-day don't, they're likely to be frustrated by any disruption this causes.

This can result in a range of problems, from individuals continuing to enter data in since-discontinued formats to personnel not being able to find critical information when they need it.

Involving business units from the start can also help identify any potential issues that may come up, and allow you to work proactively to resolve them. It's likely to be the case that data will need to be cleansed, merged or restructured in order to fit the new system, but this could cause significant knock-on effects for the people who actually use this data every day.

3. Not analyzing your source data effectively

A failure to understand the status of your original data can lead to a range of issues. Duplicated data, for instance, can cause major headaches when migrated into new systems, especially if it comes from disparate sources. This not only adds unnecessary time to the project, but can cause confusion if multiple records end up in the same system.

Similarly, disparities in how data is formatted, errors or misspelling in original files and gaps in information can also add time and complexity to the project, so it's essential you're aware of where these issues lie and have a plan in place to address them when scoping out budgets and timelines.

You should be particularly wary of less structured data elements. For example, a database may have specific fields for certain categories of information, but also have a more open-ended section where users could add notes or additional information. If so, how will this be replicated in the new system?

4. Not testing and validating your implementation

Another common mistake is assuming that once you have migrated your data, this should be considered a success. Testing and validating the system to ensure any changes you've made to your data are reflected properly is a must. This should be done using full-volume, real-world data to cover all possible scenarios. Using limited samples of data may be more convenient, but it could lead to you missing potential problems.

Testing should take place throughout the process, not only at the end. For example, if there’s no clear strategy for this in place, you may discover far too late that your data is incompatible with the new system. Make sure everyone knows who's responsible for this in your initial data migration plan, as well as who will sign off on the results.