6 Things to Consider Before Starting a Data Migration Project

xLM Solutions, LLC, is a company with vast experience in data migration. We have performed migrations from various sources (legacy/migration from) to a variety of destinations (new/migration to) systems. Our migration projects have ranged from straightforward, vanilla migrations to extremely complex ones involving whole business transformations. Our migration history includes dealing with a multitude of different systems such as SmarTeam, ENOVIA V6 (MatrixOne), 3DEXPERIENCE SolidWorks PDM, Autodesk Vault, DBWorks, SharePoint, flat files and more. Having been involved in so many migrations and encountering issues of every nature, we know what a customer considering a migration needs.

Data Migration

This blog discusses the six things to consider before starting a data migration project.

Mappings

How will data be mapped from the source system to the destination system? When dealing with PLM migrations, we see such mappings mainly in the following areas:

  • Object Type Mapping. This type of mapping uses different names based on the system in question. For instance, an object type would mean Class in SmarTeam, Type in 3DEXPERIENCE or Item Type in Aras. In order to plan a successful migration, the migrator needs to understand how different types of data/objects will be mapped to corresponding types/objects in the source system.
  • Field Level Mapping. Usually each object type has various fields associated to it containing meta-data. The company, with input and direction from the migrator, will identify the field mappings from the source fields to the destination fields.
  • State Mapping. This concept deals with how the source data lifecycle states; i.e. WIP, Pending, Release, etc., map to the destination system lifecycle states.
  • Revision Schema Mapping. Also an important issue; do the source and destination systems have the same revision pattern or are they different. Also, do both systems assign revisions at the initial creation of the object or are revisions only assigned upon release of the object
  • Data organizations. This incorporates concepts like projects, folders and workspaces. How will the migrated data be organized in the destination system.

In most cases, mappings are not one-to-one and logic may need to be put in place to determine the rules to map data from the source system to the destination system. Depending on the scope of the migration project, such mapping rules can be determined in as little as a one-hour phone call or may require an on-site assessment to analyze and develop a more complex mapping plan. To further complicate things, the destination system may not be fully configured and any initially defined mapping rules may change as the system evolves and gets implemented. Accordingly, the migration needs to be flexible enough to recognize such changes are inevitable and be able to easily make modifications to accommodate them.

Data Clean-up

In migrations, we encounter data in one of two ways:

  • source system data that is corrupted in such a manner that when migrated could cause the destination system to malfunction
  • bad data due to poor data entry that should be cleaned up but would not necessarily break the destination system

In regard to the first problem, the person conducting the migration should understand the conditions that would cause the source system not to work and be able to run reports or other tools to identify such bad data. Once the bad data is identified, options can be explored on how to fix it. Such options may include simply deleting the bad data, manually fixing it or possibly developing automated tools or queries to fix it.

For the second problem, bad data could mean missing fields, duplicate part numbers, and incorrectly labeled data. In this case, such ”bad data” scenarios should be identified. With that information in hand, reports can be written to clean up the data.

It is essential for the migrator to work closely with the company to determine which scenarios to look for and which rules can be automated to fix bad data.

Bussing Process Rules/Data Translation

When conducting a data migration from one system to another, it is generally an appropriate time to re-evaluate the company’s business processes to determine if they can be managed more efficiently. Such re-engineering of process may cause the new/destination system to be modelled drastically differently from the source system. In these cases, data translation rules/logic need to be put in place to “transform” the existing source data into a format that the new destination system can handle.

Two simple examples of this are:

  • In the source system, there are three fields to enter the description into (Description 1, Description 2 and Description 3). But, in the new system there is only one description field (Description). A rule can be put in place to concatenate the description values from the three fields in the source system to the one field in the destination system.
  • When modifying and renaming physical files using a new naming convention, especially when dealing with CAD files, any referencing files must have their references updated to point to the newly named file.

Testing

Testing is one of the most critical parts of the migration. No migration is ever completed perfectly the first time. The customer needs to account for multiple migrations and have resources available to test out the migration. Testing can be done by simply sampling a number of records in the destination system. Obviously, the larger the sample that is checked, the higher the confidence level achieved. Not only should the sample data be viewed, but actual day-to-day operations should be performed on the data, such as check in and out, moving the data through a workflow/release process, etc. Test plans should be used and documented and followed through multiple test cycles to identify any regressions. Performance needs to be evaluated as well (make sure such tests are performed on production level hardware).

Tools/report and logging can be developed to validate data as well, however these can never replace actual user testing and sign off. For instance, the migration tool should report situations where data could not be migrated and logged to be analyzed for further inspection. Post migration reports can compare the number of records in the source system to the records in the destination (factoring any data translation that may be done).

Production Cut-Over Planning

It is important to plan the steps to cut-over from the original source system to the new production system. An initial step is estimating the time it will take to migrate the data. For instance, does the company understand that it may encounter down time while the migration takes place? In many instances, migrations occur over a weekend when most users are not using the system. However, there should be a plan in place if the migration is not concluded over a weekend. The migrator must inquire whether the system can be down for a longer period or should the migration process be re-scheduled. Perhaps increasing the computer processing power, performing migration tasks in parallel, identifying a better technology (API code vs. direct database updates, for example) or splitting the migration into delta loads.

Other topics to consider:

  • Who will install the necessary software on the end user computer for go live?
  • When will the software be installed?
  • How and when will the end user be trained to use the new system?

Finally, it will be necessary to schedule a date when all the necessary resources required for the final cut-over and migration are available to perform their duties.

Technology to Use

In our opinion, these first 5 points are the most difficult to accomplish; i.e., the developing of rules and plans for how the migration will work. However, once the rules are established, it is then necessary to ascertain which technology will be used to actually perform the migration. Are there off-the-shelf products that can be used? Or would custom code need to be developed to perform the migration? Perhaps only SQL queries can be used to extract the data. Since a migration involves two sides – the source side and destination side – you need to make sure that whoever is performing the migration understands and has the experience with both systems.

When xLM Solutions is involved, it is our approach to break such migrations down into two parts: namely, a data extraction from the source system and a data import into the destination system. We usually use a combination of existing tools and custom code we have developed. This allows us to modify our tools to meet specific customer requirements. We find the two-step process to be the most reliable with better recovery and roll back options in case of problems. Logging and monitoring the process are better and more efficient this way.

Conclusion

Data migrations can be quite complex. It is not as easy as saying I want the data in system X to show up in system Y…though some companies think there is not much more to it than that. Indeed, you need to carefully review the requirements of every migration and plan out all the various details involved in it. This requires both business and process skills along with technical skills. Having extensive experience with such migrations goes a long way in allowing the migrator to anticipate various issues and being able to properly educate the customer as to what must be considered. The experienced migrator remains in communication with the customer beyond the planning stage – guiding them on the project, conducting the migration and being involved in the testing and going live phases.

Marc Young, managing partner of xLM Solutions published this post in his LinkedIn profile.