PRESS INFORMATION - FOR IMMEDIATE RELEASE - 26.04.2004

What is the Real Cost of Data Migration?

Data Migration projects are often undervalued and misunderstood. In this article, Ken Tomlinson (Progressive Edge Services), a member of the Solass Associates Program (S.A.P.), examines the reasons for this and considers how best practice experiences can be used to avoid this recurring problem.

There are many businesses today making major investments in new information management technology. Some of these investments are for strategic business improvements because the time is right for the introduction of the new technology in-line with organisational change. Others are being forced by circumstances such as company mergers that require a globalisation of the disparate business systems, or by vendors withdrawing support of older PDM systems. 

Whatever the reason there is one common factor in all of these projects, the legacy information has to be migrated into the new environment. However, a common theme is becoming all too apparent, the vast majority of the data migration projects are:

• Significantly overrunning.

• Costing much more than originally budgeted for.

• Creating a lasting bad impression over the entire business improvement programme.

So why are these unfavourable characteristics recurring? The complexity and importance of a data migration is generally not well understood and, unfortunately, the repercussions of this only become apparent far too late in the project. Let us examine the evidence.

The Value of Information

For many companies much of the value delivered by the organisation is represented by the technical information – its intellectual property. This is a key asset, but its value is generally not recognised resulting in the information not being treated with the respect it deserves. Underestimating the importance of how the information is best structured is a common pitfall in many Information Management implementations. Data migration implementations frequently fall into this trap. The data is mapped with little consideration given to the rules under which it was operating. 

The Benefits of Migration

Understanding the benefits of the migration to the business is also a problem area, especially if the need for migration has been forced by circumstance rather than driven by opportunity. In fact, even asking a business what the benefits of the current system are can lead to interesting results. Tangible measures of benefit can be hard to obtain, but a general feeling that the system is beneficial is usually forthcoming. 

Even harder to understand may be the benefit of migration to the user community, where migration to a new system is actually a hindrance to their daily operation. 

Underselling Migration Projects

Migration projects are generally undersold; a key starting point in our impending downstream problems. You can’t blame the vendors, why add further complication to the already long and drawn-out sales process? It is not in their interest to tell you about the problems associated with data migration when they are selling the benefits of their new system. 

Migration is also a back-end process; it results because of something else happening, such as a new PDM implementation. Migration is not usually considered up front in the implementation plans. A common management misconception regarding data migration is that it is expected to happen automatically.

Data Migration is Complex

Far from happening automatically, data migration is a complex combination of technical and organisational problems which need to be understood, harmonized, and effectively controlled throughout the project.  

All too often, migration is viewed as simply transferring data fields. However, a migration cannot be deemed to be complete until all the required data and associated processes have been transferred, they are operational and are providing the same level of business support in the target system as they did in the legacy system. 

The complexity of a particular migration project depends upon a number of different interrelated factors. In order to assess difficulty, we will consider five separate axes on a complexity map dealing with the organisation, the processes and the systems. Let’s take a brief look at each of these. 

Legacy Environment – This deals with all the complexities associated with the legacy environment from where the information for migration originates. There are many different areas to consider here, logically breaking down into contents, systems and responsibilities issues, for example:

• How well documented or understood are the legacy systems? Are the required experts still available?

• How many different legacy systems are involved? Are there other data sources that should be considered?

• Are the correct skills and resources readily available to build the utilities?

• How clean is the legacy data? What processes do we need to check the data?

Target Environment – This deals with the environment into which the migrated information is being transferred. In addition to the contents, systems and responsibilities we also need to consider issues associated with harmonising the information into the developing target solution, for example:

• How stable is the target implementation data model? Can all legacy functions be migrated?

• Can all the legacy data be mapped to the target system data model?

• Are the delivery timescales clearly defined? How well is the target implementation project seen to be progressing?

Impact on the Business – This deals with how the migration project interacts with the business as a whole. The ideal scenario would be to continue with business as usual, but there will inevitably be some impact on the business during the migration period. We break these down into expectations, impact and organisational issues, for example:

• Are the benefits of migration fully understood? Is there a willingness to change?

• Is the business prepared for downtime? What is the right time for this?

• Is the business infrastructure in place to support the migration process?

Project Organisation – This deals with the management and operation of the project. Getting the right level of management involvement to deal with the complex issues is seen as a major factor in the success of the project. Examples of the issues considered here are:

• What is the project organisational diagram? Are the correct stakeholders in place?

• The project involves disparate teams of people, who are they? What are their skills? What are the reporting lines?

• Is there a plan for how to enhance and/or change the data during the migration?

• What are the external constraints on the migration project?

• Are the success criteria identified and understood?

• Is there a fall-back scenario built into the plan?

Technical Solution – This deals with technical aspects of the migration solution covering migration implementation, operation and transformation issues. Some examples of the issues considered here are:

• Type of migration process, incremental or big bang? What business rules need to be applied to support the chosen process?

• Type of solution, point to point or staged solution?

• Should new or alternative technologies be considered?

• Amount of information involved? How long will the process take?

• Complexity of the information involved? Does the data mapping cover all requirements?

The Consequences

After just a brief consideration of the five axes, the complexity of migration becomes much clearer. Data Migration projects are often undervalued and misunderstood and the consequences are:

• Migration is an afterthought in the overall programme but can seriously impact the timescales for success.

• The migration project is not correctly budgeted or planned.

• The resulting technical failures lead to direct implications on the implementation as a whole. For example: data is not ready for use when the target system is rolled out, the new functions do not work on the old legacy data or constant errors are found in the migrated data.

• The programme as a whole loses credibility and management backing which leads to far more serious business consequences.

What Can We Do?

The key to this conundrum is to raise awareness of the benefits and opportunities to the organisation, the true value of the data, and the complexities of migration. Once the business is aware, proactive project management and supporting techniques can be used to operate the project efficiently and professionally. Firstly, plan the migration project early to avoid duplication of effort and add value: