So you have started your customer insights or BI modernisation journey and need to migrate the data from a legacy database into the new target database or data warehouse. Most likely, the migration is from a legacy on-premise application and database into modern, cloud-based application and data store, with all the new capabilities and benefits that brings. If you are doing a data migration project for the first time, it might seem as easy as a simple lift-and-shift, but if you want to avoid any nasty surprises, read on…
Acrotrend has done numerous Data Migration projects and from these experiences, have built a step-by-step approach and methodology to ensure smooth data migration and seamless business transition into the new and modern data capabilities. Even though some of the steps may not seem necessary (especially if you are up against a tight deadline), omitting any of them significantly increases the risk of failing the most fundamental aspect of the migration – trustworthiness of the migrated data.
(1). First things first – you need to understand the legacy datastore you are planning to migrate/modernise. Typically, the system must have been there for 5 years or more, in some cases I have worked on for 10 years even, and holds many different types of business data. The best way to understand what that data is and how the relationships between various data sets are maintained is to map out the data model of the existing database. A high level logical data model of the database that calls out the business data entities managed, with base data attributes and derived fields should suffice. If there are any entities to track the health of the database – data quality, error records, that will be useful to map now and analyse later.
(2). Source system list for legacy data. List out all the systems that feed data into this database as well as all the systems that consume the output of this database. Make a list of all the data entities that are exchanged from each system, frequency, and any high level transformations in between. Include all the legacy systems that are already shut down as well, this will help analysis of the data and certain migration decision and agreements. A good block diagram of the systems landscapes will be helpful. Agree on which data sources will be required/desired to feed into the new datastore, and ones which are redundant/not required anymore.
(3). Reports/insights generated from legacy system. List down all the KPIs and Reports generated from this data, with the data requirements for KPI calculations, their target audience/distribution list along with frequency of report publish. Do not forget to check if any of the data is shared with external vendors/partners/agencies for marketing or compliance purposes.
(4). Target database data model. Inmost cases you will already have this, but if not, you need to map out the data model for target state, including the data formats, data entities structure (in most cases, table and columns, facts and dimensions etc). Understand and agree on the gaps and differences between the legacy and target data models.
(5). Historical data migration requirements. Conduct a high level data analysis to get an idea of the quality and completeness of the data, its business and legal relevance in context of the new datastore, objectives and purpose. This is an important step to determine what data needs to be moved, which can be left behind or archived, any cleansing/dedupe requirements, traceability and credibility of data after migration etc. Specifically in aged systems, you might find some patches of data that can’t just be explained from where it came from, and in absence of a justified reason, you can leave it behind and not clutter or confuse the new system. In all likelihood, the target data model will be different from legacy data model, so you need to agree on data entity mappings and also any required data and format transformations in between.
(6) Ongoing data feeds into the new datastore. From the list of data systems identified and agreed to feed into the new system, map out the data entities and transformations required to maintain the cohesiveness on historic and new incoming data, whilst being able to store and consume new data entities into new datastore. Now is the time to make any changes in the data processing and transformation logic that might not worked so well in the legacy systems.
(7). Select you migration approach. There are 2 typical approaches to migration – big bang and trickle. Both have their own advantages and dis-advantages, but depending on various parameters of like number of source systems feeding the database, business criticality of the final data store, importance of completeness of the unified data, acceptable downtimes, migration timeline/duration expected etc, and your findings from the above steps, you need to decide and agree on which approach will work best for your case.
(8). Plan B: Always have a lifeline available as plan B. No matter which approach you decide to proceed with for migration, its a good practice to take a backup copy of the legacy data with the cut-off label, this will be helpful in testing the migration as well. And more importantly, you might want to consider running both legacy and new system in parallel for some time before you shunt the old one out.
(9). Execute and test. Execute your migration plan at a time when there is least disturbance on your BAU and also enough time to do sanity test and business UAT test. Sanity can be simple checks on counts of key business data entities, formats of timezones, currency etc, local/international language characters, and so on. Business UAT should include the reports and KPI metrics and trace-ability and relevance to legacy numbers/counts as well as newer KPIs, capabilities and objectives.
(10). Switch-over to the “New”, Run the legacy and the new system in parallel for a few weeks/months and continue to sanity test. This will give time to the new system to stabilise, not just from infrastructure and technology perspective, but also from business adoption and trust point of view. Any business queries can be answered to satisfaction by comparing again the legacy system and explaining the trace-ability, differences and reasons. This also ensures that if any capability or report is not developed yet on the new system, then that can possibly be delivered and fulfilled from the old one, until a right longer term decision is made. In about 3 months, you should be able to cut-off and decommission the legacy system (not before you take your last backup though) and have had moved everyone onto the new system seamlessly.
Above steps are mere ingredients for Data Migrations; For them to work successfully, you need the secret recipe – which is business engagement all through. Remember any data/system migration is not a technical project alone, it will need to be accepted, adopted and trusted by the business users, so keeping the conversations on at every step of the migration project will ensure deep connection and understanding from every stakeholder of the new system and hence result in higher acceptance and better experience.