Today, IT companies do not exist without specialised tools. These can be commercial solutions, open source, as well as those created internally. At VSoft, when creating systems for our customers, we rely on the internal low-code platform VSoft archITekt. This helps us to implement solutions faster, in a more flexible and optimal way. Over these 26 years, we have created dozens of systems for a variety of industries. A major success is that we create many of them together with our customers. We have developed procedures for working on projects simultaneously, so that such cooperation accelerates the intellectual and business growth of both teams.
When executing dedicated projects, we practically always have to deal with data migration. It takes place once or iteratively several times – depending on the type, quantity and quality of the data. This time it was not easy, we migrated several thousand insurance policies from the old system to our VSoft Insurance Platform.
The data to be imported was provided in the form of Excel files, separate for each object type. This number may not scare you, but when it turned out that the migration also involved plenty of policy-related objects, such as additional risks, items, entities, amounting to over 30,000, things got interesting.
The dream with any migration is to have good data quality, adjusting the dictionary fields in the input files. This time it was only a dream.
We commenced work… in the beginning 99% of the records were rejected, it quickly became apparent that the client had not adjusted the fields according to the assumptions. There was little time, hence a quick decision, validation changes and it was done. For the migration, we used an ETL-class tool called the mass data processing module, which is part of the low-code VSoft archITekt platform. Thanks to the application of this tool, the execution time of the migration is about 4 minutes 30 seconds. The module uses so-called processing packages, which are configured easily and quickly using a graphical designer. The tool generates quality reports on an ongoing basis and provides detailed information on the progress of the migration and the quality of the data, making it easier to decide on the next migration steps during the process.
Why did we choose the ETL tool, the mass data processing module? As it has a number of advantages and enhancements useful for data migration, in particular:
- has frequently worked well for this (and other) tasks – it is a modern, proven, ready and reliable technology;
- is intuitive and based on a graphical designer, with which the blocks for loading input data, transforming data and saving processed data are configured accordingly;
- has a clear and user-friendly graphic designer, which facilitates not only the implementation of the solution (creation of packages), but also testing and diagnosing problems (e.g. data view blocks, graphic presentation of loaded processing logs);
- contains a CLI runner, sufficient to run the processing package without the whole graphical part – a lighter and ample solution to launch on the target environment, interesting for both developers and process designers;
- gives the possibility, if necessary, to define more complex operations using the built-in programming language with syntax prompting;
- has a user-friendly way of defining validation rules and error handling.
During this migration, the mass processing module of the VSoft archITekt platform facilitated the task at hand.
The following improvements were particularly helpful:
- a graphic designer – can prompt and warn against non-optimal combinations of data sources and storage locations (e.g. warning against database deadlocks);
- a temporary change of format for input files from Excel to CSV – when the need arose, this was achieved in a second by changing the type of input block in the mass processing module;
- a change of data validation requirements – when it became necessary at the last moment of the tests, we managed it with a few clicks
Today, IT companies do not exist without tools. These can be commercial solutions, open source, as well as those created internally. When creating systems, VSoft relies on the internal low-code platform VSoft archITekt. This helps us to create faster, in a more flexible and optimal way. Over these 26 years, we have developed dozens of systems for a variety of industries. A major success is that we create many of them together with our customers. We have established procedures of simultaneous development, such cooperation means substantial intellectual and business growth of both teams.
Are you facing a data migration challenge or want to tell me how you are dealing with similar issues? Write to me on LinkedIn.
