Data transformation

Data transformation is the process of restructuring data from one format to another. It is a type of data management procedure that involves taking data from one form and converting it into a different form. Generally, the original data must be reorganized, restructured, or reformatted in some way in order to fit the desired output. The goal of data transformation is to make the system’s data usable, understandable, and uniform across the board.

Data transformation contains two loosely coupled activities: the extraction of data from one form, and the loading of the extracted data into another form. For example, extracting data from a relational database, such as SQL, and then loading the data into an analytics platform, such as Apache Hadoop cluster. The data transformation process involves applying data mapping, cleansing, and enrichment techniques.

Data mapping is the process of specifying a correspondence between elements in two data types. The goal here is to ensure that data elements in the original data are properly mapped to the appropriate ones in the new data format. This step also includes the creation of new unique identifiers, if necessary. Cleansing refers to the process of ensuring that all data elements meet a certain predefined set of values or integrity. This may include standardizing formats, eliminating corrupt data, and adding missing data items. Lastly, enrichment involves using additional data sources to add more context and value to the data.

Data transformation is an important part of data management and is often used to make existing data usable, understandable, and uniform. It is a complex process which often requires data mapping, cleansing, and enrichment, all of which can help to add value to the newly transformed data.