OptiETL

Since the beginning of the information age in the last century, the size of the data collected and stored electronically throughout our daily life has grown dramatically. Especially with the advent of the internet, the data explosion we are experiencing in social networks, search engines, telecom, financial companies and universities is simply astonishing. The increase in the sheer size of data also complicates the efforts to organize and access it. As the data volume is increasing, the demand for accessing, aggregating and analyzing information to make sense out of it has grown incredibly. This thesis presents a distinct approach and a unique application, OptiETL, aimed to simplify the work and effort to get data from source systems for those who do not specialize in data extraction and transfer, such as researchers, analysts and even software developers. It also provides implementation details including proof of concept, software design, and specifications of the OptiETL. The thesis presents challenges with large volume data transfer, delta data capture, and the data extraction, transformation and load process known as ETL. Furthermore, it provides a detailed usage guide for the proposed OptiETL tool includes a comparison with similar available tools or applications on the market and, lastly, summarizes possible enhancements on the project as future work