Karavan, ETL pipeline management system based on Apache Spark

One of the responsibilities of the Data Engineering Team is to make ETL pipelines to Extract the data, Transform it to the Desired information and schema, and then persist it in specific storage or provide it as an API. For example, a stream of data of user behavior that comes to the Kafka topic should be transformed to the desired schema and persist this raw data into the Data Lake of the company. Besides that, there is some stream processing that prepares the final information. Consider we want to have the number of online users visiting a specific product in an e-commerce website and we want to show the user the number of concurrent visitors on this specific page for the last 5 minutes. By growing the needs of different consumers of the Data Engineering Team such as Data Science, BI, and Marketing, the number of these pipelines will grow exponentially, and managing them will be harder. For example, we want to change our logging storage and we decide to keep the logs all in ElasticSearch. If we have 100 hundred pipelines we should change the code of all of them. There are different kinds of Pipeline management systems Like StreamSet and Apache Nifi which have solved this problem but they also have their own problems to use. In this project, we tried to make an Apache Spark-based pipeline Management System that allows us to keep tracking the pipeline structure and empower all Data Engineering Team to make their own pipeline by a Human Readable structure even without having great knowledge about spark and stream processing and also SQL