PySpark at Bare-Metal Speed

Data analytics has become the driving force for many industries and scientific research. More and more decisions are maid based on statistical analysis of large datasets and machine learning. Big data data processing frameworks, such as Apache Spark, provide an easy-to-use out-of-the-box solution, scalable to large machine clusters. Python is the most widespread programming language in the data science field due to its simplicity and the abundance of analytical tools developed for it. Many Spark users would prefer its Python frontend in their daily work. Multiple studies indicate, however, that there is a wide gap between Spark’s performance and the best handwritten code. With this thesis we bring the functional data-flow programs’ performance closer to the bare-metal speeds and show that it is possible to write productively high performance code