A System for Scalability Assessment of Apache Beam Platform in Processing Large Volumes of Spatio-Temporal Data

S povećanjem broja IoT uređaja i podacima koje dobivamo od njih, novi način obrade tih podataka je trebao biti stvoren. Ti alati koji procesiraju tokove podataka dali su ogromnu prednost nad tradicionalnim bazama podataka. Postoji velika lista alata koje nam pružaju rad s tokovima podataka, pa je pitanje koje od njih odabrati. Kako smo mi limitirani i ne može se očekivati da ćemo se naučiti sve te različite alate i načine kako pisati programe specifično za njih, solucija je došla u obliku platforme Apache Beam koja je dala unificirani model koji se može spojiti na veliki broj izvođača. S Apache Beamom jedina stvar o kojoj se moramo brinuti je kako ćemo napisati program, ne kako će se taj program izvoditi na nekom izvođaču u pozadini. Ovaj rad se fokusira na pokušaju pokretanja program na što više izvođača i testirati njihovu performansu. Sustav je bio testiran nad 3 izvođača Apache Flink, Apache Spark i Hazelcast Jet. Testiranje je pokazalo da postoji limitacija u pitanju koliko možemo povećati broj jezgri i čvorova kako bi dobili bolje performanse. Također je pokazalo limitaciju u Jetu, gdje ne može čitati više od limitiranog broja entiteta nad Kafka topicom s particijama.With an increase of IoT devices and the data that they provide, an entirely new way of processing that data had to be created. These tools for processing data flows in real time, gave a huge advantage as opposed to traditional databases. There exists a large list of tools that can provide us with the processing capabilities, so it begs the question of which to choose. As we are limited with time and can't be expected to learn all the different tools and the ways in which we can create a program for them, Apache Beam came with a solution with a unified model that would connect a large set of runners. With Apache Beam, the only thing we need to care for is what we are implementing, not how it is going to run in the background on a runner. This theses focuses on trying to execute a program to test the throughput of the tools as well as the throughput of the system on as many runners as we can and test their performance. The system was tested with Apache Spark, Apache Flink and Hazelcast Jet. The tests showed that there was a limit to how much you can increase the number of nodes and their CPU core counts to increase their performance. It also showed a limitation of Jet whereby it cannot read more than a limited number of records from partitioned Kafka topics