Log aggregation and analysis of interconnected services in near real time

Tema ovog diplomskog rada je izrada sustava za prikupljanje, obradu i prikaz dnevničkih zapisa međusobno povezanih servisa u skoro realnom vremenu. Na samom početku su obrađeni osnovni pojmovi vezani uz tokove podataka poput vremenskih domena, prozora i vodostaja. Zatim slijedi opis osnovnih funkcionalnosti tehnologija koje su korištene prilikom implementacije spomenutog prototipa sustava. Među njima se nalaze Apache Kafka, Spark Streaming i Spark Structured Streaming. Korisničko sučelje i web aplikacija ostvareni su korištenjem Vue.jsa i Dropwizarda. Sami sustav je testirana na dvama različitim tipovima dnevničkih zapisa, aplikacije za naplatu parkinga i nginx web poslužitelja sustava Edgar. Ostvaren je protočan cjeloviti cjevovod koji garantira „točno jednom” od odredišta do izvora, uz malo kašnjenje i ispravne rezultate.The topic of this thesis is building a system for collecting, processing and displaying logs of interconnected services in almost real time. At the very beginning, the basic concepts related to data flows such as time domains, windows and watermarks are described. Then follows a descriptions of basic technology functionalities used in the implementation of said prototype system. These include Apache Kafka, Spark Streaming and Spark Structured Streaming. The user interface and web application were achieved using Vue.js and Dropwizard. The system itself was tested on two different log data types, of a parking billing app and the Edgar system’s nginx web server. A complete flow pipeline is achieved that guarantees "exactly once" from the destination to the source, with little delay and correct results