Optimizing the tracking of software logs and their analysis with a centralized system for management

Tema diplomske naloge posega na področje zbiranje in analize dnevniških zapisov iz različnih sistemov in njihovo konsolidacijo. Naloga analizira različne načine sledenja aplikacijskim in baznim sistemom ter poizkuša najti najbolj kvaliteten način konsolidacije dnevniških zapisov v skupnem centraliziranem sistemu. Sistem mora biti robusten za različne tipe aplikacij, od spletnih strežnikov, ki svoje zapise vpisujejo v podatkovne baze, do sistemov, ki v svojem formatu zapise shranjujejo znotraj dnevniških datotek. Če bi želeli v sistem vpeljati nov tip dnevniškega zapisa, mora biti način vpeljave čim bolj lahek in hiter, ne glede na njegovo sestavo. Sistem mora biti razširljiv na potencialno veliko število dnevniških zapisov, zato se kot osnova indeksiranja in iskanja uporabi sistem Apache Lucene, ki deluje na podobni tehnologiji kot NoSQL in je primerno za iskanje po neurejenih podatkih s pomočjo JSON formatiranja dokumentov. Celoten sistem je sestavljen iz treh večjih delov. Prvi je črpanje podatkov iz različnih lokacij na skupno mesto. Drugi del je analiza podatkov, ki jih spremeni iz osnovne oblike v strukturo, ki je enakovredna vsem ostalim tipom podatkov. Tretji in zadnji del pa je prikaz podatkov na uporaben način končnemu uporabniku. Implementirali smo tudi centraliziran sistem z uporabo orodja OpenSearch, ki skrbi za iskanje in analizo zapisov, Logstash, ki upravlja pretok podatkov iz njihove izvorne lokacije do sistema OpenSearch in pa skrbi za ustrezno razčlenjevanje dnevniških zapisov ter sistem OpenSearch Dashboards, ki ponuja grafični vmesnik za prikaz zapisov in vizualizacijo analiz.The topic of the bachelor\u27s thesis delves into the realm of collecting and analyzing log entries from various systems and their consolidation. The thesis analyzes different ways of tracking application and database systems, aiming to find the most qualitative method of consolidating log entries in a unified centralized system. The system must be robust for various types of applications, from web servers that log their entries into databases, to systems that store their records in log files. If one wishes to introduce a new type of log entry into the system, the integration method should be as easy and quick as possible, regardless of its composition. The system must be scalable to a potentially large number of log entries, so Apache Lucene is used as the basis for indexing and searching. It operates on technology that is similar to NoSQL and is suitable for searching through unstructured data using JSON-formatted documents. The entire system consists of three major parts. The first is data extraction from various locations to a common place. The second part is data analysis, which transforms them from their basic form into a structure equivalent to all other data types. The third and final part is displaying the data in a user-friendly manner to the end user. We also implemented a centralized system using the OpenSearch tool, which handles the search and analysis of records, Logstash, which manages the data flow from their original location to the OpenSearch system and takes care of the appropriate parsing of log entries, and the OpenSearch Dashboards system, which offers a graphical interface for displaying records and visualizing analyses