Adaptive Scheduling Applied to Non-Deterministic Networks of Heterogeneous Tasks for Peak Throughput in Concurrent Gaudi

As much the e-Science revolutionizes the scientific method in its empirical research and scientific theory, as it does pose the ever-growing challenge of accelerating data deluge. The high energy physics (HEP) is a prominent representative of the data-intensive science and requires scalable high-throughput software to be able to cope with associated computational endeavors. One such striking example is GAUDI -- an experiment independent software framework, used in several frontier HEP experiments. Among them stand ATLAS and LHCb -- two of four mainstream experiments at the Large Hadron Collider (LHC) at CERN, the European Laboratory for Particle Physics. The framework is currently undergoing an architectural revolution aiming at massively concurrent and adaptive data processing. In this work, I explore new dimensions of performance improvement for the next generation GAUDI. I then propose a complex of generic task scheduling solutions for adaptive and non-intrusive throughput maximization in the multithreaded task-based realization of GAUDI. Firstly, I present the reactive graph-based concurrency control system for low latency and scalable task precedence resolution in unpredictable data processing conditions. Secondly, I demonstrate the outstanding potential of the proactive task scheduling approach that employs the technique of induced avalanche concurrency leveling for throughput maximization in conditions of tight task precedence constraints. Finally, I investigate the synergy of latency oblivious task scheduling and controlled oversubscription of central processors. I prove a significant potential of this merger for throughput maximization in general and for the adoption of the paradigm of heterogeneous computing in the context of GAUDI in particular. I exemplify all ideas in the LHCb event reconstruction scenario.CERN report number: CERN-THESIS-2016-02