Design and Implementation of a Flexible and Extensible Data Processing Runtime

학위논문 (석사)-- 서울대학교 대학원 : 공과대학 컴퓨터공학부, 2018. 2. 전병곤.Today's data analytics applications take a wide variety of characteristics. They are also executed in various resource environments, with many distinct requirements. To face these requirements, many systems have been developed with optimization techniques that are suitable for each system's needs. However, the field of data processing is continuously growing with diverse requirements for job characteristics and resource environments. With current system designs which demonstrate pre-defined runtime behaviors, it is extremely difficult to apply new optimization techniques to them. Onyx is a system that approaches to solve this problem by designing and implementing a flexible and extensible execution runtime. The Onyx execution runtime is designed and implemented around the execution properties that must be flexibly controllable and extensible in order for jobs to be executed under the desired runtime behaviors. It uses a user configurable job representation, Onyx IR, annotated with execution properties which control the underlying runtime behaviors for each job to flexibly execute jobs according to users' requirements. Examples and evaluations show that new optimization techniques are easily applicable to Onyx, which otherwise require a significant amount of engineering effort using current data processing systems.Chapter 1 Introduction 1 Chapter 2 Background 4 2.1 Data Processing Concepts 4 2.2 Optimizations for Data Processing 5 2.3 Pre-Defined Runtime Behaviors 7 2.4 Execution Properties 10 Chapter 3 Onyx Overview 12 3.1 Onyx IR 12 3.2 Optimization Passes 13 3.3 Submitting to the Execution Runtime 14 Chapter 4 The Execution Runtime 16 4.1 Design 16 4.2 The Processing Backbone 19 4.3 The Flexible and Extensible Execution Properties 21 Chapter 5 Examples 30 5.1 Push Optimization for Small Scale Workloads 30 5.2 Harnessing Transient Resources: Pado 31 Chapter 6 Evaluation 35 6.1 Small Scale Workloads 35 6.1.1 Experimental Setup 35 6.1.2 Results 35 6.1.3 Discussion 37 6.2 Harnessing Transient resources 38 6.2.1 Experimental setup 38 6.2.2 Results 38 6.2.3 Discussion 39 Chapter 7 Conclusion 40 Bibliography 42 국문초록 45Maste