UNDERSTANDING AND OPTIMIZING PYTHON-BASEDAPPLICATIONS - A CASE STUDY ON PYPYYANGGUANG LIA THESIS SUBMITTED TO THE FACULTY OF GRADUATE STUDIESIN PARTIAL FULFILMENT OF THE REQUIREMENTSFOR THE DEGREE OFMASTER OF APPLIED SCIENCEGRADUATE PROGRAM IN ELECTRICAL ENGINEERING ANDCOMPUTER SCIENCEYORK UNIVERSITYTORONTO, ONTARIOJULY 2019© YANGGUANG LI, 2019AbstractPython is nowadays one of the most popular programming languages. It has beenused extensively for rapid prototyping as well as developing real-world applications.Unfortunately, very few empirical studies were conducted on Python-based appli-cations. There are various Python implementations (e.g., CPython, IronPython,Jython, and PyPy). Each has its own unique characteristics. Among them, PyPy,which is also implemented using Python, is generally the fastest due to PyPy’sefficient tracing-based Just-in-Time (JIT) compiler. Understanding how PyPy hasbeen evolved and the rationale behind its high performance would be very usefulfor Python application developers and researchers.This thesis is divided into two parts. In the first part of the thesis, we conducteda replication study on mining the historical code changes’ of PyPy and comparedour findings against Python-based applications from five other application domains:Web, Data processing, Scientific computing, Natural Language Processing, andMedia. In the second part of the thesis, we conducted a detailed empirical studyiion the performance impact of the JIT configuration settings of PyPy. The findingsand the techniques in this thesis will be useful for Python application developersas well as researchers.iiiTable of ContentsAbstract iiTable of Contents ivList of Tables viiiList of Figures xi1 Introduction 12 Related Work 52.1 Code Diffing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2 Code Change Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 62.3 JIT Compiler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.4 Understanding and Tuning the Configuration Settings . . . . . . . . 83 The Replication Study 11iv3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.1.1 Chapter Organization . . . . . . . . . . . . . . . . . . . . . . 133.2 Python Source Code Diffing . . . . . . . . . . . . . . . . . . . . . . 143.2.1 Code Parser . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.2.2 GumTree Differencing . . . . . . . . . . . . . . . . . . . . . 193.2.3 Action Parser . . . . . . . . . . . . . . . . . . . . . . . . . . 203.2.4 Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243.2.5 Tool Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 263.3 Summary of original study . . . . . . . . . . . . . . . . . . . . . . . 283.4 Replication Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 343.4.1 (RQ1) Across Projects . . . . . . . . . . . . . . . . . . . . . 343.4.2 (RQ2) Across Versions . . . . . . . . . . . . . . . . . . . . . 373.4.3 (RQ3) Maintenance Activities . . . . . . . . . . . . . . . . . 383.4.4 (RQ4) Dynamic feature . . . . . . . . . . . . . . . . . . . . 413.5 Threats to Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . 443.5.1 Construct Validity . . . . . . . . . . . . . . . . . . . . . . . 443.5.2 Internal Validity . . . . . . . . . . . . . . . . . . . . . . . . 443.5.3 External Validity . . . . . . . . . . . . . . . . . . . . . . . . 453.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45v4 Assessing and Optimizing the Performance Impact of the Just-in-time Configuration Parameters 474.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474.1.1 Chapter Organization . . . . . . . . . . . . . . . . . . . . . . 504.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514.2.1 An Overview of the JIT Compilation Process . . . . . . . . 514.2.2 PyPy’s JIT Configuration . . . . . . . . . . . . . . . . . . . 544.3 Exploratory Study . . . . . . . . . . . . . . . . . . . . . . . . . . . 554.4 Automatically Tuning the JIT Configuration Parameters . . . . . . 774.4.1 Tailoring MOGA for JIT Configuration Tuning . . . . . . . 794.4.2 Our Performance Testing Framework . . . . . . . . . . . . . 904.4.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . 914.5 Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 924.5.1 Case Study Setup . . . . . . . . . . . . . . . . . . . . . . . . 934.5.2 Case Study Results . . . . . . . . . . . . . . . . . . . . . . . 974.6 Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1004.6.1 Optimal Configurations across Different Systems . . . . . . . 1014.6.2 Top Configurations across Different Workloads . . . . . . . . 1054.6.3 Code Jitting vs. Performance . . . . . . . . . . . . . . . . . 1084.6.4 JIT vs. Memory Usage . . . . . . . . . . . . . . . . . . . . . 114vi4.6.5 Termination criteria . . . . . . . . . . . . . . . . . . . . . . 1174.7 Threats to Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . 1194.7.1 Construct Validity . . . . . . . . . . . . . . . . . . . . . . . 1194.7.2 Internal Validity . . . . . . . . . . . . . . . . . . . . . . . . 1214.7.3 External Validity . . . . . . . . . . . . . . . . . . . . . . . . 1224.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1235 Conclusions and Future work 125Bibliography 127viiList of Tables3.1 Identified changes in 10 sampled commits of PyPy. . . . . . . . . . 283.2 Comparisons between the original study and the current study . . . 313.3 Comparisons between the original study and the current study (con-tinued) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.4 Comparisons between the original study and the current study (con-tinued) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334.1 PyPy benchmark programs description. . . . . . . . . . . . . . . . . 594.2 Comparing the response time between the warmup phase (A) andthe warmed up phase (B). . . . . . . . . . . . . . . . . . . . . . . . 654.3 List of relevant PyPy’s JIT configuration parameters and their in-formation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 674.4 The JIT configurations chosen for performance evaluation. . . . . . 684.5 Number of best performing programs/scenarios under each JIT con-figuration setting. . . . . . . . . . . . . . . . . . . . . . . . . . . . 71viii4.6 Comparing the jitting performance against the configuration underJIT off. The number of programs/scenarios whose performance un-der jitted configuration is worse or no different than JIT off settingis highlighted in bold. . . . . . . . . . . . . . . . . . . . . . . . . . . 734.7 Comparison between the configuration settings yielded the best per-formance and the configuration settings resulted in the most jittedcode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 764.8 Dominance relations among the four configuration settings in ourrunning example. “” means the the left configuration setting dom-inates the right configuration setting, “≺” means the right config-uration setting dominates, and “≈” means there is no dominancerelation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 864.9 An overview of the three Python-based systems under study. . . . . 924.10 Workload description for the three systems. . . . . . . . . . . . . . 944.11 Statistics after running the MOGA approach on the three case studysystems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97ix4.12 Comparing the performance between the optimal configuration set-tings and the default configuration setting for the three systems. Theoptimal configurations are labelled as OA, OB, and OC. WS standsfor the web server, and DB stands for the database. “-” means theESM-MOGA suggested optimal configuration setting outperformsthe default setting and “+” means otherwise. . . . . . . . . . . . . 984.13 Top three optimal configuration settings for the three studied system.1024.14 Spearman correlation between configuration and response time. Thelarge and very large correlation measures are shown in bold. . . . . 1024.15 Comparing the performance among the optimal configuration set-tings under different workloads for Wagtail. . . . . . . . . . . . . . 1064.16 Top three optimal configuration settings for Wagtail under differentworkloads. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1074.17 Configuration A and the default configuration. . . . . . . . . . . . . 1144.18 Spearman correlation between number of jitted line and memoryusage for each system. . . . . . . . . . . . . . . . . . . . . . . . . . 1154.19 Runtime statistics for ESM-MOGA under different termination cri-teria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116xList of Figures3.1 Overall process of PyDiff. . . . . . . . . . . . . . . . . . . . . . . . 153.2 An example of converting Python source code to Python AST . . . 163.3 An sample converting Python AST to JSON AST . . . . . . . . . . 173.4 An example of converting JSON AST to LAT . . . . . . . . . . . . 183.5 Two sample statement with their corresponding ASTs and mappings 193.6 Two sample statement with their corresponding ASTs and mappings 213.7 Fluri’s Taxonomy of Change Type . . . . . . . . . . . . . . . . . . . 253.8 The distribution of change type frequency across projects. . . . . . 353.9 The frequency comparison between different versions of PyPy. . . . 383.10 The frequency comparison between bug-fix commits and non-bugfixcommits of PyPy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403.11 The change frequency of each dynamic feature. . . . . . . . . . . . . 423.12 The change frequency of each dynamic feature. . . . . . . . . . . . . 434.1 A sample PyPy code snippet with the jitted code marked as “(*)”. 51xi4.2 Number of jitted lines and response time over 50 iterations for thehtml5lib program from the PyPy benchmark suite. . . . . . . . . . . 604.3 Overall process of PyPyJITTuner. . . . . . . . . . . . . . . . . . . . 784.4 The workflow for our tailored version of the MOGA method. . . . . 804.5 Visualizing the response time distributions of different scenarios un-der different configuration settings for Wagtail. . . . . . . . . . . . . 984.6 Number of jitted lines and overall average response time among allevaluated JIT configuration settings in Saleor. The red dotted lineshows the overall average response time with JIT turned off. . . . . 1094.7 Two code snippets showing the executed code and the jitted codeunder the two configuration settings: A vs. B. Configuration A isa jit-enabed configuration shown in Figure 4.6. It has worse perfor-mance than configuration B, which is JIT off. The colour schemeis defined as follows: grey coloured code is for not executed code;bolded black coloured code is for executed but not jitted code; andhighlighted bold coloured code is for jitted code under configurationA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112xii1 IntroductionPython is nowadays one of the most popular programming languages [51]. Dueto its dynamic features, rich set of library and framework, and large community,Python is also used quite often during rapid prototyping [77], especially in thefields of web (e.g., [4, 6]), data analysis (e.g., [11, 9]), and machine learning (e.g.,[20, 18]) as well as developing many real-world business systems in many large scaleand mission-critical systems inside companies like Facebook [66], Google [26], andPayPal [55].There are many empirical studies done for other programming languages, likeJava [79, 45, 42, 38, 59] and JavaScript [53]. However, very few works focused onPython, except the works of [69] and [71]. Studying Python applications havebeen increasing important, as many popular applications and libraries were writ-ten in Python. There are various Python implementations (e.g., CPython, Iron-Python, Jython, and PyPy). Each implementation has its own unique characteris-tics. For example, Jython is a Python interpreter implemented in Java and it can be1fully integrated into Java applications, IronPython is tightly integrated with .NETFramework. Among those implementations, PyPy is generally the fastest [14] dueto PyPy’s efficient tracing-based Just-in-Time (JIT) compiler [31] and it is imple-mented with Python. Understanding how PyPy has been evolved and the rationalebehind its high performance would be very useful for Python application developersand researchers.There have been very few studies on the evolution of Python-based applications,except the work of Lin et al. [69], the authors conducted an empirical study aboutfine-grained source code changes on ten Python application across five domains(Web, Data processing, Science computing, NLP, Media). However, the studydid not included any applications from the domain of compiler, whose evolutionpatterns may or may not be the same as other application domains. For example,dynamic features bring great flexibility during development, but are generally slowerto execute. PyPy, which is focused on high performance, would its JIT compiler usemany dynamic features compared to other application domains? The first part ofthis thesis will be focused on replicating the above study on the PyPy developmenthistory.PyPy mainly gains its performance via its JIT complier [14]. For dynamiclanguages, JIT compiler is being used to improve the efficiency of compiling andexecuting the source code. For example, Java’s HotSpot JVM [10] can compile2methods based on its execution frequency. JIT compiling is also used in Chrome’sV8 engine [2], which can compile JavaScript code into machine code during run-time. Existing works on the JIT compilation focus on the jitting strategies (e.g.,method [38] vs. trace-level based code jitting [31]), speeding up the process of theJIT compilations [63, 47, 70], optimizing the performance of the underlying virtualmachines [88, 89, 90], and detecting JIT unfriendly code [53]. Unfortunately, thereare very few existing studies which investigate the impact of the JIT configurationson the system performance. Software configuration is one of the main sources ofsoftware errors [92]. The configuration settings of a software system can signifi-cantly impact its performance. Optimizing the configuration settings may resultin great performance gain. Hence, the second part of the thesis will be focused oninvestigating the performance impact of PyPy’s JIT configurations.The contributions of this thesis are:1. We have developed a change extraction tool, PyDiff, which can extract fine-grained historical changes from the Python source code.2. We have replicated the empirical study in [69] by comparing our results fromPyPy (a language interpreter written in Python) against 10 Python-basedapplications from five other domains (Web, Data processing, Science comput-ing, NLP, Media) and found that 6 out of 9 findings are different from the3original study.3. This is the first empirical study on assessing and optimizing the impact ofthe tracing-based JIT configuration settings on system performance.4. In this thesis, we have detailed our search-based configuration tuning ap-proach, (ESM-MOGA) for tuning applications running PyPy.5. Our experiments on JIT configurations are carried out on both the syntheticbenchmarks as well as real systems. The empirical findings can be useful forboth software engineering and programming language researchers as well aspractitioners.Thesis Organization:The thesis is organized as follows: Chapter 2 introduces the related work. Chap-ter 3 presents our empirical studies on the historical code changes in PyPy andcompared our findings to the findings in the original study. Chapter 4 studied andoptimized PyPy performance by tuning its JIT configuration settings. Chapter 5concludes the thesis and discusses some future work.42 Related WorkIn this chapter, we discuss prior research works that are related to this thesis.2.1 Code DiffingSoftware projects are changed rapidly to meet the constantly evolving customerrequirements and deployment environments. To help developers better understandthe changes in different code commits, many tools have been developed to extractthe find-grained code changes by comparing the historical versions of the sourcecode.J-REX [79] is an evolutionary extractor which can analyze Java-based projects.It uses the token-based technique to compare the changes between each consecutivefile revisions and outputs the code revisions in terms of function additions, functiondeletions and function updates. Fluri and Gall presented a taxonomy of source codechanges to be used for change coupling analysis in his work [43]. Fluri et al. [45]also presented the original Abstract Syntax Trees (AST) differencing algorithm,5ChangeDistiller, and described inadequacies concerning the extraction of sourcecode changes. The work of Falleri et al. [42] mainly described their novel treedifferencing algorithm, the GumTree differencing algorithm, which is more efficientand accurate. However, neither GumTree nor ChangeDistller provided a mechanismfor diffing among Python-based source code. Lin et al. [69] implemented anautomatic tool to extract fine-grained source code changes in Python code, namedPyCT, based on ChangeDistiller. However, the implementation of PyCT is notavailable for download. In this thesis, we have developed a fine-grained code diffingtool, called PyDiff. Our tool extracts the ASTs from different versions of thePython-based applications and leverages the GumTree’s differencing algorithms tooutput the fine-grained code changes between two commits.2.2 Code Change AnalysisIn the work of Lin et al. [69], the authors conducted an empirical study on 10Python projects across 5 domains. They tried to investigate the pattern of changetypes across project, revisions and maintenance activities. And they provided in-sights about the change of Python dynamic features. Similar as the work of Linet al. [69], Romano et al. [75] developed the tool WSDLDiff to extract fine-grained source code changes between different versions of WSDL interfaces andthey analyzed the evolution of WSDL interface based on the fine-grained source6code changes in four real world web services. [44] used ChangeDistiller to extractchange types from each version of the software and formed a matrix with whichthey can group changes between versions into different clusters and explore thepatterns in each cluster. Thung et al. [86] combined code analysis and machinelearning approaches to analyze the bug fix changes. Their approach can predictthe code line that caused the bug and the result out-performed the state-of-art.However, there is no existing evolutionary study on Python-based compilers, whichis the focus of Chapter 3 of the thesis.2.3 JIT CompilerJIT is introduced as a technique to improve the system behavior during runtime bycompiling the frequently executed (a.k.a., “hot”) code snippets into binaries [31, 73].Currently, there are two general approaches on recognizing and compiling hot code:(1) the method-based jitting approach [38], which compiles the whole hot method;and (2) the trace-based jitting approach [31], which only compiles the frequently ex-ecuted code path(s) within one method. Both techniques have their pros and consand are adopted by different programming languages. The code jitting processtakes a while to recognize and compile the hot code snippets [30]. Hence, varioustechniques have been proposed to speed up the JIT compilations [63, 47, 70]. Sinceonly portions of the source code are jitted, during runtime, depending on the ac-7tual execution path(es), the system may switch between the native mode (a.k.a.,executing the compiled binaries) and the interpreter mode. Gong et al. [53] de-veloped a technique to detect performance anti-patterns that prohibit the systemto execute certain portions of the code natively. Our paper differs from the aboveworks, as it focuses on the configuration settings of the JIT compiler. The closestwork was done by Hoste et al. [59], which proposed a search-based technique toautomatically tune the Java compiler. Although the two programming languagesdiffer in their jitting techniques (method-based JIT for Java and tracing-based JITfor PyPy), both [59] and this paper reported the need to automatically tune JITconfigurations, as the optimal JIT configurations are system and workload depen-dent. [59] even found that the JIT configuration tuning is hardware dependent.In this paper, we further studied the characteristics of the PyPy jitting behaviorand tried to derive general patterns/guidelines on tuning the JIT configurations.For example, we have found a high correlation between the amount of jitted codeand memory utilization. Generally, the configuration parameter decay should beset with a small value and a large value in the trace limit.2.4 Understanding and Tuning the Configuration SettingsSoftware configuration settings play an important role in the performance of asoftware system. However, there can be many configuration parameters, each of8which has various possible settings. Hence, the overall configuration space for onesystem can be huge. In this subsection, we will discuss the related works in thearea of assessing and optimizing the configuration settings for one system. We havedivided the existing techniques into the following three categories: Understanding the Performance Impact of Various ConfigurationParameters Through Experimentation: Not all configuration parame-ters can impact the system performance. Hence, researchers have devised aset of experiments with various combinations of the configuration settings toassess the impact of the configuration parameters [33, 84]. Various experi-mental design techniques (e.g., screening design [93], and covering array [58])have been applied to assess the performance impact of various configurationparameters. These techniques require a much smaller set of experiments thanexhaustively enumerating all the possible combinations of the configurationsettings, while still able to identify the high performance impacting configu-ration parameters. Modeling System Performance Under Different Configuration Set-tings: Instead of isolating the impact of various configuration parameters,another approach to assess the performance impact of configuration settingsis through performance modeling. Siegmund et al. [81] predicted the system9performance by detecting performance-relevant feature interactions. In theirlater work [80], Siegmund et al. leveraged machine learning and samplingheuristics to build performance models, which can describe the performanceinfluences among different configuration options and their interactions, froma small set of experiments. Libicˇ et al. [68] used queuing theory to modelthe performance of the JVM garbage collectors (GCs). Singer et al. [82] builtmachine learning models using the data from the existing configurations ofthe GCs. Recently, Jamshidi et al. [61] proposed to use the transfer learningtechnique to model and infer the system performance under different config-uration settings. Automated Tuning of the Configuration Settings: There are two gen-eral approaches to automatically tuning the configuration settings of a soft-ware system: (1) through reduction of the possible candidates of optimal con-figurations (e.g., based on the similarities among configuration settings [74]or through iterative experimentations [85]); (2) through the use of the search-based algorithms (e.g., hill-climbing [91, 87], ParamILS [67], or multi-objectivegenetic algorithms [59, 83]).However, there is no existing works focusing on assessing and tuning the per-formance of PyPy. This is the foucs of Chapter 4 of the thesis.103 The Replication StudyIn this study, we have performed an evolutionary study on the PyPy project. Inparticular, we compared our findings against one existing evolutionary study onPython-based appications [69].3.1 IntroductionIt has been well understood that software has to be adapted to changing require-ments and environments. The source code changes between different versions canprovide developers insight into how a software project evolves. Researches in miningsoftware repositories are becoming more and more popular.There are many works which studied the evolution in source code changes inprojects. Romano et al. [75] analyzed the evolution of the WSDL interface basedon the fine-grained source code changes in four real-world web services. [44] extractchange types from each version of the software and formed a matrix with which theycan group changes between versions into different clusters and explore the patterns11in each cluster. However, few of these work studied the evolution of Python-basedapplications, except the work of Lin et al. [69], the authors conducted an empiricalstudy about fine-grained source code changes on ten Python application acrossfive domains (Web, Data processing, Science computing, NLP, Media). However,the study did not include any applications from the domain of compilers, whoseevolution patterns may or may not be the same as other application domains.For example, dynamic features bring great flexibility during development but aregenerally slower to execute. PyPy, which is focused on high performance, it wouldbe interesting to verify if the JIT compiler use as many dynamic features as otherapplication domains.In order to study the evolution of the Python-based applications, tools to extractthe fine-grained differences between different commits are needed. For example, J-REX [79], ChangeDistiller [44], and GumTree [42] can extract fine-grained Javasource code changes. Unfortunately, they do not support Python. PyCT [69] canbe used to extract fine-grained changes from Python source code and is basedon ChangeDistiller’s AST differencing algorithm. But its implementation is notavailable publically. In this project, we leveraged the framework from PyCT andthe GumTree differencing algorithm and developed PyDiff, which can extract fine-grained historical changes from python source code. We conducted a replicationstudy which analyzes the source code evolution for PyPy, which is a Python-based12application in the domain of interpreter. The contributions of this chapter are:1. We have developed a change extraction tool, PyDiff, which can extract fine-grained historical changes from the Python source code. This tool is publiclyavailable for use [13].2. We have replicated the empirical study in [69] by comparing our results fromPyPy (a language interpreter written in Python) against Python applicationsfrom five other domains (Web, Data processing, Science computing, NLP,Media) and we found that 6 out of 9 findings are different from the originalstudy.3.1.1 Chapter OrganizationThe rest of the chapter is organized as follows: we introduced the implementationof our Python source code diffing tool, PyDiff, in detail in section 3.2. Section 3.3gives a summary of the original study and an overview of the findings. Section 3.4describes the findings in our replication study and discusses the implications. Sec-tion discusses the threats and validity and Section 3.6 concludes the chapter.133.2 Python Source Code DiffingPyDiff analyzes two consecutive revisions of source code and provides fine-grainedchange information between these two revisions. The source code changes are cate-gorized into many change types to help software engineer understand how the codegot changed. Fig 3.1 gives us an overview of PyDiff. The input of PyDiff is histor-ical revisions of a project, which is usually stored in the Code Repositories (e.g.,CVS, SVN, and git.). Within PyDiff, there are three components: the Code Parser(Section III-A) parses Python source code into abstract syntax tree (AST) that canbe processed by GumTree; the GumTree Differencing (Section III-B) performs theGumTree differencing algorithm on two ASTs and generates an edit script whichcontains Actions must be performed on the first AST to obtain the second one;and, the Action Parser (Section III-C) groups these Actions according to their linenumber and identifies the change type for each group by analyzing the Actions inthat group. The output of PyDiff is a CSV file which contains change types foreach changed line between two consecutive revisions.3.2.1 Code ParserThe Code Parser is responsible for converting the Python source code to AST thatcould be analyzed by GumTree. GumTree has the functionality to directly convert14 Figure 3.1: Overall process of PyDiff.Java, C, and C++ source code into a Tree that it can process. However, no sucha function has been developed for Python source code. Fortunately, GumTree canaccept JSON data as input and can convert JSON data into a Language-AgnosticTree (LAT) format which it can also analyze. Therefore, we can bring the gapbetween Python source code and LAT by converting Python AST into JSON ASTand use the JSON AST as input for GumTree. The Code Parser takes the followingfour steps to convert Python source code to LAT that can be analyzed efficientlyby GumTree:3.2.1.1 Python Source Code to Python ASTPython has a module, named ”ast”, which can generate an AST from python sourcecode. We can use it to parse the Python source code to Python AST easily. Theresult will be a tree of objects whose classes all inherit from AST. Figure 3.2 gives15Figure 3.2: An example of converting Python source code to Python ASTan example about how to convert a Python statement ”a = 1” to Python AST.3.2.1.2 Python AST to JSON ASTJSON data structure is similar to XML. In JSON, each node is constructed by aname/value pair, in which name is a string and value could be number, string, listor another node. For each node in Python AST, we can get its class name as well asits child nodes. This structure is very similar to JSON data structure. Therefore,when we covert Python AST to JSON AST, we use the class name of the node asname and use its child nodes as value to construct a field in JSON data structure.By converting all nodes in Python AST, we can obtain a JSON AST that keeps theoriginal structure of Python AST. Figure 3.3 shows the JSON AST for statement”a=1”.16Figure 3.3: An sample converting Python AST to JSON AST3.2.1.3 JSON AST to LATAs we mentioned above, GumTree provides an API called TreeGenerator whichcan take JSON data as input and convert JSON data into a LAT. We can useTreeGenerator from GumTree to generate LAT from JSON AST. Figure 3.4 showsa LAT generated by TreeGenerator from JSON AST of statement ‘a = 1’. In theLAT, each node has two fields: node type and label. Node type 4, 10, 15 16 do nothave a label because they donate array, field, object, and number in JSON dataseparately. And node type 17 means string and node type 14 means numbers inJSON.17Figure 3.4: An example of converting JSON AST to LAT3.2.1.4 Simplify LATAs shown in Figure 3.4, the above parsing steps could introduce many uninforma-tive nodes. For example, a field node will be added to LAT for each field in JSONAST, an object node will be added to LAT for each object in JSON AST. Theseuninformative trivial nodes make our LAT redundant. They can also introducemisclassification in the GumTree algorithm and bring a large number of uninfor-mative Actions in GumTree’s differencing results which are hard to interpret. Toaddress this issue, we simplify the LAT by eliminating the uninformative nodes,like field nodes, object nodes, array nodes, number nodes, string nodes, etc. Thesimplified LAT of statement ‘a = 1’ can be seen in Figure 3.5. In the simplifiedLAT, each node has two fields: node id and label.18 a = 1 Figure 3.5: Two sample statement with their corresponding ASTs and mappings3.2.2 GumTree DifferencingThe GumTree Differencing component takes two LATs as input and applies theGumTree differencing algorithm to these two LATs. It can compute a sequence oftree edit actions that transform a LAT into another. GumTree Differencing worksin two steps: establishing mappings and deducing an edit script. The mappingbetween two LATs will be computed by two successive phases. 1. A greedy top-down algorithm to find isomorphic subtrees of decreasing height. Mappings areestablished between the nodes of these isomorphic subtrees. They are called anchorsmappings. On the sample trees of Fig 3.6, this step finds the mappings shown withdashed lines. 2. A bottom-up algorithm where two nodes match (called a containermapping) if their descendants (children of the nodes, and their children, and soon) include a large number of common anchors. When two nodes match, it finallyapplies an optimal algorithm to search for additional mappings (called recoverymappings) among their descendants. On the sample trees of Fig 3.6, this step findsthe container mappings shown using short-dotted lines. And there are no recovery19mappings in this example. Then the mapping results can be used by the RTEDalgorithm [36] to compute the actual edit script. It will generate a tree edit scriptwhich contains Actions that must be performed on the first LAT to obtain thesecond LAT. And those Actions can be used to identify fine-grained source codechange types. There are four types of tree edit actions: Insert, Delete, Update andMove. In the example of Fig 3.6, the edit script generated contains only one Action,which is “update label ’1’ to label ’2’ in node 5”.Then the mapping results can be used by the RTED algorithm [36] to computethe actual edit script. It will generate a tree edit script which contains Actionsthat must be performed on the first LAT to obtain the second LAT. And thoseActions can be used to identify fine-grained source code change types. There arefour types of tree edit actions: Insert, Delete, Update and Move. In the exampleof Figure 3.6, the edit script generated contains only one Action, which is “updatelabel ‘1’ to label ‘2’ in node 5”.3.2.3 Action ParserGumTree differencing can only give us changes between two ASTs, because GumTreeignores the semantic meanings of each node in the LAT when it’s performing thedifferencing. Therefore, those changes cannot tell us anything about the sourcecode changes, which is useless in software analysis. To gather source code level201:"Assign"2:"targets"3:"a"4:"value"5:11:"Assign"2:"targets"3:"a"4:"value"5:2Tree 1:“a = 1”Tree 2:“a = 2”{"Assign": {"targets": [{"Name": {"ctx": "Store", "id": "a"}}], "value": {"Num": {"n": 1}}}}Figure 3.6: Two sample statement with their corresponding ASTs and mappingschange types, we introduced Action Parser to interpret Actions from the GumTreedifferencing result and obtain source code level change types for all changed lines.This can be done in two steps:3.2.3.1 Map source code line with ActionsEdit script contains a sequence of Actions from different code lines. With just oneAction, we cannot identify the change type for a code line correctly. Therefore,we need to aggregate Actions that are taken in each code line by its line number.For each Action in the edit script, it contains node information of the node thathas been changed. And the line number of that node can help us identify whichstatement that action belongs to.However, the line number can represent the line in the source code before a21change and it can also represent the line in the source code after a change. There-fore, we cannot simply map the line number back to the before source code linenumber. In GumTree differencing, the source of the line number is determined bythe type of action. The line numbers of changed nodes in Update, Delete and MoveActions represent line number in before source code. The line numbers of changednodes in Insert Actions represent line number in after source code. In order to dealwith the difference, Action Parser processes Insert Actions and other Actions sep-arately by dividing them into before Actions and after Actions. Therefore, it hastwo groups of Actions which represent the changes made to before and after AST.And it aggregates those two groups of Actions by their line number and builds amap of the source code line and the Actions.3.2.3.2 Identify change typeIn Fluri’s work [43], they presented a taxonomy of source code changes that canbe used for change coupling analysis. The source code change types are definedaccording to tree edit operations in the AST. And this classification allows one toassess error-proneness of source code entities, qualify change couplings, or identifyprogramming patterns. The defined change types are shown in Table 3.7. Althoughthe change types are mainly defined for Java programming language and there aremany change types not applicable for Python, we took it as a guideline and defined22our change type taxonomy base on the characteristic of Python. The third columnin the Table 3.7 indicates whether a change type is applicable for Python or not.As shown in Table 3.7, ’Additional Object State’ and ’Removed Object State’are not applicable because Python doesn’t have attribute declaration. Since Pythondon’t have attribute declaration in Python, change types, like ’Decreasing Acces-sibility Change’, ’Increasing Accessibility Change’, ’Attribute Type Change’, ’At-tribute Renaming’, ’Final Modifier Insert’ and ’Final Modifier Delete’ are also notapplicable. In method declaration of Python, we cannot define the return typeof a method, thus ’Return Type Insert’, ’Return Type Delete’ and ’Return TypeUpdate’ are also not applicable. Base on Fluri’s taxonomy of change types, wedefined our own fine-grained change types according to our needs. For example, inFluri’s work, he identifies a class declaration statement as a normal statement andinserting or deleting the class declaration will be regarded as ’statement insert’ or’statement delete’ action. Therefore, we defined two additional actions ’AdditionalClass’ and ’Removed Class’ to help me understand insert and delete changes madeto class declaration statements. And we grouped defined fine-grained change typesinto eight groups: Class Change, Function Change, Statement Change, SelectionStructure Change, Loop Structure Change, Exception Handling Change, ImportChange, and Others Change.For each changed statement, Action Parser identifies the change type for the23changed statement by analyzing Actions taken under the statement. For eachchanged statement, it can analyze its aggregated Actions by two steps. It firstlyidentifies if there is any Action made to the root node of that statement. If theroot node of the statement is changed, the change type of the statement would bestatement level Insert, Delete or Move, which is determined by Action type. If it’snot a statement level change, it will identify a detailed change type according tothe type of statement and the action type. If these Actions do not contain changesto the statement’s root node, the change type would be statement Update. And amore detailed change type can be determined by doing an analysis of the Actions.From the example in Fig 3.6, Action Parser can obtain an action “update ’1’ to’2’ in node 5”. According to the strategy, node 5 is not a root node for the variableassignment statement. Therefore, it can roughly infer that the change type for thestatement is statement update. Then, it does further analysis on the change typeby combining the statement type, which is assigned statement in our case, andthe action made to the statement. And it can determine the change type for thatstatement is ’statement update’.3.2.4 OutputIn order to help researchers understand the changes in the software, PyDiff willoutput all changes made to a file throughout the software’s life circle into a CSV24 a = 1 Figure 3.7: Fluri’s Taxonomy of Change Type25file. For each change, it will contain information about change type, file path,before revisionID, after revisionID, changed line number, changed location.3.2.5 Tool EvaluationIn this section, we will explain how our experiment was designed to evaluate PyDiff.In order to test the robustness of PyDiff, we need to choose well-maintained, longhistory project to evaluate PyDiff. Thus we chose the Python project PyPy toevaluate PyDiff. PyPy has more than 17 years of development history. And it’sopen-source on Bitbucket, so we can easily obtain all the historical revisions. Basedon SVN logs, PyPy was first committed on Feb 24, 2003. With more than 17 years’development history, PyPy contains 97,000 committed revisions and about 4000python source files. And the evaluation process is as follows:1. Extracting PyPy history source code from SVM: In order to extract allchanged files in each commit, we implemented a small tool to extract changedsource files in the current commit and its parent commit by three steps: a. Ex-tract the change log, which contains information about changed files, changetype of the file, commit id, revision id, etc., for each commit. b. Recursivelyextract changed file’s source code for each commit and its parent commit. c.Create a JSON file for each commit in PyPy and store changed source codein the JSON file. In this way, we can extract the changed source code for all26commits, on which we can apply PyDiff.2. Applying PyDiff on PyPy source code: After we obtained the historical sourcecode, we used them as input for PyDiff. PyDiff will run the differencingalgorithm on two consecutive source code and generate the change type foreach changed line between these two source code files. Then, PyDiff will storeall changes in that commits into a CSV file.3. Manual Examine: After obtaining the output for all commits in PyPy, weevaluated the differencing results by conducting a manual examination. Be-cause there are more than 97,000 commits in PyPy, it is not practical toexamine the output of all commits. Therefore, we conducted a manual exam-ine on 10 sample commits. For each selected commit, we manually checked ifthe identified change type matches the source code change or not and recordthe number of correctly identified changes.After manually checking the source code change result from 10 randomly se-lected commits, we measured the number of changes correctly identified and mis-classified in each change type. Table 3.1 shows us the result of the examination. Aswe can see, PyDiff identified 293 lines of changes among the 10 randomly selectedcommits. Statement change is the most common change type which takes up 59percent of all source line changes and achieved 98.8 percentage of accuracy. And27Table 3.1: Identified changes in 10 sampled commits of PyPy.Change Type Total Misclassified Accuracy(%)Class 3 1 66.6Function 31 1 96.7Statement 174 2 98.8Selection Structure 32 1 96.8Loop Structure 6 0 100.0Exception Handling 43 0 100.0Import 2 1 50.0Others 2 0 100.0All 293 6 97.9Import change and Other changes occurred the least in the examined commits.Overall, the PyDiff can achieve a precision of 97.9 percent for all changes types.3.3 Summary of original studyIn the work of Lin et al., the authors conducted an empirical study about fine-grained source code changes in Python applications (Django, Tornado, Pandas,Pylearn2, Numpy, Scipy, Sympy, Nltk, Beets, Mopidy) from different domains28(Web, Data processing, Science computing, NLP, Media). In their study, theyfirst defined 77 kinds of fine-grained changes types and then they developed an au-tomatic tool - PyCT, which can quickly compare Python source code and extractfine-grained source code changes. The author proposed four research questions cov-ered different aspects of the source code change behavior throughout the project’shistory. Then, the authors applied PyCT on ten open source Python projects fromfive different domains and answered the proposed research questions. Based onthe source code diffing result, the author reported 11 major findings, as shown inthe Table 3.4, summarised below (Note: Original Fining is denoted as F and Newfinding is denoted as NF).First, they compared the change type frequency between projects and domains.They found that Function and Statement changes are the most frequent changetypes and Loop Structure changes is the least frequent (Finding 1). Also, thedistribution of change types shares similar trends between projects (Finding 2) anddomains (Finding 3).Second, they use Scipy and Pandas as case study and compared the change typefrequency between different versions of a project. In project Scipy, they found thatthe change type frequency distribution is significantly different between bug-fix andnon-bugfix versions (Finding 4) and language evolution may have a big impact onsoftware evolution (Finding 5). In project Pandas, they found the change type29frequency distribution are similar across different versions (Finding 6).Then, they conducted a detailed comparison for change type frequency be-tween bug-fix and non-bugfix commits, in which they found that if structure relatedchange types are more related to bug-fix activities (Finding 7). And the non-bugfixactivity related change types are import statement and function structure changes(Finding 8).Finally, they investigated the dynamic feature changes. The result shows thatisinstance, type, hasattr, and getattr are the most frequently changed dynamicfeatures (Finding 9). The most common change action on dynamic features isUpdate (Finding 10). 23% dynamic feature changes are related to bug-fix activitiesand 77% dynamic feature changes are related to non-bugfix activities (Finding 11).We applied the PyDiff tool on PyPy to get the answer for each research question.And we compared our findings against the findings in the original study and reportwhether they are similar or different (as shown in Table 3.4). In summary, we foundthat PyPy, as a compiler, has very different code change behavior compared to theapplications from other domains and 6 out of 9 findings are different comparedwith the original study. In the following sections, we detailed our approach forreplication and results for each research question. And we summarised our findingsand discussed the implications.30Table 3.2: Comparisons between the original study and the current studyResearch QuestionsFinding Comparison Implications ComparisonAcross ProjectsF1:Function Change and StatementChange are the most common changetypes, whereas Loop Structure Changeis the least common change type.PyPy as an implementation ofa language rarely uses thirdparty dependencies.Therefore, the Import changeis the least common changetype.DifferentNF1: Statement Change and Func-tion Change are the most commonchange types, whereas Import Changeis the least common change type.F2: The distributions of change typefrequency share similar trends acrossstudied projects.PyPy, as a Python writtencompiler, is different fromother Python projects inchange behavior. Morereplication studies are neededto generalize the evolutionarybehavior of Pythonapplications.DifferentNF2: The distribution of change typefrequency is significant different be-tween PyPy and the projects in theoriginal study.F3: There are no significant dif-ferences among the distributions ofchange type frequency across studieddomains.The domain of compiler doesnot share similar change typefrequency distribution withthe projects from the otherapplication domains. It isworthwhile to study otherPython-based compilers to seewhether our findings wouldgeneralize in this domain.DifferentNF3: PyPy is significantly differentfrom the studied Python-based appli-cations in other application domainsin terms of the distribution of changetype frequency.31Table 3.3: Comparisons between the original study and the current study(continued)Research Questions Finding Comparison Implications ComparisonAcross VersionsF4:Considering project Scipy, the dis-tributions of change type frequency aredifferent significantly between bug-fixversion and non-bugfix version.Not applicable Not applicableF5: Language evolution may have ahuge impact on software evolution.Not applicable Not applicableF6: The distributions of change typefrequency across versions show no sig-nificant difference.Similar to Project Pandas, thedistributions of change typefrequency across versions inPyPy are similar.SimilarMaintenanceActivitiesF7: If structure related change typesare more related to bug-fix, especiallyCondition Expression Update and IfInsert.In PyPy, bugs are more likelyto be fixed by StatementUpdate. Bug predictionmodels for Python-basedapplications trained in onedomain cannot be easilytransferred to other domains.DifferentNF7: Statement Update is the onlychange type that is related to bug-fix.F8: Import statement and FunctionStructure change types are mainly re-lated to non-bugfix activity.Like the original study, devel-opers are most likely to in-volve changes like AdditionalFunction, Function Renaming,and Parameter Insert to addednew features to the project.Similar32Table 3.4: Comparisons between the original study and the current study(continued)Research Questions Finding Comparison Implications ComparisonDynamic FeaturesF9: In their studied projects, isin-stance, type, hasattr, and getattr arethe top four dynamic features thatchange frequently.Similar to other projects, PyPypractitioners should pay moreattention to dynamic featureslike isinstance, type, getattr,hasattr and delete as well.SimilarF10: The change actions on dynamicfeatures follow the pattern that Up-date is the most common type, fol-lowed by Insert and Delete respec-tively.PyPy might involves moredynamic features than normalPython projects. The highperformance of PyPy is not aresult of constraining the useof dynamic features.DifferentFN10: The most common change ac-tion for dynamic features in PyPy isInsert. Delete changes in more thanUpdate changes in most of the dy-namic features.F11: Among the changes contain-ing dynamic features in the studiedprojects, about 23% are related to bugfixes, while 77% are related to otherchanges.In PyPy project, bug-fixchanges do not involve asmany dynamic featurechanges as the originallystudied projects do.DifferentNF11: Among the changes containingdynamic features in PyPy, about 19%are related to bug fixes.333.4 Replication ResultsIn our replication study, we selected PyPy as our subject, because PyPy, as acompiler written in Python, is from a domain that was not studied in the originalstudy. Since the first commit on Feb 24, 2003, there are about 97,000 commitssubmitted to the project after 16 years of development. About 44 releases published.And the earliest release (pypy-2.1-beta1-arm) was published on June 12th, 2013with 4180 Python files and 1,393,401 lines of code. The latest release (release-pypy3.6-v7.1.1) was published on Apr 14th, 2019 with 3930 Python files and 987,939lines of code. The size of the project has been expended for about 10% over thepast years. We applied PyDiff on PyPy for source code change extraction. Andwe answered each of the research questions proposed by the original study in thesubsections below. For each RQ in the original study, we described the approachthat we conducted the experiment and compared our findings with the findings inthe original study and discussed the implications.3.4.1 (RQ1) Across ProjectsIn order to get the diffing result for all PyPy commits, we first extracted the changedsource code files for each commit and its parent commit. Then we applied the PyDiffin each commits to extract the source code changes within each commit. For each34 0%10%20%30%40%50%60%70%80%90%100%PyPy Django Tornado Pandas Pylean2 Numpy Scipy Sympy Nltk Beets MopidyClass Function Statement Selection StructureLoop Structure Exception Handling Import Others0%10%20%30%40%50%60%70%80%90%100%PyPy Django Tornado Pandas Pylean2 Numpy Scipy Sympy Nltk Beets MopidyClass Function Statement Selection StructureLoop Structure Exception Handling Import OthersFigure 3.8: The distribution of change type frequency across projects.change type, we first counted the number of changes of the type in the history,and then we divided it by the total number of changes and got the change typefrequency. Figure 3.8 shows the change type frequency for all studied projects.As shown in Figure 3.8, the Statement change has the highest frequency (75%)and the frequency for Function change and Selection Structure change ranked atthe second and third place with a frequency of 52% and 44% respectively. On theother hand, the most uncommon change type in PyPy is Import changes.New Finding 1: Statement Change and Function Change are the most com-mon change types, whereas Import Change is the least common change type.Implications: PyPy as an implementation of a language rarely uses third partydependencies. Therefore, the Import change is the least common change type.35Based on the measured change type frequency, we applied Wilcoxon rank sum(WRS) test [27] to statistically compare the change type frequency distributionbetween PyPy and the originally studied projects. A p-value larger than 0.05indicates there is no significant difference in change type frequency distributionbetween PyPy and the originally studied projects. In addition, we group the changetype frequency based on the application domains. And we applied the WRS test tostatistically compare the change type frequency distribution between the domainof compiler with five other domains in the original study.The statistical result indicates that the change type frequency distribution inPyPy is significant different from the projects in the original study. And the changetype frequency in the domain of compiler is significantly different from all the fivedomains in the original study.New Finding 2: The distribution of change type frequency is significant differ-ent between PyPy and the originally studied projects.Implications: PyPy, which is a compiler written in Python, is different fromother Python projects in terms of change behavior. More replication studiesare needed in order to generalize the evolutionary behavior of Python-basedapplications.36New Finding 3: PyPy is significantly different from the studied Python-basedapplications in other application domains in terms of the distribution of changetype frequency.Implications: The domain of compiler does not share similar change type fre-quency distribution with the projects from the other application domains in theoriginal study. It is worthwhile to study other Python-based compilers to seewhether our findings would generalize in this domain.3.4.2 (RQ2) Across VersionsBase on the source code diffing result for each commit in RQ1, we grouped thecommits into releases. For all commits in each release, we measured the change typefrequency. Figure 3.9 shows the change type frequency for the 11 PyPy releases.As we can see from Figure 3.9, the change type frequency distribution across dif-ferent versions show a similar trend. To further examine the finding, we conductedthe WRS test between the change type frequency distribution between differentPyPy releases. The statistic result shows that there is no significant difference inchange type frequency distribution across different PyPy releases.In addition, we looked into the release logs and check if any of the releases ispurely a bug-fix release. But the release log shows that all the 11 release are mixedwith changes about bug-fix, new features, and improved test coverage. Therefore,37 0%10%20%30%40%50%60%70%80%90%100%Class Function Statement Selection StructureLoop Structure Exception Handling Import OthersFigure 3.9: The frequency comparison between different versions of PyPy.Finding 4 and Finding 5 in the original study do not hold in the PyPy project.Finding 6: The distributions of change type frequency across versions show nosignificant difference.Implications: Similar to Project Pandas, the distributions of change type fre-quency across versions in PyPy are similar.3.4.3 (RQ3) Maintenance ActivitiesIn the original study, they found that the distribution of change type frequency inpure bug-fix versions are different from others. In order to understand the codechanges, they conducted a study comparing changes types frequency in bug-fix ac-tivities and non-bugfix activity. First of all, we divided the commits into bug-fix38and non-bugfix by mining the commit messages. For each commit, we got thecommit message and convert the words into word stems. Then, we checked if theword stems contain the bug-fix related keywords. In particular, we identified acommit as a bug-fix commit if the commit message contains word stems for thefollowing bug-fix related keywords: ‘error’, ‘bug’, ‘fix’, ‘issue’, ‘mistake’, ‘incorrect’,‘fault’, ‘defect’ and ‘flaw’. Figure 3.10 provides an overview of the change typefrequency among bug-fix and non-bugfix commits. And then, for each change type,the bug-fix and non-bugfix commits are divided into four groups (1) bug-fix com-mits containing this change type; (2) bug-fix commits not containing this changetype; (3) non-bugfix commits containing this change type; (4) non-bugfix commitsnot containing this change type. At last, we applied Fisher’s exact test on the fourgroups which can measure the significance of the association between two classifica-tions. In our case, a p-value smaller than 0.05 indicates that the change type differsbetween bug-fix and non-bugfix behavior. And the Odds Ratio from Fisher’s exacttest can quantify the strength of the association between two events. An OddsRatio equals one means the change type behave the same in bug-fix and non-bugfixcommits. If the odd ratio is larger than 1, the change type is more related to bug-fixcommits and vice versa.In the result of Fisher’s exact test, only Statement Update is is related to bugfixes with a p-value equals 3.54e-48 and an Odds Ratio of 1.27. However, there are in390%10%20%30%40%50%60%70%80%90%100%Class Function Statement SelectionStructureLoopStructureExceptionHandlingImport Othersbug nonbugFigure 3.10: The frequency comparison between bug-fix commits and non-bugfixcommits of PyPy.total 62 fine-grained changes types which is not related to bug fixes. In particular,many function level changes, e.g. Additional Function, Function Renaming, andParameter Insert are non-bugfix related.New Finding 7: Statement Update is the only change type that is related tobug-fix.Implications: In PyPy project, bugs are more likely to be fixed by StatementUpdate. Bug prediction models for Python-based applications trained in onedomain cannot be easily transferred to other domains (e.g., compilers).40Finding 8: Import statement and Function Structure change types are mainlyrelated to non-bugfix activity.Implications: Like the originally studied projects, developers are most likelyto involve changes like Additional Function, Function Renaming, and ParameterInsert to added new features to the project.3.4.4 (RQ4) Dynamic featureTo study the pattern of dynamic feature changes in PyPy project, we further imple-mented the PyDiff tool and included the functionality to identify dynamic featurechanges in the source code files. And then, we re-applied the PyDiff in all com-mits with the dynamic feature change function turned on and got the dynamicfeature changes in each commit. Based on the dynamic change results in PyPy, wemeasured the dynamic feature change type frequency by dividing the number ofcommits containing this dynamic feature change with the total number of commitscontaining dynamic feature changes.Figure 3.11 shows the distribution of the frequency of various change typesrelated to the dynamic features in PyPy and the originally studied projects. Amongall dynamic feature changes, more than 60% commits contain isinstance change.The second most frequently changed dynamic feature in PyPy is getattr, with apercentage of 22%. In addition, dynamic features type, hasattr, and del are also41 0%10%20%30%40%50%60%70%PyPyOthers0%10%20%30%40%50%60%70%80%90%100%PyPyOthersFigure 3.11: The change frequency of each dynamic feature.commonly changed in PyPy.Finding 9: In their studied projects, isinstance, type, hasattr, and getattr arethe top four dynamic features that change frequently.Implications: Similar to other projects, PyPy practitioners should pay moreattention to dynamic features like ininstance, type, getattr, hasattr and delete aswell.We grouped the dynamic feature changes into three categories based on thechange type: Insert, Delete, and Update. Figure 3.12 shows the percentage of eachchange actions within a dynamic feature change for the most commonly changedfive dynamic features. As shown in Figure 3.12, Insert is the most common changeaction for all the five dynamic feature changes, with a percentage from 48% to 54%.And the second common change action is Delete for most of the dynamic features42 00.10.20.30.40.50.6getattr isinstance type hasattr delInsert Delete Update0%10%20%30%40%50%60%70%80%90%100%getattr isinstance type hasattr delInsert Delete UpdateFigure 3.12: The change frequency of each dynamic feature.changes except getattr.New Finding 10: The most common change action for dynamic features inPyPy is Insert. Delete changes in more than Update changes in most of thedynamic features.Implications: PyPy might involves more dynamic features than normal Pythonprojects. The high performance of PyPy is not a result of constraining the useof dynamic features.In order to understand the how the dynamic features changed under differentmaintenance activities, we leveraged the identified bug-fix commits and non-bugfixcommits in RQ3 and divide the commits that contain dynamic features into two43groups: commits containing dynamic feature change that are bug-fix related andcommits containing dynamic feature change that are non-bugfix related.New Finding 11: Among the changes containing dynamic features in PyPy,about 19% are related to bug fixes.Implications: In PyPy project, bug-fix changes do not involve as many dynamicfeature changes as the originally studied projects do.3.5 Threats to ValidityIn this section, we will discuss the threats to validity.3.5.1 Construct ValiditySince the source code change can be very complicated, correctly identifying codechanges between two commits can be hard. To make sure that our source codediffing tool work properly, we manually examined 10 commits which contains morethan 10 changes. The results show that all the code changes that we identified havemore than 98% confidence level.3.5.2 Internal ValidityThe taxonomy of Python source code is defined by ourselves. In order to make aproper comparison with the original study, we must make sure the change types wedefined are the same as the original study. Therefore, we defined our classification44scheme based on the Fluri’s [43] taxonomy for source code change in object-orientedlanguage, which is also used by the original study.3.5.3 External ValidityThe study is conducted in Python-based applications, the findings may not begeneralizable to applications in other programming languages (e.g. Java, C). Thefindings we got from the source code change result of PyPy project from the domainof compiler is very different from the other Python applications and domains. Weplan to extend our study to the applications in other domains as well as otherprogramming languages in the future.3.6 ConclusionLin et al. [69] conducted the first work in studying the evolution of Python-basedprojects by studying 10 Python applications from 5 different application domains.They studied the Python source code change behavior from four different angles:across projects, across versions, maintenances, and dynamic features and got 11interesting findings. We performed a replication study on the PyPy project whichbelongs to the domain of compiler that has never been studied. We found that 6 outof 9 findings are different which indicates the evolution of projects from the domainof compiler is very different from the project. In the next chapter, we will dig more45into the reason for PyPy’s fast speed and find ways to optimize the performance ofPython-based applications running under PyPy.464 Assessing and Optimizing the PerformanceImpact of the Just-in-time ConfigurationParametersIn the previous chapters, we have studied the evolution of PyPy. In this chapter,we assess and optimize the performance of applications running under PyPy bytuning its JIT configuration parameters.4.1 IntroductionSoftware performance is one of the crucial factors related to the success and thesustainability of large scale software systems, which serve hundreds or even millionsof customers’ requests every day. Failure to provide satisfactory performance wouldresult in customers’ abandonment and loss of revenue. For example, Amazon re-ported that one second delay in loading their webpages could result in $1.6 billionloss in their sales revenue annually [41]. BBC has also recently found that 10% of47the users will leave their website even if there is merely one additional second ofperformance delay [37]. Various strategies (e.g., asynchronous requests [60], datacompression [54], just-in-time (JIT) compilation [10], load balancing [25], and re-sult caching [34]) have been developed to further enhance the performance of thesesystems.In general, there are three types of system executions depending on the pro-gramming languages: (1) executing natively on top of the operating systems (e.g.,C and C++), (2) executing the source code by the interpreters (e.g., Python, PHP,and JavaScript), and (3) executing compiled intermediate artifacts on the virtualmachines (e.g., Java and C#). Compared to the native execution mode, systemsexecuted under the interpreted mode (a.k.a., by interpreters or virtual machines)are generally slower due to their additional layers. To cope with this challenge,the JIT compilation is introduced so that frequently executed code snippets arecompiled into binaries, which can be executed natively.Existing works on the JIT compilation focus on the jitting strategies (e.g.,method [38] vs. trace-level based code jitting [31]), speeding up the process of theJIT compilations [63, 47, 70], optimizing the performance of the underlying virtualmachines [88, 89, 90], and detecting JIT unfriendly code [53]. Unfortunately, thereare very few existing studies which investigate the impact of the JIT configurationson the system performance. Software configuration is one of the main sources of48software errors [92]. The configuration settings of a software system can significantlyimpact its performance. Most of the existing configuration tuning and debuggingstudies are focused on the configurations of the studied systems [40, 93, 84, 61]. Fortuning the configuration settings of interpreters or virtual machines, the focus ismainly on optimizing the performance of the garbage collectors [82, 67, 33], exceptthe work by Hoste et al. [59]. In [59], Hoste et al. provided an automated approachto tuning the JIT compiler for Java, which is a method-based JIT. Hence, in thispaper, we seek to investigate the impact of the tracing-based JIT configurations onthe system performance by using PyPy as our case study subject.Python is nowadays one of the most popular programming languages [51].Python has been used extensively to develop real-world business systems, includingmany large scale and mission-critical systems inside companies like Facebook [66],Google [26], and PayPal [55]. Among various implementations of the Python pro-gramming language (e.g., CPython, IronPython, Jython, and PyPy), PyPy is gen-erally the fastest [14] mainly due to PyPy’s efficient tracing-based JIT compiler [31].Hence, in this paper, we focus on assessing and optimizing the performance impactof PyPy’s JIT configuration settings. The contributions of this chapter are:1. This is the first empirical study on assessing and optimizing the impact ofthe tracing-based JIT configuration settings on system performance.492. Our experiments are carried out on both the synthetic benchmarks as wellas real systems. The empirical findings in this paper can be useful for bothsoftware engineering and programming language researchers as well as prac-titioners.3. Compared to [59] which also used a search-based approach to automaticallytuning the JIT configuration settings, many of the details (e.g., the initialsetup, the configuration settings, and the evaluation details) are not clear.It is not easy to reapply the approach to other Java-based systems or otherJIT compilers. In this paper, we have detailed our search-based configurationtuning approach, (ESM-MOGA) to ease replication.4. To enable replication and further research on this topic, we have provided areplication package [16] which includes the implementation for our configu-ration tuning framework, PyPyJITTuner, as well as the experimental data.4.1.1 Chapter OrganizationThe organization of this chapter is as follows: Section 4.3 describes the exploratorystudy that we have conducted to understand the relationship between JIT configu-ration settings and its performance. Section 4.4 proposes our approach to automat-ically tuning the JIT configuration parameters. Section 4.5 shows the result from50 6 def test(): 7 even = 0 8 oddLarge = 0 9 oddSmall = 0 10 c = 0 (*) 11 for i in range(1000000): (*) 12 if i%2 == 1: (*) 13 even += 1 (*) 14 elif i * i <= 100: 15 oddSmall += 1 (*) 16 elif i * i > 100 and i < 1000000: (*) 17 oddLarge += 1 18 else: 19 c += 3 20 return c Figure 4.1: A sample PyPy code snippet with the jitted code marked as “(*)”.the case study conducted on three real word systems and evaluate the effective-ness of our approach. Threats and validity is discussed in Section 4.7. Section 4.8concludes this chapter.4.2 BackgroundIn this section, we will first give an overview of the JIT compilation process in Sec-tion 4.2.1. Then we will explain PyPy’s JIT configuration setting in Section 4.2.2.4.2.1 An Overview of the JIT Compilation ProcessJIT compilers are introduced for systems executed by interpreters (e.g., Python,PHP, and Ruby) or virtual machines (e.g., Java and C#) to further speed up thesystem performance during runtime. By default, there are no JIT compilations51upon the initial system startup and these systems are executed under the inter-preted mode by their interpreters or virtual machines. Hence, their performance isusually slower than natively executed systems (e.g., systems programmed in C orC++), whose binaries are executed directly on top of the operating systems. Tocope with this limitation, the JIT compiler is introduced so that, during runtime,various parts of the systems are converted into machine executable code (a.k.a.,code jitting). However, the code jitting process is usually slow, as it takes time toload and compile the corresponding code snippets. Hence, only the commonly used(a.k.a., “hot”) code snippets are usually jitted [31, 73]. For such systems, there isusually a warmup period after the initial system startup before these systems reachthe peak performance [30]. During the warmup period, the frequently executedcode will be profiled to locate the “hot” spots and various code snippets are beingjitted [70]. In general, there are two approaches for code jitting depending on theirgranularity: Method-based JIT Compiling: if one method has been used many times(a.k.a., “hot method”), the method-based JIT will compile this entire methodinto the binary executable format. Hotspot (Oracle’s implementation of theJava Virtual Machine) and Chakra (Microsoft’s JavaScript engine) use themethod-based JIT Compiling.52 Trace-based JIT Compiling is more fine-grained, in which only the com-monly executed code path (a.k.a., “hot path”) inside a method is compiledinto the binary executable format. PyPy and TracingMonkey (Mozilla’sJavaScript engine) use the trace-based JIT Compiling.For the method-based JIT compiler, there will be a threshold value (e.g., 1500 asthe default value for the configuration parameter -XX:CompileThreshold in Oracle’sHotSpot JVM), which defines the number of invocations for a particular methodbefore this method is considered to be hot. As soon as a method has been called1500 times, the whole method will be compiled into the binary executable format.For the trace-based JIT compiler, the process is a bit more complicated. Wewill use the sample code snippet shown in Figure 4.1 to explain. There can bevarious configuration parameters which define a particular code path to be hot. Forexample, in PyPy, there is a configuration parameter, called threshold, which definesthe number of times a loop has to be run before it can be considered hot. Duringthe system execution, the PyPy JIT compiler counts the number of iterations foreach loop and all code paths in the loop will be potential candidates for code jitting.For example, the code lines marked with star (*) in Figure 4.1 are the resultingjitted lines, if the method test is executed in PyPy under the default configurationsetting. After the loop reaches 1039 (the default value for threshold) iterations,the JIT compiler starts to trace the code path in the next iteration. And the code53path which contains the if branch, and the second elif branch will be recordedand compiled into efficient machine code. The first elif and the else branches arenot jitted, as they are not in the code path of the traced iteration.4.2.2 PyPy’s JIT ConfigurationThe types and the values of the JIT configuration parameters vary depending onthe programming languages and the compilers. For example, there are more config-uration parameters for PyPy’s JIT compiler than Java’s JIT compiler. Even withinthe same programming language, different language implementations may use dif-ferent configuration parameters. For example, in Java, the configuration parameterwhich indicates the threshold value for the number of invocations for a methodbefore code jitting is called -XX:CompileThreshold in Oracle’s HotSpot JVM [10],and -Xjit:count for IBM’s JVM [7]. In this paper, we have selected PyPy’s JITconfiguration parameters as our case study subject, due to the popularity of thePython programming language [51] and the fast execution under PyPy with itsefficient JIT compiler [31, 14]. The list of JIT configuration parameters can be ob-tained through running the pypy --jit help command. For PyPy version 5.7.1,which is the PyPy version used in this paper, there are 19 of them.544.3 Exploratory StudyTo motivate the importance of this work, we have conducted an exploratory studyon the performance impact of PyPy’s JIT configuration settings. We seek to answerthe following three research questions: RQ1: How different is the system performance before and after its code hasbeen jitted?When the system initially starts up, all of its code is executed under theinterpreted mode. The code jitting process will not start, until certain regionsof code have been repeatedly executed many times. In this RQ, we want toquantify the performance differences between the warmup and the warmedup phases. RQ2: What is the performance impact by varying JIT configurations?The system after the warmup phase would achieve its peak performance. Butwould the peak performance be different among different JIT configurationssettings (e.g., the default config, random configurations, or disabling JIT)?In this RQ, we seek to find the performance impact of different JIT configu-rations. RQ3: Do systems containing more jitted lines yield better performance?55Different JIT configuration settings would result in different amount of sourcecode been jitted. Intuitively, a higher portion of the jitted code could lead tomore code being executed natively, and hence result in better performance.However, the code jitting process is very resource heavy, which involves pro-filing system executions and compiling the hot code path into the binaryexecutable format. In addition, the systems may need to constantly switchbetween the two running modes (the interpreted vs. the native executionmode). The goal of this RQ is to examine whether there is any relationbetween the portions of the jitted code and the system performance.The remaining three subsections in this section will address the above threeresearch questions. For each research question, we will first explain the experimen-tation process. Then we will describe the data analysis techniques, present theresult findings, and discuss their implications.(RQ1) How different is the system performance before and after its codehas been jitted?During the benchmarking and the performance testing processes, it is considered asa common practice to wait for a period of time (a.k.a., the warmup phase) for thesystem to stabilize [32, 8], before starting the actual benchmarks or the performancetests. During the warmup phase, various regions of the code are getting jitted and56the system caches are slowly being filled up. Hence, the performance of the warmupphase is generally considered as suboptimal and is discarded during the subsequentperformance analysis. In this RQ, we want to quantitatively compare the systemperformance during and after the warmup phase.ExperimentTo tackle this research question, we selected the following two microbenchmarksuites, which assess the performance of different software systems: The PyPy benchmark suite is run daily on PyPy’s nightly builds and is mainlyused to compare the performance of various Python implementations (PyPyvs. cPython). The benchmark suite consists of about 60 small Python pro-grams, which perform various computation tasks like the n-queens solver,HTML table building, etc. For each run of the benchmark, the same bench-mark programs will be run under PyPy and cPython (the default Pythonimplementation). The performance results are uploaded and visualized in thePyPy’s Speed Center [14]. In this exploratory study, we randomly selectedseven benchmark programs as shown in Table 4.1 for experimentation. Wefurther instrumented these benchmark programs to gather additional perfor-mance information (e.g., individual request response time).57 The TechEmpower Web Framework Benchmark suite1 [19], whose main ob-jective is to evaluate among various web frameworks, consists of more com-plicated web application-related tasks like JSON serializations, database ac-cesses, and server-side template compositions. Different from the PyPy bench-mark suite, whose programs are usually short-lived and computation inten-sive, the TechEmpower benchmark suite executes on long running web appli-cation servers built with various frameworks. For example, the benchmarkincludes Java-based web frameworks (e.g., Jetty), as well as Python-basedweb frameworks (e.g., Tornado and Flask). In this paper, we only focus onthe Django web application frameworks.We ran the two microbenchmark suites under the default PyPy configurationsetting. To avoid measurement bias and errors [50], for the PyPy benchmark, inwhich the studied programs are short-lived and computational intensive, we re-peated the benchmark for 30 times. For the TechEmpower benchmark, which ex-amines the performance of processing web requests for long-running servers, we setthe duration for each benchmark task to be two hours. During the benchmarkingprocess, resource utilizations (e.g., CPU, memory, and disk) for the servers weremonitored and recorded using pidstat [12]. We also added additional instrumenta-tion using the JIT logging function from PyPy’s jitlog module to record the JIT1To ease explanation, we will call this the TechEmpower benchmark in the rest of this paper.58Table 4.1: PyPy benchmark programs description.Program Descriptionai Test the performance of simple AI solvers.bm mako Benchmark for test the performance of Mako templates engine.chaos Test the performance of the Chaos benchmark. Create chaosgamelike fractals.django This will have Django generate a 100x100 table as many times asyou specify.rietveld This will have Django render templates from Rietveld with canneddata as many times as you specify.html5lib Test the performance of the html5lib parser.pidigits Test the pidigit calculation performance.59logs to the disk. The recorded JIT logs can be further parsed with VMProf [23]to obtain the exact lines of code that were jitted. However, the recorded JIT logsdo not have timestamps to mark when a code snippet is jitted. To estimate theexact timing when individual code snippets are jitted, we decided to periodicallytake snapshots of the JIT log files. We took snapshots of the JIT log files after eachiteration of the PyPy benchmark and every minute for the TechEmpower bench-mark. These snapshots would help us gain insights on the time and the locationof the jitted code regions. Finally, we also archived the benchmarking logs, so thatwe can extract the response time for each individual request.|<------ jitting ------>|<------------------------------------ jitted ------------------------------------>|1.52.02.55 10 15 20 25 30 35 40 45 50Number of iterationsResponse time (second)4604805005205 10 15 20 25 30 35 40 45 50Number of iterationsNumber of jitted linesResponse time (second)Number of jitted linesFigure 4.2: Number of jitted lines and response time over 50 iterations for thehtml5lib program from the PyPy benchmark suite.60Data AnalysisWe parsed the JIT logs using VMProf to obtain the regions of the jitted codeduring each snapshot period. We also processed the benchmarking logs to extractthe response time for each iteration of the PyPy benchmark and the response timefor individual requests in the TechEmpower benchmark.To understand the performance of the systems during and after the warmupphase, we need to determine the duration of the warmup phase. For the PyPybenchmark, we kept track of the amount of the jitted code during each iteration. Weconsidered the warmup phase to be completed, when the amount of the jitted coderemains stable during the remaining of the benchmarking run. Figure 4.2 shows theresult for the html5lib program from the PyPy benchmark. The topper subgraphof Figure 4.2 shows the how the response time evolve over different number ofiterations. Since each program within the PyPy benchmark is repeatedly executed30 times, we aggregated the response time for that iteration across the 30 runs usingboxplots. For example, the first boxplot contains all the response time values forthe first iteration during the 30 runs. The bottom subgraph of Figure 4.2 shows theevolution of number of jitted lines across different iterations. The red dotted lines inboth subgraphs indicates the iteration when the number of the jitted lines becomesstable. Hence, we considered the first 11 iterations as the warmup phase (“jitting”)61and the remaining iterations (a.k.a., the 12th to the 50th iterations) as the warmedup phase (“jitted”). The response time is the highest during the initial iteration,and gets slowly improved when the amount of the jitted code increases. After the11th iteration, the response time stabilizes. For the TechEmpower benchmark, weused a similar approach as the PyPy benchmark and divided response time intothe warmup phase and the warmed up phase based on the time when the numberof jitted lines gets stabilized.We applied statistical techniques to rigorously compare and quantify the differ-ences between the response time distributions from the two phases. Statistical testlike the Wilcoxon Rank Sum (WRS) test would give us a rigorous measurementif the distributions of the performance data from these two phases are different.However, in some cases, even if the distributions are different, the differences be-tween the two distributions can be small. For example, if the response time for onerequest is long (e.g., more than five minutes) and the differences of the responsetime between the two experiments are very small (e.g., one millisecond), such per-formance differences would not be useful for our study, as it will not be noticed bythe end-users. Hence, we also need to quantify the strength of the differences be-tween the two distributions, called the effect size [65]. In this paper, we used Cliff’sDelta (CD) as our effect size measures. Both CD and WRS are non-parametrictechniques. Hence, they do not hold any assumptions regarding the distributions62of the data. We consider two datasets as statistically different, when the p-valuefrom the WRS test is lower than 0.05. The strength of the differences and thecorresponding range of CD values [76] are shown below:effect size =trivial if |CD| < 0.147small if 0.147 ≤ |CD| < 0.33medium if 0.33 ≤ |CD| < 0.474large if 0.474 ≤ |CD|We used the following criteria to judge if the response time from the warmedup phase (denoted as B) is getting better (>), worse (<), or relatively the same(∼) as the warmup phase (denoted as A):difference =A > B if CD ≤ -0.33 and p-value < 0.05A ∼ B if |CD| < 0.33 or p-value ≥ 0.05A < B if CD ≥ 0.33 and p-value < 0.05The p-values shown above are calculated from the WRS test. In other words,B improves over A (B > A) when the WRS test and the CD value satisfy thefollowing three conditions: (1) the two distributions are statistically significantlydifferent (p-value < 0.05), and (2) the differences between the two distributionshave medium or large effect sizes, and (3) CD value is positive, indicating B issmaller than A. The conditions for B degrades from A (B < A) is similar, except63the CD value is negative, indicating B is bigger than A. If A and B are relatively thesame (B ∼ A), when there is no statistical difference between the two distributions(a.k.a., p-value ≥ 0.05) or the effect size between A and B is small or trivial.We extracted the performance data from the warmup and the warmed up phasesbased on the time when the number of jitted lines stabilizes for all the runs of thetwo microbenchmark suites. We compared the response time between the twophases for each run. Table 4.2 shows the results.Table 4.2 shows almost all the programs/scenarios (except one) for both mi-crobenchmark suites exhibit better performance during the warmed up phase. How-ever, in the PyPy benchmark suite, the ai program is not showing significant per-formance improvement. This is because at the end of the the first iteration whilerunning the ai program, the majority of the code jitting process has already beencompleted. Only a few lines from the test library, which does test setup, got jittedduring the benchmarking process (at the 34th iteration). These additional jittedlines have no impact on the actual performance of the benchmark program. Thus,the performance differences between the two phases are very small.64Table 4.2: Comparing the response time between the warmup phase (A) and thewarmed up phase (B).Performance # of scenarios in the # of scenarios in theDifference PyPy benchmarks TechEmpower benchmarksB < A 0 0B ∼ A 1 0B > A 6 6Total # of scenarios 7 6Findings: For most of the studied programs/scenarios, the performance in thewarmed up phase is statistically much better than the warmup phase. Thisclearly highlights the huge performance gain contributed by the JIT compilations.Implications: Only performance data from the warmed up phase can be repre-sentative of the performance of systems due to the big difference in performancebetween the warmup and the warmed up phase. Thus, performance analystsshould be careful when conducting the analysis and focus on the data after thewarmup phase. In addition, the warmup phase for each system should be as shortas possible, due to its inferior performance. Existing techniques for speeding upthe jitting processes [63, 47, 70] can be very useful in this aspect.65(RQ2) What is the performance impact by varying JIT configurations?In RQ1, we have found that the system performance significantly improves afterthe warmup phase. Hence, the data during the warmup phase is normally discardedduring the performance analysis phase for a benchmark or a performance test. Inthis RQ we only focus on the system performance after the warmup phase. We studythe performance impact of various JIT configuration settings. In particular, wewould like to (1) verify whether the system configured with the default configurationsetting would yield the optimal performance amongst other configuration settings,and (2) measure the performance impact of the jitting process (a.k.a., comparingthe performance against completely disabling the jitting process).ExperimentSimilar to RQ1, we still used the same two microbenchmark suites as our experi-mental subjects. However, instead of keeping the default JIT configuration setting,we varied the values for the following six JIT configuration parameters: decay,function threshold, threshold, loop longevity, trace eagerness, trace limit. Table 4.3shows the detailed information about these six configuration parameters. We pickedthese six parameters because we think they are tunable and can have an impacton where and when certain regions of the source code will be jitted. Other pa-66Table 4.3: List of relevant PyPy’s JIT configuration parameters and theirinformation.Parameter Range Default Descriptionsdecay [0, 1000] 40 The amount that PyPy decrease thecounters for each loop or functionperiodicallyfunction threshold (0,∞) 1619 Number of times a function must runfor it to become traced from startloop longevity (0,∞) 1000 A parameter controlling how longloops will be kept before being freedthreshold (0,∞) 1039 Number of times a loop has to runfor it to become hottrace eagerness (0,∞) 200 Number of times a guard has to failbefore we start compiling a bridgetrace limit [0, 16385] 6000 Number of recorded operationsbefore we abort tracing withABORT TOO LONG67Table 4.4: The JIT configurations chosen for performance evaluation.Group Config threshold function threshold decay trace limit trace eagerness loop longevityDefault X 1039 1619 40 6000 200 1000Group114X 260 405 10 1500 50 25012X 520 810 20 3000 100 500X 1039 1619 40 6000 200 10002X 2078 3238 80 12000 400 20004X 4156 6476 160 12000 800 4000Group2R1 64 101 120 375 200 2000R2 519 809 5 375 200 2000R3 519 101 20 1500 200 4000R4 3117 1619 2 1500 25 2000R5 259 4857 120 375 200 2000Group 3 JIT Off - - - - - -rameters like enable ops, inlining, and off need to be kept as default to enable thejitting process; whereas other parameters: vec, vec all, vec cost are not included inour study, as they are not relevant to the selected microbenchmark suites. Sincethere can be many possible combinations of these parameter settings, due to timeconstraints, we decided to run the two microbenchmark suites under the followingeleven configuration settings from three different groups:1. Group 1 (Varying Default Configurations) consists of five configuration set-tings (14X, 12X, X, 2X, and 4X) by mutating the default configuration setting.68X refers to the default configuration setting. 2X means doubling the defaultconfiguration values, whereas 12X means cutting the default configuration val-ues by half and rounding to the nearest integer values. As shown in Table 4.4,the default PyPy JIT configuration setting, X, is: (1039, 1619, 40, 6000, 200, 1000),which corresponds to the configuration parameters (threshold, function threshold,decay, trace limit, trace eagerness, loop loogevity). Hence, the 2X settingwould be: (2078, 3238, 80, 12000, 400, 2000) and the 12X setting would be:(519, 809, 20, 3000, 100, 500). To avoid PyPy command line parsing errors,when the value of parameter trace limit exceeds the upper bound, we just setit to be two times of the default value (12000).2. Group 2 (Randomly Generated Configurations) consists of five randomly gen-erated configuration settings (R1 , R2 , R3 , R4 , and R5). For each parameterin the configuration setting, we randomly generated an integer value withinthe defined boundary. As we can see from Table 4.4, the randomly generatedJIT configurations in Group 2 are very different from the JIT configurationsfrom Group 1.3. Group 3 (JIT Off) consists of only one configuration setting, which sets theparameter off to be true. This setting will completely disable the jittingprocess.69Similar to RQ1, to avoid the measurement errors and noise, we repeatedly ex-ecuted each PyPy benchmark program for 30 times, and ran each TechEmpowerbenchmark scenario for two hours. We also collected the same kind of performancedata (a.k.a., the resource utilization metrics, the JIT logs, and the benchmarkinglogs) for further analysis.Data AnalysisFor each experiment, we first parsed the JIT log snapshots. Based on the timewhen the number of jitted lines stabilizes, we divided the benchmark runs into thewarmup and the warmed up phase. We extracted response time from the warmedup phase for further analysis. We used the same statistical analysis techniques asin RQ1 to compare the response time among all the runs. For each program insidethe PyPy benchmark suite, we compared the performance between each pair of theJIT configuration settings and identify the best performing configuration setting.Similarly, we also located the best performing JIT configuration settings for eachscenario inside the TechEmpower benchmark suite. Table 4.5 shows the results.There are ties when ranking the top performing configuration settings across differ-ent programs/scenarios. We noted with a “*” besides a configuration setting if itshares the first place with other configuration settings in any programs/scenarios.As shown in Table 4.5, Only 3 out of the 7 programs from the PyPy benchmark70Table 4.5: Number of best performing programs/scenarios under each JITconfiguration setting.Settings# of best performing programs/scenariosPyPy Benchmark TechEmpoer Benchmark14X 1* 012X 0 1*X 3* 6*2X 2* 1*4X 1* 0R1 0 0R2 0 0R3 2* 0R4 1 0R5 1 0JIToff 0 071suite, where the default configuration setting yields the best performance. Fur-thermore, there are many configuration settings which perform best for some PyPybenchmarks (e.g. R4 , R5) or share the top performance with other configurations(e.g. 14X, 4X). For the TechEmpower benchmark suite, all of the scenarios performthe best (with one scenario tied with 2X and 12X) under the default configurationsetting.To quantify the performance impact of the jitting process, we also compared theperformance of different JIT configuration settings against the JIT off setting. Foreach JIT enabled configuration setting, we measured the number of programs/sce-narios that perform worse (<), similar (∼), or better (>) than the JIT off setting.Table 4.6 shows the result. The number of programs/scenarios whose performanceunder jit enabled configurations is worse or no different than jit off is marked asbold.From Table 4.6, we can see that, among the PyPy benchmark suite, all theJIT enabled configuration settings perform better than the JIT off setting, exceptR1 in which the performance of two PyPy benchmark programs is even worsethan completely disabling the jitting process (a.k.a., JIT off)! Similarly, in theTechEmpower benchmark suite, R1 is still the odd one, as none of its scenariosis better than the JIT off setting. In addition, only three of the TechEmpowerscenarios under R5 are better than the JIT off setting.72Table 4.6: Comparing the jitting performance against the configuration under JIToff. The number of programs/scenarios whose performance under jittedconfiguration is worse or no different than JIT off setting is highlighted in bold.ConfigsPyPy Benchmark TechEmpower Benchmark< ∼ > < ∼ >14X 0 0 7 0 0 612X 0 0 7 0 0 6X 0 0 7 0 0 62X 0 0 7 0 0 64X 0 0 7 0 0 6R1 2 0 5 5 1 0R2 0 0 7 0 0 6R3 0 0 7 0 0 6R4 0 0 7 0 0 6R5 0 0 7 2 1 373Findings: Programs/scenarios running under the default configuration settingdo not necessarily yield the best performance when comparing to other configu-ration settings. The optimal JIT configuration setting can vary depending on theprograms/scenarios. The performance of some of the JIT enabled configurationscan be worse than turning JIT off.Implications: PyPy’s JIT configuration settings have a big impact on the sys-tem performance. It is important to find the optimal JIT configuration settingfor each system to achieve the best performance.(RQ3) Do systems containing more jitted lines yield better performance?In RQ2, we have found that the default JIT configuration setting does not neces-sarily result in the optimal performance. Different JIT configuration settings wouldresult in different portions of the code been jitted. However, does more jitted codealways lead to better performance? In this RQ, we want to study the relationshipbetween these two aspects.ExperimentWe used the same data from RQ2 and did not run any additional experiment forthis RQ.74Data AnalysisWe first selected the configuration setting that has the best performance for eachprogram/scenario based on the results of RQ2. Then we also selected the config-uration settings with the highest number of jitted lines. If there are two differentconfiguration settings corresponding to the above two criteria, we further performedthe WRS test and calculated the CD value between the performance data underthose configuration settings.Table 4.7 shows the effect size between the best performing and the most jittedconfiguration settings for each program/scenario. Since there can be ties in eithercategory, we compared all pairs of configuration settings from the best performingcategory to the category of the largest portion of jitted code. We label True at thethird column, if there is at least one common configuration setting in both categoriesfor one program/scenario. In 71% (57) of the programs in the PyPy benchmark suiteand all the scenarios in the TechEmpower benchmark suite, the best performingconfiguration setting is different from the one that has the highest number of thejitted lines. When comparing the performance differences, we compared all thepairs of these configuration settings from the two categories. In the end, twelveof the programs/scenarios have a medium to large effect size differences. In otherwords, the results show that more jitted lines do not necessarily lead to better75Table 4.7: Comparison between the configuration settings yielded the bestperformance and the configuration settings resulted in the most jitted code.PyPy Benchmark TechEmpower BenchmarkPrograms Effect Size Same Scenarios Effect Size Sameai large False db large Falsebm mako trivial,medium False fortune large Falsechaos large False json large Falsedjango large False plaintext large Falsehtml5lib - True query large Falsepidigits large True update large Falserietveld large Falseperformance. For the PyPy benchmark ‘html5lib’, we marked the effect sizes as‘-’, since the best performing configuration settings and the highest amount of thejitted code configuration settings are exactly the same. Hence, we do not calculatethe effect sizes for this case.76Findings: JIT configuration settings, which resulted in the highest number ofthe jitted lines, do not necessarily yield the best performance.Implications: We cannot just arbitrary choose the configuration settings thatfavor more jitted lines of code while tuning the system performance. A moresophisticated approach is needed to locate the optimal configuration setting(s)for one system.4.4 Automatically Tuning the JIT Configuration Parame-tersIn the previous section, we have found that PyPy’s JIT configuration settings dohave a significant impact on the system performance. Furthermore, there is nostraightforward way to recommend a performance-efficient JIT configuration set-ting, since such setting can be application-dependent and a higher portion of thejitted code does not necessarily result in better performance. Hence, in this section,we will propose our automated approach, ESM-MOGA (Effect Size Measurement-based Multi-Objective Genetic Algorithm), to tuning the JIT configuration param-eters for one system.Figure 4.3 provides an overview of our tool which we called the PyPyJITTuner.It leverages a search-based technique called Multi-Objective Genetic Algorithm(MOGA) [39] and a statical measure called effect size, for the exploration of the77MOGALoad GeneratorSystemUnderStudyTest SchedulerJIT ConfigurationsPerformanceDataOptimal JIT ConfigurationsFigure 4.3: Overall process of PyPyJITTuner.JIT configuration space. Genetic Algorithm (GA) is a search-based method inspiredby evolutionary biology, in which a population of solutions is evolved during eachgeneration. The solutions from the next generation should be generally better thanthe previous generations evaluated based on some objective functions. MOGA is atype of GA, in which multiple objectives are being considered. We chose MOGA, asthere can be multiple objectives associated with a system’s performance (e.g., op-timizing the response time for multiple scenarios). One machine, which is deployedwith the tailored-version of the MOGA, acts as the configuration advisor. Whennew solutions have been created, this machine continuously sends the JIT config-uration settings (solutions) to the test scheduler machine, which will deploy andconfigure the system under study (SUS). Once the test scheduler machine receivesthese settings, it will reset the test environment (a.k.a., clean up the database, andremove the testing data from the previous run), and start up the SUS under thesuggested JIT configuration setting. The same performance test (a.k.a., the same78workload) will be executed. Once the test is completed, the performance data willbe collected and sent to the configuration advisor machine for further analysis. Theconfiguration advisor machine will evaluate the newly received performance dataagainst the data from other configuration settings and leverage the MOGA meth-ods to select the best solutions and generate the next generation. If the solutions inthe next generation are good enough, the MOGA will stop the evolution and out-put one or multiple “optimal” configuration settings. Otherwise, the MOGA willcontinue with another round of iteration. The newly generated JIT configurationsettings will be sent to the test scheduler machine for another round of testing.The rest of this section is organized as follows: Section 4.4.1 explains the generalidea behind the ESM-MOGA approach. Section 4.4.2 presents our performancetesting framework, and Section 4.4.3 describes briefly our implementation.4.4.1 Tailoring MOGA for JIT Configuration TuningGA is a search-based method inspired by evolutionary biology. GA encodes thecandidate solutions into a set of values, called “chromosomes”. Inside the chro-mosomes, the set of values, which are to be optimized are called the “genes”. GAstarts off with a population of the initial solutions and keeps iterating until any ofthe termination criteria is met. During each iteration, GA improves the populationvia crossover (combining existing solutions to produce new solutions), mutation79(randomly changing some values in the solutions), and selection (picking the bestcandidate solutions). The termination criteria can either be the optimization con-ditions (e.g., the resulting solutions are better than a predefined threshold) or themaximum number of iterations. The MOGA, which is a type of GA, evaluates mul-tiple objectives simultaneously. In general, as illustrated in Figure 4.4, the MOGAconsists of six phases: the problem formulation phase, the initialization phase, thetournament phase, the evolution phase, the selection phase, and the stopping phase.The process of going through the tournament, the evolution, and the selection phasecan be repeated multiple times, with each iteration called one generation. At theend of each generation, a new population will be produced. This process will berepeated until any termination criteria described in the stopping phase is met.Initialization Tournament Evolution Selection StoppingParents+PPopulationQNYP = RTesting FrameworkOptimalConfigurationPopulationRPopulationPJIT ConfigurationPerformance DataJIT ConfigurationPerformance DataProblem FormulationFigure 4.4: The workflow for our tailored version of the MOGA method.In this subsection, we will explain the ESM-MOGA approach by using a runningexample. For illustration purposes, we assume the SUS in our running example is80a simple e-commerce system, which consists of only three scenarios: login, browse,and purchase.4.4.1.1 Phase 0 - Problem FormulationWe formulated our problem of automated tuning of JIT configuration settings into amulti-objective optimization problem. Our objective is to find one or more optimalJIT configuration settings that yield the best performance in all the scenarios inthe system. Below we define our solution encoding and objectives: Solution Encoding: The ESM-MOGA requires us to encode its solution intobinary strings. As shown in Section 4.2.2, all the studied JIT configurationparameters are integers and have a large range (a.k.a., many possible values).Assume we use 232 as the upper bound for unbounded parameters, we haveto do 5.5e+45 performance tests if we want to try out all combinations whichis apparently impossible.Hence, we decided to select eight representative values from the input domainof each configuration parameter: (4X, 3X, 2X, X, 12X, 14X, 18X, 116X), whereX refers to the default value for that configuration parameter. We chose theabove eight levels, as these eight values cover a wide range of the input domainand each configuration parameter value can be easily encoded into a binarystring of length three. In this way, the smallest configuration value ( 116X),81the default configuration value (X), and the largest configuration value (4X)for each parameter are encoded as 111, 011, and 000, respectively. We setthe largest configuration values to be 4X, as Section 4.3 shows that largeJIT configuration settings usually do not yield good performance. We setthe smallest configuration values to be 116X, as Section 4.3 shows very smallconfiguration settings could result in a high number of jitted code, but worseperformance. To ease explanation, in our running example, there are onlytwo configuration parameters. Hence, the default configuration setting canbe encoded as a binary string: 011011. Objectives: For a real world system, there can be more than one aspectassociated with the performance of the system. Examples of optimizing per-formance aspects can be optimizing the resource utilizations (e.g., CPU, mem-ory, and disk) or the responsiveness of different scenarios in a system. Someof these concerns can be conflicting with each other. In our approach, we fo-cus on optimizing the response time for different scenarios in a system. Eachobjective refers to a list of response time for each scenario during the warmedup phase, measured through performance testing. In our running example,the objectives are to optimize the response time for the above three scenariosin the e-commerce system.824.4.1.2 Phase 1 - InitializationDuring the initialization phase, the ESM-MOGA defines an initial population (P)consisting of n solutions. In our approach, P consists of the default configurationsetting, and n − 1 randomly generated configuration settings. The ESM-MOGAwill intentionally included the default configuration setting in the initial popula-tion, as we want to ensure the default configuration setting is evaluated among itsalternatives and the final “optimal” setting(s) will be at least as good or betterthan the readily available default configuration setting. Once the initialization pro-cess is completed, the ESM-MOGA enters the iterative process of going throughthe tournament, the evolution, and the selection phase to refine and improve itspopulation until any termination criteria is met.To ease explanation, we set n = 4 in our running example and compose theinitial population (P) with the following solutions:P =C1 : 011011,C2 : 001010,C3 : 000101,C4 : 101110Once the initial population is generated, the solutions in the initial populationwill be sent to the test scheduler machine in Section 4.4.2. Multiple performance83tests with the same workload will be conducted under each given configurationsetting. The response time of the three scenarios under warmed up phase will becollected as performance data and assigned as the objectives for each solution.4.4.1.3 Phase 2 - TournamentDuring the tournament phase, the ESM-MOGA will first randomly select two so-lutions from a pool which contains all solutions of the current population. Then, apairwise comparison is done to recognize the better solution from the two. The bet-ter solution will be used as one of the parents for the next phase. These evaluatedsolutions will not be put back to the pool for efficient concerns. The process willbe repeated until all solutions in the pool have been evaluated pairwisely. In ourapproach, the pairwise comparison is done using a pre-defined dominant compari-son function and the dominating configuration setting (a.k.a., the better solution)will be selected. In this dominant comparison function, the configuration setting Adominates the configuration setting B, if the response time distributions under thetwo configuration settings satisfy the following two criteria:1. The response time for all the scenarios under A are statisticallyno worse than under B: The response time under configuration A for onescenario is statistically no worse than under B, if (1) the response time distri-butions for that scenario under the two settings are not statistically signifi-84cantly different under the WKS test, or (2) they are statistically significantlydifferent under the WKS test, but there is only a trivial to small effect sizecalculated by the CD. This relation has to be held for all the scenarios whencomparing the two settings.2. There is at least one scenario whose response time under A is statis-tically better than under B: The response time under configuration A forone scenario is statistically better than B, if the response time distributionsfor that scenario under the two settings are statistically significantly differentunder the WKS test and there is a medium to large effect size calculated bythe CD.In other words, one configuration setting (A) only dominates the other one (B),if (1) the performance of all the scenarios under A is at least as good as B, and (2)there will be at least one scenario under A whose performance is better than B.The dominance comparison among all the pairs of the configuration settings areshown in Table 4.8. Each row in Table 4.8 corresponds to the comparison results ofone configuration setting pair. For example, the second row shows the comparisonresults between configuration setting C1 and C3 . The response time for the loginscenario is statistically better under C1 than C3 . The performance of the othertwo scenarios are statistically not different between the two settings. Hence, the85Table 4.8: Dominance relations among the four configuration settings in ourrunning example. “” means the the left configuration setting dominates theright configuration setting, “≺” means the right configuration setting dominates,and “≈” means there is no dominance relation.Config Pairs Login Browse Purchase Dominance(C1 , C2) Better Worse Better ≈(C1 , C3) Better Equal Equal (C1 , C4) Better Better Equal (C2 , C3) Better Equal Equal (C2 , C4) Better Better Equal (C3 , C4) Equal Worse Worse ≺configuration setting C1 dominates C3 . Assume, from the pool that contains allsolutions of population (P), we selected (C1 and C3), (C2 and C4) for pairwisecomparison. The two configuration settings, C1 and C2 , will be selected as parentsfor the next phase.4.4.1.4 Phase 3 - EvolutionDuring the evolution phase, the parents from the Tournament phase will undergothe following two actions to produce new solutions:86 Crossover: Two solutions from the Parents are randomly selected as par-ents. A new solution will be created by randomly selecting some bits fromone solution and the remaining bits from another solution. In our runningexample, C1 and C2 will be selected as parents. A new solution C5 (011010)can be created by inheriting the first three bits from C1 and the remainingbits from C2 . Mutation: Some of the newly produced solutions will be mutated by ran-domly flipping some bits (a.k.a., turning 0s into 1s and 1s into 0s). For ourrunning example, after flipping the first and the last bits of C5 , it becomes111011.Similar as the Initialization phase, a performance test with the same workloadbut a new configuration setting (C5) will be conducted. Once completed, the per-formance data will be sent back as objectives for the new configuration setting.The overall population (Q) at the end of this phase will consist of the new solu-tions produced after the crossover and the mutation operations as well as existingsolutions from P.874.4.1.5 Phase 4 - SelectionIn this phase, the “best” n solutions in Q will be selected with NSGA-II selec-tion [39] [5]. It will first use the non-dominated sorting algorithm to sort thesolutions into different levels (L0 , L1 , ...). Solutions that were dominated by thesmallest number of solutions will be assigned to the top level (L0). Solutions in L0are the “best” solutions in this iteration, followed by the solutions in L1 , and thenL2 , and so on. The solutions within the same levels (e.g., L0) are not dominantover each other. For example, if configurations settings A and B are both withinL0 , it means that A and B are not dominant over each other. In other words,some scenarios are better performed under A and whereas some other scenarios arebetter performed under B. When selecting the top n solutions, we will first startpicking solutions from the top level (L0), followed by solutions from L1 , and so on.If there are more solutions in a level than we needed (a.k.a., exceeding the total nsolutions), we will rank solutions within that level with crowding distance sortingand select the top ranked solutions in that level.Suppose for our running example, after sorting the solutions in Q using thenon-dominated sorting algorithm, they are divided into the following levels:88Q =L0 : C5 ,L1 : C1 , C2 ,L2 : C4L3 : C3Since at the end of the selection phase, only n solutions will be kept. Hence,our resulting population (R) will be C1 , C2 , C4 , C5 .4.4.1.6 Phase 5 - StoppingDuring this phase, the resulting population Q formed during this generation will beevaluated to decide whether its solutions are good enough comparing to the previousgeneration. The main idea is to decide whether any progress has been made duringthis generation. In other words, we want to check whether there are any bettersolutions produced during this generation. We used the Mutual Dominance Rate(MDR) [72] to measure the improvements made between the current population Band the population A from the previous generation:MDR(B,A) = dom(B,A)‖B‖ − dom(A,B)‖A‖ ,where dom(A,B) is defined as the number of solutions in population A thatare dominated by at least one solutions in B. Hence, for our running example,dom(P,R) would be 4, since all four solutions in P are dominated by at least one89solutions in R. dom(R,P ) would be 1, since only C4 in R is dominated by thesolutions in P. Hence, MDR(R,P ) = 14− 44= −34. The closer the MDR valuegets to −1, the larger the improvement has been made in the current generation.If MDR is close to 0, it means little progress has been made to the population.The iteration should be stopped if the improvement between two generations isinsignificant (a.k.a., |MDR| is smaller than some threshold values), or it has beenrunning for too long (e.g., over 100 iterations).When the termination criterion is met, we will output the top configurationsettings for the current generation with the NSGA-II selection. Since there is onlyone solution (C5) at L0 for our running example, C5 will be the optimal configurationsetting outputted.4.4.2 Our Performance Testing FrameworkFor any newly generated solutions, we need to measure their performance using aperformance test. Each solution, which is sent to the test scheduler machine insidethe Testing Framework, will first be parsed into the corresponding JIT configura-tion setting. The test scheduler machine will then start the system with the newconfiguration setting and measure the system performance under a predefined work-load. At the end of each performance test, performance data during the warmedup phase will be collected and sent to the configuration advisor machine, so that90they can be used as objectives to evaluate among solutions in the ESM-MOGA.For our running example, a total of five performance tests with the same work-load but different JIT configuration settings, which correspond to the four initialsolutions in P and the new solution in Q, will be run. Once each test is completed,the test scheduler shuts down the SUS, collects the performance data (responsetime for the individual scenarios, the resource utilization metrics, and JIT logs)and sends the data to the configuration advisor machine.4.4.3 ImplementationWe implemented the ESM-MOGA using the NSGA-II algorithm [39], which is afast and efficient multi-objective genetic algorithm, from the DEAP framework(Distributed Evolutionary Algorithms in Python) [3]. The framework containsthe relevant library functions for NSGA-II, like assigning crowding distance, non-dominated sorting algorithm, and NSGA-II selection. We had to implement thedominance function, and input encoding ourselves to fit into NSGA-II algorithm.We re-implemented the non-dominate sorting and NSGA-II selection functions sothat they can use our dominance function to compare among solutions.We also implemented the automated performance testing framework, whichleverages JMeter [1] as the load generator. The performance testing frameworkcan startup, initialize, execute, and stop a performance test under one particular91Table 4.9: An overview of the three Python-based systems under study.Name Version LOC Application Domain Technology StackSaleor 2017.07.0 48482 E-commerce Gunicorn,Tornado,Postgres,DjangoWagtail 1.12.1 85006 Content Management Gunicorn,Tornado,Postgres,DjangoQuokka 0.2.1 34468 Content Management Gunicorn,Tornado,MongoDB,Flaskconfiguration setting.4.5 Case StudyIn this section, we evaluated the performance of our automated approach to tun-ing the JIT configuration parameters on three Python-based open source systems:Saleor [17], Wagtail [24], and Quokka [15]. As shown in Table 4.9, these three sys-tems vary from system sizes, application domains, and technology stacks. All threesystems can be deployed on top of the Tornado WSGI server which uses Gunicornfor worker process management. And they all require a database to be functional.Saleor is an e-commerce system, built using the Django framework, and uses Post-gres as its database. Wagtail shares the same technology stack as Saleor (a.k.a.,Django and Postgres), but is from a different application domain: the ContentManagement System (CMS). Although Quokka is also a CMS, it is built with the92Flask framework and uses MongoDB as its database.The rest of this section is organized as follows. Section 4.5.1 describes the casestudy setup. Section 4.5.2 explains the case study results.4.5.1 Case Study SetupWe deployed the above three systems on the same physical machines, which have thefollowing hardware configurations: Intel i7-4790 CPU, 16 GB memory, and 2 TBhard-drive. JMeter was deployed on another physical machine with the followinghardware configuration: Intel(R) Core(TM)2 Duo CPU, 4 GB memory, 160 GBhard-drive. And all machines have Ubuntu 14.04 deployed. The reason for theseparate deployment of JMeter and the SUSs is to ensure no overhead caused bythe load generator to the SUSs [64]. The version of the PyPy that we used forevaluation is 5.7.1 which corresponds to Python version 2.7.13. Similar to theexploratory study, we focused on the same six JIT configuration parameters. Hence,each solution (a.k.a., configuration setting) requires 18 bits to be encoded into ourtailored MOGA method. For example, the default configuration setting would beencoded as 011011011011011011. As for the rest of the MOGA configurations, weset the initial population (P ) size as 40, and the mutation rate as 0.10 based on somesmall trials. We also configured our termination criteria to be either |MDR| ≤ 0.1holds for two consecutive generations or the MOGA has iterated for 10 generations.93Table 4.10: Workload description for the three systems.System Workload Mix Workload IntensitySaleorload index page10 req/secload login pagelogin requestview categoryview productadd product to cartview cartcheck out cartselect shipping methodselect shipping addresspaymentpayment confirmWagtailadd blog(1)add event(1)edit blog(2)view blog page(3)view event page(3)Quokkaadd blog(1)edit blog(2)view blog(7)94Table 4.10 shows the workload that we have set for the three systems. Theworkload tries to simulate how real users use the systems in the field. The over-all workload intensity (10 requests/sec) is the same for all three systems and theworkload mix for each system is shown below. For the e-commerce system, Saleor, the workload tries to mimic the purchas-ing workflow from a real customer. Hence, we divided this scenario into 12actions, which correspond to 12 different webpage operations. The overallworkload intensity (10 request/sec) corresponds to 10 different users per-forming the above 12 actions at the same time. Hence, all the actions in thisworkload are assigned with the same ratio in the workload mix. For the content management system, Wagtail, the workload tries to mimicusers reading, posting, and editing blogs or events. Since, in the majorityof the time, users will be viewing the blogs or events, we assigned a higherratio for these two actions. The scenarios of adding a new blog or a newevent happen the least frequently. Hence, they are assigned with the smallestweight in the workload mix. Quokka’s workload is similar to Wagtail’s, as they are in the same applica-tion domain. Since Quokka only supports reading/editing/adding blogs, weadjusted the workload mix accordingly.95As Section 4.3 indicates, only the performance data from the warmed up phaseis representative of the actual system performance. Hence, in the case study, wewant to make sure the system has been running long enough (a.k.a., finished thewarmup up phase). To properly decide the test duration, we first did a test run withthe default configuration setting, in which the predefined workload was executedfor three hours. We leveraged a similar technique as Alghmadi et al. [29] to testwhen the system’s performance behavior gets repetitive, so that we can identifythe duration of the warmup phase. We divided the collected performance datainto intervals of every 20 minutes. Then we performed statistical analyses with theWRS test and CD values on the response time between two adjacent time intervals.We considered the system to be fully warmed up when the response time from thetwo adjacent time intervals show insignificant difference for all scenarios (a.k.a.,not statistically different by the WRS test or CD values show trivial to small effectsizes).We performed the above process in all three case study systems. We foundthat all three systems finish the warmup phase in the first 40 minutes before theirperformance behaviors start to be repetitive. Hence, for consistency concerns, weset the test duration to be 50 minutes for each test and only used the data from thelast 10 minutes (a.k.a., the data from the warmed up phase) for further analysis inthe ESM-MOGA method.96Table 4.11: Statistics after running the MOGA approach on the three case studysystems.System # of # of DurationGenerations Configurations Evaluated (hour)Saleor 3 100 36Wagtail 7 199 67Quokka 3 96 354.5.2 Case Study ResultsFor all the three systems, we applied our ESM-MOGA method with the aforemen-tioned setup. Table 4.11 shows the runtime statistics for the ESM-MOGA methodafter running on the three systems. The search algorithm all terminates under ter-mination criteria |MDR| ≤ 0.1. For Saleor, it takes 36 hours and evaluated 100solutions, 67 hours and 199 solutions for Wagtail, and 35 hours and 96 configurationsettings for Quokka. For all three systems, the ESM-MOGA method found optimalsolutions when it terminated.For each system, we used the NSGA-II selection to select the top three con-figuration settings. We compared the response time and the resource utilizationbetween the optimal configuration settings and the default configuration setting.97lllllCreate Blogdefault top1 top2 top3010002000ConfigurationsResponse time (MSec)lllllllllEdit Blogdefault top1 top2 top30500100015002000ConfigurationsResponse time (MSec)lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllView Blogdefault top1 top2 top30200400600ConfigurationsResponse time (MSec)Figure 4.5: Visualizing the response time distributions of different scenarios underdifferent configuration settings for Wagtail.Table 4.12: Comparing the performance between the optimal configurationsettings and the default configuration setting for the three systems. The optimalconfigurations are labelled as OA, OB, and OC. WS stands for the web server, andDB stands for the database. “-” means the ESM-MOGA suggested optimalconfiguration setting outperforms the default setting and “+” means otherwise.SystemTop Comparing scenarios Average differences (%) Average resource usage difference (%)Configurations better equal worse min max WS CPU WS Memory DB CPU DB MemorySaleorOA12 0 0 -27.11 -56.17 -33.71 +24.80 -3.61 -0.24OB12 0 0 -27.32 -58.93 -25.68 +31.97 -7.59 +1.18OC12 0 0 -9.66 -60.28 -32.38 +12.74 -7.30 +0.84WagtailOA5 0 0 -33.47 -45.10 -23.86 +158.71 -5.96 +1.55OB5 0 0 -29.59 -44.93 -18.81 +202.53 -2.90 -2.04OC5 0 0 -35.93 -44.28 -25.18 +157.94 -3.46 -3.34QuokkaOA2 1 0 -9.39 -34.95 -15.39 +59.34 -13.80 -5.5688OB3 0 0 -13.45 -22.71 -16.51 +61.08 -21.89 +0.942OC1 2 0 -4.98 -25.44 -11.94 +62.03 -19.05 +2.2198Figure 4.5 visually compares the response time distributions between the defaultand the top three optimal configuration settings for Wagtail. Due to space lim-itations, it only contains the performance comparisons of the three blog-relatedscenarios inside Wagtail. Each sub-figure corresponds to one scenario. Within eachsub-figure, the four violin plots correspond to the response time distributions of thatscenario under the default and the three optimal configuration settings. Among allthe sub-figures, the response time under the default configuration setting is signif-icantly much higher than the optimal configuration settings. A similar trend alsoholds for the two event-related scenarios in Wagtail.To quantify the differences between the optimal and the default configurationsettings, we performed the WRS test and CD test between the response time distri-butions under each configuration pair. We used the same criteria as in Section 4.3to judge whether one response time distribution is better, or same, or worse thanthe other one. Table 4.12 shows the results. Each row corresponds to one opti-mal configuration setting for one particular system. There are no orderings amongthe top three optimal configurations (OA, OB, and OC). The second to the fourthcolumns contain the number of scenarios which show better, equal, or worse dif-ference when comparing this configuration setting against the default. For Saleorand Wagtail, all the scenarios performed better under the suggested optimal con-figuration settings. For Quokka, at least one scenarios performed better under the99suggested optimal configuration settings, while the remaining scenarios performedno worse than the default configuration setting. The fifth and the sixth columnsshow the minimum and maximum percentage of differences when comparing theaverage response time under the suggested configuration setting with the averageresponse time under the default for all scenarios. The percentage improvement interms of the average response time can vary between 5% to 60%.Since each system consists of a web server and a database, we further comparedthe CPU and the memory utilizations between the optimal and the default config-uration settings. Last four columns in Table 4.12 shows the comparison results forthe resource utilizations. The CPU usage for both components drops. The decreasein CPU is more significant in the web server, with the average improvement rangesbetween 12% to 33.7%. However, the memory usage for the web server dramati-cally increases (12.7% to 202.5%) across all the optimal configuration settings. Wesuspect this may be due to the storage of the complied jitted code. For a moredetailed discussion, please refer to Section 4.6.4.4.6 DiscussionsIn this section, we discussed the findings based on the case study results and theirimplications.1004.6.1 Optimal Configurations across Different SystemsAs we can see from the previous section (Section 4.5), the optimal configurationsettings obtained using ESM-MOGA significantly out-performed the default con-figuration. In this subsection, we would like to compare the optimal configurationsettings against the default configuration setting in order to see if we can derivesome rules or provide some guidance for PyPy users when tuning the JIT configu-ration settings for their systems.For each of the studied systems above, we obtain its top three optimal configu-ration settings, whose values are shown in Table 4.13. We also included the defaultconfiguration setting in the table to ease comparison.One common pattern as we can see from Table 4.13 is that trace eagerness issignificantly smaller in all the optimal configuration settings when comparing to thedefault ones. trace eagerness refers to the eagerness to compile a non-jitted branchwithin a loop. A system can go through various branches within a loop. A smallertrace eagerness is preferred, so that the branch(es) corresponds to frequently ex-ecuted scenarios will be jitted faster. There is no pattern found in the other fiveconfiguration parameters, as they can be either bigger or smaller than the defaultvalues among the nine optimal configuration settings.We are also interested in understanding the correlation of JIT configuration101Table 4.13: Top three optimal configuration settings for the three studied system.System Config. decay function threshold loop longevity threshold trace eagerness trace limitSaleorOA 10 101 2000 129 25 12000OB 2 1619 250 1039 12 12000OC 2 1619 1000 3117 100 12000WagtailOA 10 101 1000 1039 12 12000OB 2 404 250 259 12 12000OC 2 101 4000 4156 50 12000QuokkaOA 80 404 500 519 12 12000OB 2 3238 2000 4156 12 6000OC 2 202 4000 4156 25 3000Default - 40 1619 1000 1039 200 6000Table 4.14: Spearman correlation between configuration and response time. Thelarge and very large correlation measures are shown in bold.System Correlation decay function threshold loop longevity threshold trace eagerness trace limitSaleorcorr. coeff. -0.5120 -0.1338 0.0291 0.1869 -0.2961 0.6293p− value 6.057e-08 0.1864 0.7748 0.0638 0.0029 3.028e-12scale large small trivial small small largeWagtailcorr. coeff. -0.3719 -0.2750 0.1506 0.1392 -0.5446 0.5790p− value 6.346e-08 8.452e-05 0.0336 0.0498 2.2e-16 2.2e-16scale moderate small small small large largeQuokkacorr. coeff. -0.0403 -0.1932 0.1157 0.0568 0.0506 0.8024p− value 0.7554 0.1323 0.3703 0.6608 0.6956 4.476e-15scale trivial small small trivial trivial very large102parameters to the system performance. We collected all the evaluated configurationsettings and the corresponding response time for different scenarios. We summedup the average response time for all scenarios as the overall response time undera JIT configuration setting. Then we calculated the Spearman’s rank correlationbetween each configuration parameter and the overall response time. Spearman isa non-parametric correlation metric measuring the strength of the relation betweenthe two variables. The scale of the Spearman’s ρ correlation coefficient is indicatedbelow [57]:ρ =trivial if 0 ≤ ρ < 0.1small if 0.1 ≤ ρ < 0.3moderate if 0.3 ≤ ρ < 0.5large if 0.5 ≤ ρ < 0.7very large if 0.7 ≤ ρ < 0.9near perfect if 0.9 ≤ ρ ≤ 1.0Table 4.14 shows the result of the correlation between each configuration pa-rameter and the overall system performance. We highlight the cell in bold if thecorrelation measure is “large” or “very large”. Each system has at least one config-uration parameter which has a “large” or “very large” correlation measure. How-ever, highly correlated configuration parameters vary among the three systems, ex-103cept trace limit. A large trace limit enables the system to compile large frequentlyexecuted loops, which can subsequently improve the system performance. Mean-while, the three JIT configuration parameters function threshold, loop longevity andthreshold have very low correlations (small or trival) with the overall system per-formance.Findings: The configuration parameter trace eagerness should be generallyset lower than the default values in order to obtain better performance.trace limit is highly correlated with the overall system performance, whereasfunction threshold, threshold and loop longevity have no or weak correlations.Implications: There are some general guidance in terms of tuning the PyPy JITconfiguration settings on web frameworks. However, the optimal configurationsettings are still highly system dependent. In this paper, we only evaluatedthe impact of PyPy JIT configuration settings on benchmark programs or webapplications. However, Python is also popular in data statistic analysis andmachine learning (e.g. scipy, tensorflow), which could be time consuming totrain a model. One of the interesting future research area would be to deriverules or general guidance to improve the performance of various machine learningor statistic analysis packages by tuning their JIT configuration parameters.1044.6.2 Top Configurations across Different WorkloadsIn the previous study, we compared the optimal JIT configuration settings andit’s performance cross different applications under the same level of workload. Inthis subsection, we want to compare the optimal JIT configuration settings underdifferent workloads. We selected Wagtail as our experiment subject.In addition to the default Wagtail workload (10 req/sec), we generated two otherworkloads for comparison: 15 req/sec and 5 req/sec, while keeping the workloadmixes. For each of newly generated workload, we ran PyPyJITTuner to derivethe optimal JIT configuration settings. For 15 req/sec workload, the frameworkiterated for 5 generations before termination. And it takes 4 generations for thePyPyJITTuner to be terminated under 5 req/sec workload. The resulting config-uration settings yield significant performance gain (20% - 50%) when compared tothe default configuration setting.Table 4.15 shows the actual performance for the top three configuration settingsunder each workload. It shows the average response time for each scenario in mil-liseconds, as well as various resource usage metrics like CPU and memory usagefor the web server and the database, respectively. In addition, it also shows thenumber of jitted lines under each configuration setting. Although the workloadsare different, all the nine top optimal configuration settings share similar perfor-105Table 4.15: Comparing the performance among the optimal configuration settingsunder different workloads for Wagtail.ExperimentsWorkload 5 req/sec 10 req/sec 15 req/secTop configs (OAOBOC) (OAOBOC) (OAOBOC)add blog 1316.00 1286.00 1237.00 1276.00 1275.00 1191.00 1379.00 1428.00 1366.00Response time add event 1104.00 1073.00 1090.00 1033.00 1190.00 1133.00 1326.00 1409.00 1375.00per scenario edit blog 1077.00 1068.00 920.00 1025.00 1085.00 987.40 1080.00 1143.00 1100.00(msec) view blog 102.50 104.60 111.00 101.30 109.00 104.90 119.00 115.40 117.10view event 89.55 94.81 95.03 89.98 89.06 90.10 98.10 101.0 98.17Resource usageWS CPU (%) 27.89 25.62 23.82 46.16 48.04 44.27 61.63 62.19 63.27WS Memory (MB) 2532.00 2229.00 1848.00 3337.00 3770.00 3214.00 2998.00 3123.00 2944.00DB CPU (%) 0.21 0.22 0.21 0.41 0.40 0.40 0.61 0.59 0.59DB Memory (MB) 87.49 87.85 86.96 90.58 93.90 95.09 97.84 97.90 95.20JIT # of jitted lines 12987 9993 8780 11045 13065 12298 11002 11451 11402mance in terms of response time for each scenario. The number of jitted lines aresimilar across different workloads. As workload increases, the CPU and memoryconsumption for both the web server and the database increases. Hence, in thiscase, the workload intensity is the main reason behind the increase of memory andCPU consumption of the two server components.Table 4.16 shows the top three optimal JIT configuration settings under differentworkloads. The optimal JIT configuration settings under different workloads sharevery similar properties (e.g., large trace limit and small decay and trace eagernessvalues).106Table 4.16: Top three optimal configuration settings for Wagtail under differentworkloads.Workload decay function threshold loop longevity threshold trace eagerness trace limit5 req/sec2 404 250 129 12 1200040 101 250 519 12 1200020 404 500 1039 25 1200010 req/sec10 101 1000 1039 12 120002 404 250 259 12 120002 101 4000 4156 50 1200015 req/sec2 1619 125 3117 25 120002 1619 62 2078 25 1200010 101 62 2078 25 12000Default 1039 1619 40 6000 200 1000107Findings: Varying the workload intensity would not impact the optimized JITperformances. And different workloads would result in different optimal configu-ration values. However, there are some common properties (e.g., large trace limitand small decay and trace eagerness values) shared across them.Implications: Researchers could further improve the efficiency of the ESM-MOGA by developing machine learning algorithms to proactively eliminate someof the performance deficient configuration settings from each generation.4.6.3 Code Jitting vs. PerformanceIn Section 4.3, we have shown that more jitted lines do not necessarily lead to betterperformance. Furthermore, the performance under some of the JIT configurationsettings are even worse than turning the JIT completely off! In this subsection, wewould like to perform a more in-depth study to find out the reasons.In Figure 4.6, we plotted the number of jitted lines with respect to systemperformance across all the runs we did for the Saleor system. And the red dottedline shows the overall average response time under the JIT off configuration setting.As the number of jitted lines increases, the average response time for the systemgradually decreases. As we can see from the figure, there are a few JIT configurationsettings which are even worse than turning the JIT off! We conducted furtheranalysis to understand the reason why some jitted code would even lead to worse108A2500500075005000 10000Number of jitted linesResponse time (MSec)Figure 4.6: Number of jitted lines and overall average response time among allevaluated JIT configuration settings in Saleor. The red dotted line shows theoverall average response time with JIT turned off.109performance.We focused on comparing the code structure under two configuration settings:configuration A, which is a configuration with JIT enabled. As shown in Figure 4.6,the performance of configuration A is worse than turning JIT off. For brevity, wecall the JIT off configuration as configuration B.We first applied cProfile [21] to gather the high-level performance numbers forSaleor. cProfile is a profiling tool for Python-based systems. It can provide infor-mation like the execution time, the number of execution for each of the executedfunctions. We enabled cProfile and ran the Saleor under the default workload twice:one run under configuration A, and the other run under configuration B. After theprofiling, we extracted the total execution time and the frequency of the executionsfor each function. Although the cProfile can provide us with function level profiling,it cannot provide information on which lines are executed during runtime. Hence,we implemented a simple tracer based on Python’s tracing library [22]. Since bothruns executed exactly the same workload, the lines of the executed source codeshould be the same. Hence, we only ran our tracer once. Finally, we parsed thejitted logs we collected for configuration A in order to know the exact lines of sourcecode that were jitted.Based on the cProfiling results, we calculated the average execution time for eachfunction and computed their differences between the two configuration settings.110We sorted the differences in decreasing order and selected top 30 functions whoseperformance is worse in configuration A for manual examination.We found that the main reason behind the worse performance is configurationA is due to the overhead of time switching between the two execution modes (in-terpreted vs. native execution). When a function is executed each time, PyPy willbe running under the default interpreted mode. When a code region is marked asjitted, PyPy will switch from the interpreted mode to the native execution modeand executed the compiled binary code. After the binary code is executed, PyPywill have to switch back to interpreted mode to execute the rest of the function.Executing the compiled binary code is much faster than running the same codeunder the interpreted mode. However, the switching between the two executingmode takes time. Figure 4.7 shows two such examples. The different text stylesare defined as follows: grey: not executed; bold: executed but not jitted; bold & highlighted: jitted under configuration A.As we can see from the Figure 4.7, only a single line is jitted in both functions.Both lines are related to Python list comprehension, which internally execute afor loop. In both cases, PyPy has to switch the execution mode twice: starting111 def patch_vary_headers(response, newheaders): """ Adds (or updates) the "Vary" header in the given HttpResponse object. newheaders is a list of header names that should be in "Vary". Existing headers in "Vary" aren't removed. """ # Note that we need to keep the original order intact, because cache # implementations may rely on the order of the Vary contents in, say, # computing an MD5 hash. if response.has_header('Vary'): vary_headers = cc_delim_re.split(response['Vary']) else: vary_headers = [] # Use .lower() here so we treat headers as case-insensitive. existing_headers = set(header.lower() for header in vary_headers) additional_headers = [newheader for newheader in newheaders if newheader.lower() not in existing_headers] response['Vary'] = ', '.join(vary_headers + additional_headers) def get_javascript_catalog(locale, domain, packages): app_configs = apps.get_app_configs() allowable_packages = set(app_config.name for app_config in app_configs) allowable_packages.update(DEFAULT_PACKAGES) packages = [p for p in packages if p in allowable_packages] paths = [] # paths of requested packages for package in packages: p = importlib.import_module(package) path = os.path.join(os.path.dirname(upath(p.__file__)), 'locale') paths.append(path) trans = DjangoTranslation(locale, domain=domain, localedirs=paths) trans_cat = trans._catalog def patch_vary_headers(response, newheaders): """ Adds (or updates) the "Vary" header in the given HttpResponse object. newheaders is a list of header names that should be in "Vary". Existing headers in "Vary" aren't removed. """ # Note that we need to keep the original order intact, because cache # implementations may rely on the order of the Vary contents in, say, # computing an MD5 hash. if response.has_header('Vary'): vary_headers = cc_delim_re.split(response['Vary']) else: vary_headers = [] # Use .lower() here so we treat headers as case-insensitive. existing_headers = set(header.lower() for header in vary_headers) . additional_headers = [newheader for newheader in newheaders if newheader.lower() not in existing_headers] response['Vary'] = ', '.join(vary_headers + additional_headers) (a) Code snippet 1def get_javascript_catalog(locale, domain, packages): app_configs = apps.get_app_configs() allowable_packages = set(app_config.name for app_config in app_configs) allowable_packages.update(DEFAULT_PACKAGES) packages = [p for p in packages if p in allowable_packages] paths = [] # paths of requested packages for package in packages: p = importlib.import_module(package) path = os.path.join(os.path.dirname(upath(p.__file__)), 'locale') paths.append(path) trans = DjangoTranslation(locale, domain=domain, localedirs=paths) trans_cat = trans._catalog def patch_vary_headers(response, newheaders): """ Adds (or updates) the "Vary" header in the given HttpResponse object. newheaders is a list of header names that should be in "Vary". Existing headers in "Vary" aren't removed. """ # Note that we need to keep the original order intact, because cache # implementations may rely on the order of the Vary contents in, say, # computing an MD5 hash. if response.has_header('Vary'): vary_headers = cc_delim_re.split(response['Vary']) else: vary_headers = [] # Use .lower() here so we treat headers as case-insensitive. existing_headers = set(header.lower() for header in vary_headers) additional_headers = [newheader for newheader in newheaders if newheader.lower() not in existing_headers] response['Vary'] = ', '.join(vary_headers + additional_headers) def get_javascript_catalog(locale, domain, packages): app_configs = apps.get_app_configs() allowable_packages = set(app_config.name for app_config in app_configs) allowable_packages.update(DEFAULT_PACKAGES) packages = [p for p in packages if p in allowable_packages] paths = [] # paths of requested packages for package in packages: p = importlib.import_module(package) path = os.path.join(os.path.dirname(upath(p.__file__)), 'locale') paths.append(path) trans = DjangoTranslation(locale, domain=domain, localedirs=paths) trans_cat = trans._catalog (b) Code snippet 2Figure 4.7: Two code snippets showing the executed code and the jitted codeunder the two configuration settings: A vs. B. Configuration A is a jit-enabedconfiguration shown in Figure 4.6. It has worse performance than configuration B,which is JIT off. The colour scheme i defined as follows: grey coloured code is fornot executed code; bolded black coloured code is for executed but not jitted code;and highlighted bold coloured code is for jitted code under configuration A.112from the interpreted mode to the native execution mode and back. The time savedunder the native execution mode is much smaller than the time takes for switchingbetween the two modes, which causes the performance degradation in configurationA (enabling JIT) when comparing against configuration B (JIT off).Such jitting behavior can be explained using the configuration settings. Theconfiguration A is shown in Table 4.17. For reference, we also included the defaultconfiguration values in the table. The ‘trace limit’ in configuration A is set to bea very small value, which would only allow a small region of code to traced andjitted. The smaller the jitted code region is, the less the performance gain codejitting can bring. In this case, the overhead of frequently switching between thetwo modes out-weights the gain from code jitting.Findings: Enabling code jitting does not necessarily lead to good performance.Some JIT configuration settings can perform worse than the disabling the JITcompletely. This is mainly due to the overhead of switching between the inter-preted and the native execution mode.Implications: Programming language researchers may look into adaptive JITcompilation techniques, which can disable inefficient code jitting behavior duringruntime.113Table 4.17: Configuration A and the default configuration.JIT Configuration Parameter A Defaultdecay 5 40function threshold 404 1619loop longevity 1000 1000threshold 4156 1039trace eagerness 100 200trace limit 374 60004.6.4 JIT vs. Memory UsageThe case studies have shown that by using our automated approach, we are ableto locate JIT configuration settings whose performance significantly outperformthe default configuration setting. The CPU for all the components are better or noworse in the optimal configuration settings than the default. This is mainly becausethe CPU can process the same amount of work much more efficiently when morecode is compiled into efficient machine code. However, the memory usage for theTornado web servers are much worse. The memory usage for the worker processesfor Wagtail even tripled in the optimal configuration settings. Hence, we want toinvestigate whether there is any relation between the amount of code jitted and the114Table 4.18: Spearman correlation between number of jitted line and memoryusage for each system.System correlation p− value scaleSaleor 0.8244878 2.2e-16 very largeWagtail 0.882144 2.2e-16 very largeQuokka 0.8549971 2.2e-16 very largeamount of memory used in the worker processes.For all the performance tests in each case study, we processed the JIT logs toobtain the amount of the jitted source code and extract the memory usage at theend of the test. Then we calculated the Spearman’s rank correlation between thelines of the jitted code and the amount of memory usage.As shown in Table 4.18, there is a very large correlation between the memoryusage of the web server processes and the amount of the jitted code. In otherwords, the larger the amount of the jitted code, the higher the memory usage forthe worker processes. As more code is jitted, a larger amount of compiled machinecode is generated. The generated machine code will be kept in the memory duringthe system execution.115Table 4.19: Runtime statistics for ESM-MOGA under different terminationcriteria.Saleor Wagtail QuokkaStoppage Criteria T0 T1 T2 T0 T1 T2 T0 T1 T2# of Generation 3 3 3 7 3 3 3 3 3Evaluated Configurations 100 100 100 199 97 97 96 96 96Duration (hours) 36 36 36 67 34 34 35 35 35Findings: The improvement in response time using the JIT compilation processis at the cost of higher memory usage.Implications: More jitted code can generally lead to more responsive system.However, the number of jitted lines should be kept in a moderate range as: (1)more jitted code means higher memory usage, and (2) the configuration settingswith the highest amount of jitted lines will not guarantee the best performance.One of the interesting future research work would be incorporating memory usageas one of the objectives in the ESM-MOGA while searching for the optimalconfiguration settings.1164.6.5 Termination criteriaThe above case studies show that among the three studied systems, all top threeconfiguration settings significantly outperform the default configuration setting.We set (T0 :) |MDR| ≤ 0.1 in the hope that there is a higher chance to obtainthe optimal configuration settings, as solutions in the previous generation are goodenough so that little optimization can be made during the last generation beforetermination. However, such conditions may be too strict and cost too much time(≥ 35 hours as shown in Table 4.11). Many systems nowadays need to be updatedmore frequently (e.g., daily or even a few times a day) under the continuous in-tegration/continuous delivery processes. Hence, during the case studies, we alsoexamined the following two termination conditions: (1) (T1 :) |MDR| ≤ 0.25, and(2) (T2 :) |MDR| ≤ 0.50, in the hope that the search process terminates earlier(a.k.a., saving the time for searching), while we are still able to locate the optimalsolutions.Table 4.19 shows the runtime statistics for ESM-MOGA under three differenttermination criteria: T0 , T1 , and T2 . All termination criteria stopped at the samenumber of generations for Saleor and Quokka. However, for Wagtail, the less stricttermination criteria T1 and T2 are met after the third generation. Hence, we alsoextracted the top three configuration settings for Wagtail at the end of the third117generation for further comparison.We performed a pairwise comparison using the dominant comparison functionbetween the top three configuration settings under termination criteria T1 and T2(a.k.a., stopped after the third generation) and the top three configuration settingsunder termination criterion T0 (a.k.a., stopped after the seventh generation). Theresults show that two out of the three top configuration settings under T0 dominateall top three configuration settings under T1 and T2 . The other remaining top threeconfiguration settings under T0 show no dominance when comparing against one ofthe top configuration settings under T1 and T2 . The system performance under thetop three configuration settings under T1 and T2 also shows large improvement forall scenarios when comparing against the default setting.118Findings: We have evaluated the ESM-MOGA under three different terminationcriteria: (1) MDR ≥ 0.50, (2) MDR ≥ 0.25, and (3) MDR ≥ 0.10. The ESM-MOGA can terminate successfully (a.k.a., finding the optimal solutions) underall three criteria for the three case study systems. And the less strict terminationcriteria T1 and T2 can obtain some configuration settings which are as good assome top configuration settings under termination criterion T0 .Implications: There are various configuration parameters within ESM-MOGA.It requires further research to systemically tune ESM-MOGA configuration pa-rameters in order to achieve the best performance (finding the optimal solutionswithin the shortest amount of time). Furthermore, it would be beneficial to com-pare ESM-MOGA against other hyper-parameter tuning techniques (e.g., GoogleViser [52]).4.7 Threats to ValidityIn this section, we will discuss the threats to validity.4.7.1 Construct ValidityTo avoid measurement errors and noise, we repeated each experiment 30 times [50].We leveraged techniques from Alghmadi et al. [29] to determine the durationof performance tests, when the performance behavior becomes repetitive. We have119found that after 40 minutes under the default configuration settings for the threesystems, the performance behavior becomes repetitive. For consistency concerns,we took additional 10 minutes of the performance data for our performance testsfor the warmed up period. We assumed the warmup period would be similar underother configuration settings. To verify this threshold, we randomly sampled fiveperformance test from all performance testing runs in each case study. For eachsampled test, we divided the performance data into intervals of ten minutes andcompared the performance behavior among the adjacent intervals. Our analysesconfirmed that the performance behavior also became repetitive after 40 minutesunder these five sampled configuration settings.Since the JIT logs do not contain timestamps, the only way to monitor thejitting progress for PyPy is to periodically take snapshots of the existing JIT logs.However, regularly taking snapshots of the JIT logs would bring huge performanceoverhead for a server-based system. Hence, to minimize the measurement impact,we did not take snapshots in the middle of the performance tests in our case studyin Section 4.5. Instead, we estimated the jitting progress by judging whether thesystem performance behavior stabilizes (a.k.a., becoming repetitive).Our approach, ESM-MOGA, is a tailed version of the Multi-Objective GeneticAlgorithm, which uses effect size measures to compare the results of different testruns. MOGA is an efficient search-based technique, which automatically explores120the solution space. It has been used widely in various software engineering researchareas (e.g., test case generation [28, 46, 78], software architecture [56], and bugprediction [35]) and is shown to be highly effective. Although our case study resultsshow that the configurations derived from the ESM-MOGA approach yield muchbetter performance than the default configuration, the ESM-MOGA might notbe the most efficient approach to locate the optimal JIT configuration setting(s).Furthermore, the search time that it takes to find the optimal solutions using ESM-MOGA varies depending on the systems and their associated workload. One of thefuture areas of research is to evaluate the effectiveness of various hyper-configurationtuning techniques in the context of tuning JIT configuration parameters.4.7.2 Internal ValidityWe kept all the other factors (e.g., the versions of the systems, the deploymentinfrastructure, and the workload) the same, while varying the JIT configurationsettings for each performance tests.In this paper, we assumed the systems which undergo the JIT tuning process,can handle the exercised workload. In other words, the systems are not in a bottle-neck state when we tune their JIT configurations. It’s a common practice that thetop priority for bottlenecked systems would be performance diagnosis and migrationactions instead of tuning their JIT configuration settings.121We used the WRS test and the values from CD to implement our dominancefunctions in the ESM-MOGA. WRS is a statistical test which compares the distri-butions of two datasets. CD is an effect size measure, which indicates the strengthof the difference between two datasets. Both the WRS test and the CD are non-parametric tests, which do not hold any assumptions regarding the underlyingdistributions of the data. The two techniques have been used together in previousworks [49, 48] to evaluate the system performance between two alternatives.4.7.3 External ValidityIn this paper, we have conducted a case study on the performance impact of theJIT configuration settings from PyPy. The experimented PyPy version is PyPy5.7.1, which corresponds to Python version 2.7.13. The empirical findings in theexploratory studies may not be generalizable to other Python versions (e.g, Python3), other Python implementations (e.g., Jython), or other programming languages(e.g., Java or C#).Our case study results have shown that the optimal JIT configuration settingsvary from systems to systems. Our search-based configuration tuning frameworkcan be used to automatically search for configuration settings, which are muchbetter than the default. Our automated tuning technique can also be used to tunethe JIT performance of other programming languages, whose parameters are integer122types. We plan to extend our approach in the future to accommodate other typesof configuration parameters (e.g., string, float and boolean types).As each experiment requires starting the Python-based applications with a dif-ferent set of configuration parameters, it is not yet practical to apply ESM-MOGAtechniques into the continuous integration and continuous delivery process. One ofthe future areas of research is to look into techniques like transfer learning [62] toinfer the optimal configurations for the newer releases of the same systems or evenconfiguration parameters other systems.4.8 ConclusionThe JIT compilation is introduced to improve the runtime performance of softwaresystems. During the system execution, various regions of the systems are compiledinto the binary executable format, so that they can be executed more efficiently.In this paper, we have performed an empirical study on the performance impactof PyPy’s JIT configuration settings. In particular, we have compared the perfor-mance differences between the default and some other configuration settings. Wehave shown that systems running under the default configuration setting does notnecessarily yield the best performance. In addition, we have shown that there isno strong connection between the JIT coverage and the system performance andthe optimal JIT configuration settings are system dependent. To cope with such123findings, we have developed a search-based approach, called ESM-MOGA, whichautomatically tunes the JIT configuration settings for a given system. Case stud-ies have shown that systems running under the resulting configuration settings aresignificantly faster than the default configuration setting.1245 Conclusions and Future workPython, as one of the most popular programming language, has been widely usedby different domains, like science computing, web service, and data process, dueto its dynamic features, enriched library, and large community. Unfortunately, nomany empirical studies in the software engineering research field have been devotedto Python-based applications. In this thesis, we have conducted an empirical studyon the evolution and the performance of PyPy, a Python-based compiler project.In the first part of this thesis, we studied the evolution of PyPy and comparedour findings with the work of Lin et al. [69], who studied the fine-grained sourcecode changes on ten Python application across five domains (Web, Data processing,Science computing, NLP, Media) and reported 11 interesting findings from fourdimensions. We first implemented a public available Python source code diffingtool PyDiff and applied PyDiff on all PyPy’s historical commits. The result of thereplication study shows that 6 out of 9 findings in the original study doesn’t hold inthe PyPy project. The evolution pattern like change type frequency and dynamic125change actions show significant differences between PyPy and the originally studiedprojects.PyPy mainly gains its performance via its JIT compiler. However, the tuningof the JIT compiler configuration is usually ignored by the PyPy users and leadto suboptimal performance. In the second part of the thesis, we performed anexploratory study on PyPy’s JIT configuration parameters. Our study shows theconfiguration of the JIT compiler can make a big difference to the performance ofapplications running on the top of the compiler. Based on our findings, we proposedour search-based configuration tuning approach, (ESM-MOGA) to tuning the JITconfiguration for the applications running on the top. Case studies on three opensource systems show that systems running under the resulting configuration settingssignificantly out-perform (5% - 60% improvement in average peak performance) thedefault configuration setting.In the future, we plan to extend the scale of our replication study in sourcecode changes. In particular, we want to understand whether similar findings canbe found in Python applications from other domains. Also, we want to see ifthis research can be applied to applications that are written by other dynamicprogram languages (e.g. Java). For our work of JIT configuration tuning, we planto further expand the ESM-MOGA to accommodate more objectives (e.g., memoryand network efficiency) during its search process. We also would like to apply126the ESM-MOGA on other Python-based applications or frameworks (e.g., machinelearning based frameworks like TensorFlow). In addition, we plan to explore theuse of data mining or experimental design techniques to further reduce the numberof performance tests conducted during the search process. Finally, we would liketo evaluate the effectiveness of various hyper-parameter tuning techniques in thecontext of tuning JIT configurations.127Bibliography[1] Apache JMeter. http://jmeter.apache.org/, visited 2015-10-23[2] Chrome V8 JavaScript engine. https://opensource.google.com/projects/v8. Last accessed 2/19/2019[3] Distributed Evolutionary Algorithms in Python (DEAP). https://github.com/DEAP/deap. Last accessed 10/06/2017[4] Django web framework. https://www.djangoproject.com/. Last accessed02/19/2019[5] Evaluation tools in Python (DEAP). http://deap.readthedocs.io/en/master/api/tools.html?highlight=dominance. Last accessed 10/06/2017[6] Flask web framework. http://flask.pocoo.org/. Last accessed 02/19/2019[7] IBM Java 8 JIT and AOT command-line options. https://www.ibm.com/support/knowledgecenter/SSYKE2_8.0.0/com.ibm.java.aix.80.doc/diag/appendixes/cmdline/commands_jit.html. Last accessed 10/06/2017128[8] Java Microbenchmark Harness (JMH). http://openjdk.java.net/projects/code-tools/jmh/. Last accessed 10/06/2017[9] Numpy: scientific computing with Python. http://www.numpy.org/. Lastaccessed 02/19/2019[10] Oracle Java 8 Advanced JIT Compiler Options. https://docs.oracle.com/javase/8/docs/technotes/tools/windows/java.html#BABDDFII. Last ac-cessed 10/06/2017[11] Pandas: python data analysis library. https://pandas.pydata.org/. Lastaccessed 02/19/2019[12] Performance monitoring tools for Linux. https://github.com/sysstat/sysstat, visited 2015-10-23[13] Pydiff. https://bitbucket.org/YangguangLi/pydiffer/src/master/[14] PyPy speed center. http://speed.pypy.org/. Last accessed 10/04/2017[15] Quokka CMS (Content Management System) - Python, Flask and MongoDB.http://quokkaproject.org/. Last accessed 10/06/2017[16] Replication package. https://github.com/seasun525/PyPyJITTuner129[17] Saleor - An e-commerce storefront for Python and Django. http://getsaleor.com/. Last accessed 10/06/2017[18] scikit-learn: machine learning in Python. https://scikit-learn.org/stable/. Last accessed 02/19/2019[19] TechEmpower Web Framework Benchmarks. https://www.techempower.com/benchmarks/. Last accessed 10/04/2017[20] Tensorflow: machine learning framework. https://www.tensorflow.org/.Last accessed 02/19/2019[21] The Python Profilers. https://docs.python.org/2/library/profile.html. Last accessed 10/28/2018[22] Tracing a Program As It Runs. https://pymotw.com/2/sys/tracing.html.Last accessed 10/28/2018[23] vmprof - a statistical program profiler. http://vmprof.com/. Last accessed10/06/2017[24] Wagtail CMS: Django Content Management System. https://wagtail.io/.Last accessed 10/06/2017[25] What is Load Balancing? https://www.nginx.com/resources/glossary/load-balancing/. Last accessed 10/06/2017130[26] What is python used for at Google? https://www.quora.com/What-is-python-used-for-at-Google. Last accessed 10/06/2017[27] Wilcoxon rank-sum test. https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test[28] Abdessalem, R.B., Panichella, A., Nejati, S., Briand, L.C., Stifter, T.: Testingautonomous cars for feature interaction failures using many-objective search.In: Proceedings of the 33rd ACM/IEEE International Conference on Auto-mated Software Engineering (ASE) (2018)[29] Alghmadi, H.M., Syer, M.D., Shang, W., Hassan, A.E.: An automated ap-proach for recommending when to stop performance tests. In: 2016 IEEEInternational Conference on Software Maintenance and Evolution (ICSME),pp. 279–289 (2016)[30] Barrett, E., Bolz-Tereick, C.F., Killick, R., Mount, S., Tratt, L.: Virtualmachine warmup blows hot and cold. Proceedings of the ACM Program-ming Language 1(OOPSLA), 52:1–52:27 (2017). DOI 10.1145/3133876. URLhttp://doi.acm.org/10.1145/3133876[31] Bolz, C.F., Cuni, A., Fijalkowski, M., Rigo, A.: Tracing the meta-level: Pypy’stracing jit compiler. In: Proceedings of the 4th Workshop on the Implementa-131tion, Compilation, Optimization of Object-Oriented Languages and Program-ming Systems (ICOOOLPS), pp. 18–25 (2009)[32] Bondi, A.B.: Automating the analysis of load test results to assess the scal-ability and stability of a component. In: Proceedings of the 2007 ComputerMeasurement Group Conference (CMG), pp. 133–146 (2007)[33] Brecht, T., Arjomandi, E., Li, C., Pham, H.: Controlling garbage collectionand heap growth to reduce the execution time of java applications. ACM Trans.Program. Lang. Syst. 28(5), 908–941 (2006). DOI 10.1145/1152649.1152652.URL http://doi.acm.org/10.1145/1152649.1152652[34] Candan, K.S., Li, W.S., Luo, Q., Hsiung, W.P., Agrawal, D.: Enabling dy-namic content caching for database-driven web sites. In: Proceedings of the2001 ACM SIGMOD International Conference on Management of Data (SIG-MOD), pp. 532–543 (2001)[35] Canfora, G., Lucia, A.D., Penta, M.D., Oliveto, R., Panichella, A., Panichella,S.: Multi-objective cross-project defect prediction. In: Proceedings of the2013 IEEE Sixth International Conference on Software Testing, Verificationand Validation (ICST) (2013)132[36] Chawathe, S.S., Rajaraman, A., Garcia-Molina, H., Widom, J.: Change de-tection in hierarchically structured information. In: ACM SIGMOD Record,vol. 25, pp. 493–504. ACM (1996)[37] Clark, M.: How the BBC builds websites that scale. http://www.creativebloq.com/features/how-the-bbc-builds-websites-that-scale(2017). Last accessed 10/06/2017[38] Cramer, T., Friedman, R., Miller, T., Seberger, D., Wilson, R., Wolczko, M.:Compiling java just in time. IEEE Micro 17(3), 36–43 (1997)[39] Deb, K., Agrawal, S., Pratap, A., Meyarivan, T.: A fast elitist non-dominatedsorting genetic algorithm for multi-objective optimization: Nsga-ii. In: Inter-national Conference on Parallel Problem Solving From Nature, pp. 849–858.Springer (2000)[40] Duan, S., Thummala, V., Babu, S.: Tuning Database Configuration Pa-rameters with iTuned. Proceedings of the VLDB Endowment 2(1), 1246–1257 (2009). DOI 10.14778/1687627.1687767. URL http://dx.doi.org/10.14778/1687627.1687767[41] Eaton, K.: How One Second Could Cost Amazon $1.6 Billion InSales. https://www.fastcompany.com/1825005/how-one-second-could-cost-amazon-16-billion-sales. Last accessed 10/06/2017133[42] Falleri, J.R., Morandat, F., Blanc, X., Martinez, M., Monperrus, M.: Fine-grained and accurate source code differencing. In: Proceedings of the 29thACM/IEEE international conference on Automated software engineering, pp.313–324. ACM (2014)[43] Fluri, B., Gall, H.C.: Classifying change types for qualifying change couplings.In: Program Comprehension, 2006. ICPC 2006. 14th IEEE International Con-ference on, pp. 35–45. IEEE (2006)[44] Fluri, B., Giger, E., Gall, H.C.: Discovering patterns of change types. In: Pro-ceedings of the 2008 23rd IEEE/ACM International Conference on AutomatedSoftware Engineering, pp. 463–466. IEEE Computer Society (2008)[45] Fluri, B., Wuersch, M., PInzger, M., Gall, H.: Change distilling: Tree differ-encing for fine-grained source code change extraction. IEEE Transactions onsoftware engineering 33(11) (2007)[46] Fraser, G., Arcuri, A.: Evosuite: Automatic test suite generation for object-oriented software. In: Proceedings of the 19th ACM SIGSOFT Symposiumand the 13th European Conference on Foundations of Software Engineering(ESEC/FSE) (2011)[47] Gal, A., Eich, B., Shaver, M., Anderson, D., Mandelin, D., Haghighat, M.R.,Kaplan, B., Hoare, G., Zbarsky, B., Orendorff, J., Ruderman, J., Smith, E.W.,134Reitmaier, R., Bebenita, M., Chang, M., Franz, M.: Trace-based just-in-timetype specialization for dynamic languages. In: Proceedings of the 30th ACMSIGPLAN Conference on Programming Language Design and Implementation(PLDI), pp. 465–478 (2009)[48] Gao, R., Jiang, Z.M.J.: An exploratory study on assessing the impact of en-vironment variations on the results of load tests. In: Proceedings of the 14thInternational Conference on Mining Software Repositories (MSR) (2017)[49] Gao, R., Jiang, Z.M.J., Barna, C., Litoiu, M.: A framework to evaluate theeffectiveness of different load testing analysis techniques. In: 2016 IEEE Inter-national Conference on Software Testing, Verification and Validation (ICST)(2016)[50] Georges, A., Buytaert, D., Eeckhout, L.: Statistically Rigorous Java Perfor-mance Evaluation. In: Proceedings of the 22nd International Conference onObject-Oriented Programming, Systems, Languages and Applications (OOP-SLA) (2007)[51] Gewirtz, D.: Which programming languages are most popular (and what doesthat even mean)? http://www.zdnet.com/article/which-programming-languages-are-most-popular-and-what-does-that-even-mean/ (2017).Last accessed 10/06/2017135[52] Golovin, D., Solnik, B., Moitra, S., Kochanski, G., Karro, J., Sculley, D.:Google vizier: A service for black-box optimization. In: Proceedings of the23rd ACM SIGKDD International Conference on Knowledge Discovery andData Mining (KDD) (2017)[53] Gong, L., Pradel, M., Sen, K.: Jitprof: Pinpointing jit-unfriendly javascriptcode. In: Proceedings of the 2015 10th Joint Meeting on Foundations ofSoftware Engineering (ESEC/FSE), pp. 357–368 (2015)[54] Grigorik, I.: Optimizing Encoding and Transfer Size of Text-Based As-sets. https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/optimize-encoding-and-transfer.Last accessed 10/06/2017[55] Hashemi, M.: 10 Myths of Enterprise Python. https://www.paypal-engineering.com/2014/12/10/10-myths-of-enterprise-python/ (2014).Last accessed 10/06/2017[56] Henard, C., Papadakis, M., Harman, M., Traon, Y.L.: Combining multi-objective search and constraint solving for configuring large software prod-uct lines. In: Proceedings of the 37th International Conference on SoftwareEngineering (ICSE) (2015)136[57] Hopkins, W.G.: A new view of statistics. [Online accessed 2017-10-14] http://www.sportsci.org/resource/stats/index.html (2016)[58] Hoskins, D.S., Colbourn, C.J., Montgomery, D.C.: Software performance test-ing using covering arrays: Efficient screening designs with categorical factors.In: Proceedings of the 5th International Workshop on Software and Perfor-mance (WOSP)[59] Hoste, K., Georges, A., Eeckhout, L.: Automated just-in-time compiler tuning.In: Proceedings of the 8th Annual IEEE/ACM International Symposium onCode Generation and Optimization (CGO), pp. 62–72 (2010)[60] Insights, G.P.: Remove Render-Blocking JavaScript. https://developers.google.com/speed/docs/insights/BlockingJS. Last accessed 10/06/2017[61] Jamshidi, P., Siegmund, N., Velez, M., Ka¨stner, C., Patel, A., Agarwal, Y.:Transfer learning for performance modeling of configurable systems: An ex-ploratory analysis. In: Proceedings of the International Conference on Auto-mated Software Engineering (ASE) (2017)[62] Jamshidi, P., Siegmund, N., Velez, M., Ka¨stner, C., Patel, A., Agarwal, Y.:Transfer learning for performance modeling of configurable systems: an ex-ploratory analysis. In: Proceedings of the 32nd IEEE/ACM InternationalConference on Automated Software Engineering (ASE) (2017)137[63] Jantz, M.R., Kulkarni, P.A.: Exploring single and multilevel jit compilationpolicy for modern machines 1. ACM Transactions on Architecture and CodeOptimization (TACO) 10(4), 22:1–22:29 (2013)[64] Jiang, Z.M.J., Hassan, A.E.: A survey on load testing of large-scale softwaresystems. IEEE Transactions on Software Engineering (TSE) (2015)[65] Kampenes, V.B., Dyb˚a, T., Hannay, J.E., Sjøberg, D.I.K.: Systematic review:A systematic review of effect size in software engineering experiments. Infor-mation Software Technology 49(11-12), 1073–1086 (2007)[66] Komorn, R.: Python in production engineering. https://code.facebook.com/posts/1040181199381023/python-in-production-engineering/(2016). Last accessed 10/06/2017[67] Lengauer, P., Mo¨ssenbo¨ck, H.: The Taming of the Shrew: Increasing Per-formance by Automatic Parameter Tuning for Java Garbage Collectors. In:Proceedings of the 5th ACM/SPEC International Conference on PerformanceEngineering (ICPE), pp. 111–122 (2014)[68] Libicˇ, P., Bulej, L., Horky, V., Tu˚ma, P.: On the limits of modeling genera-tional garbage collector performance. In: Proceedings of the 5th ACM/SPECInternational Conference on Performance Engineering (ICPE) (2014)138[69] Lin, W., Chen, Z., Ma, W., Chen, L., Xu, L., Xu, B.: An empirical studyon the characteristics of python fine-grained source code change types. In:2016 IEEE International Conference on Software Maintenance and Evolution(ICSME), pp. 188–199. IEEE (2016)[70] Lion, D., Chiu, A., Sun, H., Zhuang, X., Grcevski, N., Yuan, D.: Don’t getcaught in the cold, warm-up your jvm: Understand and eliminate jvm warm-up overhead in data-parallel systems. In: Proceedings of the 12th USENIXConference on Operating Systems Design and Implementation (OSDI), pp.383–400 (2016)[71] Malloy, B.A., Power, J.F.: An empirical analysis of the transition from python2 to python 3. Empirical Software Engineering pp. 1–28 (2018)[72] Mart´ı, L., Garc´ıa, J., Berlanga, A., Molina, J.M.: An approach to stoppingcriteria for multi-objective optimization evolutionary algorithms: The mgbmcriterion. In: Evolutionary Computation, 2009. CEC’09. IEEE Congress on,pp. 1263–1270. IEEE (2009)[73] Oaks, S.: Java Performance: The Definitive Guide, 1st edn. O’Reilly Media,Inc. (2014)[74] Osogami, T., Kato, S.: Optimizing System Configurations Quickly by Guess-ing at the Performance. In: Proceedings of the 2007 ACM SIGMETRICS In-139ternational Conference on Measurement and Modeling of Computer Systems(SIGMETRICS) (2007)[75] Romano, D., Pinzger, M.: Analyzing the evolution of web services using fine-grained changes. In: 2012 IEEE 19th International Conference on Web Ser-vices, pp. 392–399. IEEE (2012)[76] Romano, J., Kromrey, J.D., Coraggio, J., Skowronek, J.: Appropriate statis-tics for ordinal level data: Should we really be using t-test and Cohen’sd forevaluating group differences on the NSSE and other surveys? In: Annualmeeting of the Florida Association of Institutional Research (2006)[77] Rossum, G.: Python reference manual (1995)[78] Shamshiri, S., Rojas, J.M., Fraser, G., McMinn, P.: Random or genetic algo-rithm search for object-oriented test suite generation? In: Proceedings of the2015 Annual Conference on Genetic and Evolutionary Computation (GECCO)(2015)[79] Shang, W., Jiang, Z.M., Adams, B., Hassan, A.E.: Mapreduce as a generalframework to support research in mining software repositories (msr). In: Min-ing Software Repositories, 2009. MSR’09. 6th IEEE International WorkingConference on, pp. 21–30. IEEE (2009)140[80] Siegmund, N., Grebhahn, A., Apel, S., Ka¨stner, C.: Performance-influencemodels for highly configurable systems. In: Proceedings of the 2015 10thJoint Meeting on Foundations of Software Engineering, ESEC/FSE 2015. ACM(2015)[81] Siegmund, N., Kolesnikov, S.S., Ka¨stner, C., Apel, S., Batory, D., Rosenmu¨ller,M., Saake, G.: Predicting Performance via Automated Feature-interactionDetection. In: Proceedings of the 34th International Conference on SoftwareEngineering (ICSE) (2012)[82] Singer, J., Brown, G., Watson, I., Cavazos, J.: Intelligent selection ofapplication-specific garbage collectors. In: Proceedings of the 6th Interna-tional Symposium on Memory Management, ISMM ’07[83] Singh, R., Bezemer, C.P., Shang, W., Hassan, A.E.: Optimizing theperformance-related configurations of object-relational mapping frameworksusing a multi-objective genetic algorithm. In: Proceedings of the 7thACM/SPEC on International Conference on Performance Engineering (ICPE),pp. 309–320 (2016)[84] Sopitkamol, M., Menasce´, D.A.: A method for evaluating the impact of soft-ware configuration parameters on e-commerce sites. In: Proceedings of the1415th International Workshop on Software and Performance (WOSP), pp. 53–64(2005)[85] Thonangi, R., Thummala, V., Babu, S.: Finding Good Configurations inHigh-Dimensional Spaces: Doing More with Less. In: 2008 IEEE Interna-tional Symposium on Modeling, Analysis and Simulation of Computers andTelecommunication Systems (2008)[86] Thung, F., Lo, D., Jiang, L.: Automatic recovery of root causes from bug-fixing changes. In: 2013 20th Working Conference on Reverse Engineering(WCRE), pp. 92–101. IEEE (2013)[87] Wang, K., Lin, X., Tang, W.: Predator - An experience guided configurationoptimizer for Hadoop MapReduce. In: 4th IEEE International Conference onCloud Computing Technology and Science Proceedings (2012)[88] Wimmer, C., Brunthaler, S.: Zippy on truﬄe: A fast and simple implemen-tation of python. In: Proceedings of the 2013 Companion Publication forConference on Systems, Programming, & Applications: Software for Hu-manity (SPLASH) (2013)[89] Wu¨rthinger, T., Wimmer, C., Humer, C., Wo¨ß, A., Stadler, L., Seaton, C.,Duboscq, G., Simon, D., Grimmer, M.: Practical partial evaluation for high-performance dynamic language runtimes. In: Proceedings of the 38th ACM142SIGPLAN Conference on Programming Language Design and Implementation,PLDI 2017, pp. 662–676. ACM, New York, NY, USA (2017). DOI 10.1145/3062341.3062381. URL http://doi.acm.org/10.1145/3062341.3062381[90] Wu¨rthinger, T., Wimmer, C., Wo¨ß, A., Stadler, L., Duboscq, G., Humer,C., Richards, G., Simon, D., Wolczko, M.: One vm to rule them all. In:Proceedings of the 2013 ACM International Symposium on New Ideas, NewParadigms, and Reflections on Programming & Software (Onward!) (2013)[91] Xi, B., Liu, Z., Raghavachari, M., Xia, C.H., Zhang, L.: A Smart Hill-climbingAlgorithm for Application Server Configuration. In: Proceedings of the 13thInternational Conference on World Wide Web (WWW), pp. 287–296. ACM,New York, NY, USA (2004). DOI 10.1145/988672.988711. URL http://doi.acm.org/10.1145/988672.988711[92] Xu, T., Jin, X., Huang, P., Zhou, Y., Lu, S., Jin, L., Pasupathy, S.: EarlyDetection of Configuration Errors to Reduce Failure Damage. In: Proceedingsof the 12th USENIX Conference on Operating Systems Design and Implemen-tation (OSDI) (2016)[93] Yilmaz, C., Porter, A., Krishna, A.S., Memon, A.M., Schmidt, D.C., Gokhale,A.S., Natarajan, B.: Reliable effects screening: A distributed continuous qual-ity assurance process for monitoring performance degradation in evolving soft-143ware systems. IEEE Transactions on Software Engineering (TSE) 33(2), 124–141 (2007). DOI 10.1109/TSE.2007.20144

VyPR Analysis Library

Abstract

Extracted data

VyPR Analysis Library

Abstract

Extracted data

Related items

Related items