W&M ScholarWorks Dissertations, Theses, and Masters Projects Theses, Dissertations, & Master Projects 2020 Deep Learning In Software Engineering Cody Watson William & Mary - Arts & Sciences, cawatson@email.wm.edu Follow this and additional works at: https://scholarworks.wm.edu/etd Part of the Computer Sciences Commons Recommended Citation Watson, Cody, "Deep Learning In Software Engineering" (2020). Dissertations, Theses, and Masters Projects. Paper 1616444371. http://dx.doi.org/10.21220/s2-3bn1-fv48 This Dissertation is brought to you for free and open access by the Theses, Dissertations, & Master Projects at W&M ScholarWorks. It has been accepted for inclusion in Dissertations, Theses, and Masters Projects by an authorized administrator of W&M ScholarWorks. For more information, please contact scholarworks@wm.edu. Deep Learning in Software EngineeringCody Allen WatsonWilliamsburg, Virginia, USABachelor of Arts, Wofford College, 2015Master of Science, College of William & Mary, 2017A Dissertation presented to the Graduate Facultyof The College of William & Mary in Candidacy for the Degree ofDoctor of PhilosophyDepartment of Computer ScienceCollege of William & MaryAugust 2020© Copyright by Cody Watson 2019APPROVAL PAGEThis Dissertation is submitted in partial fulfillment ofthe requirements for the degree ofDoctor of PhilosophyCody WatsonApproved by the Committee, August 2020Committee ChairDenys Poshyvanyk, Associate Professor, Computer ScienceCollege of William & MaryAndreas Stathopoulos, Professor, Computer ScienceCollege of William & MaryRobert Michael Lewis, Associate Professor, Computer ScienceCollege of William & MaryZhenming Liu, Assistant Professor, Computer ScienceCollege of William & MaryMassimiliano Di Penta, Associate Professor, Computer ScienceUniversity of Sannio DPABSTRACTSoftware evolves and therefore requires an evolving field of Software Engineering.The evolution of software can be seen on an individual project level through thesoftware life cycle, as well as on a collective level, as we study the trends and usesof software in the real world. As the needs and requirements of users change, somust software evolve to reflect those changes. This cycle is never ending and hasled to continuous and rapid development of software projects. More importantly, ithas put a great responsibility on software engineers, causing them to adoptpractices and tools that allow them to increase their efficiency. However, thesetools suffer the same fate as software designed for the general population; theyneed to change in order to reflect the user’s needs. Fortunately, the demand forthis evolving software has given software engineers a plethora of data and artifactsto analyze. The challenge arises when attempting to identify and apply patternslearned from the vast amount of data.In this dissertation, we explore and develop techniques to take advantage of thevast amount of software data and to aid developers in software development tasks.Specifically, we exploit the tool of deep learning to automatically learn patternsdiscovered within previous software data and automatically apply those patternsto present day software development. We first set out to investigate the currentimpact of deep learning in software engineering by performing a systematicliterature review of top tier conferences and journals. This review providesguidelines and common pitfalls for researchers to consider when implementing DL(Deep Learning) approaches in SE (Software Engineering). In addition, the reviewprovides a research road map for areas within SE where DL could be applicable.Our next piece of work developed an approach that simultaneously learneddifferent representations of source code for the task of clone detection. We foundthat the use of multiple representations, such as Identifiers, ASTs, CFGs andbytecode, can lead to the identification of similar code fragments. Through the useof deep learning strategies, we automatically learned these different representationswithout the requirement of hand-crafted features. Lastly, we designed a novelapproach for automating the generation of assert statements through seq2seqlearning, with the goal of increasing the efficiency of software testing. Given thetest method and the context of the associated focal method, we automaticallygenerated semantically and syntactically correct assert statements for a given,unseen test method.We exemplify that the techniques presented in this dissertation provide ameaningful advancement to the field of software engineering and the automation ofsoftware development tasks. We provide analytical evaluations and empiricalevidence that substantiate the impact of our findings and usefulness of ourapproaches toward the software engineering community.TABLE OF CONTENTSAcknowledgments viDedication viiList of Tables viiiList of Figures ix1 Introduction 21.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Motivation - Why Deep Learning? . . . . . . . . . . . . . . . . . . . . 41.3 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.4 Contributions and Outline . . . . . . . . . . . . . . . . . . . . . . . . 82 Background 122.1 Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.2 Systematic Literature Reviews . . . . . . . . . . . . . . . . . . . . . . 142.3 Recursive Autoencoders . . . . . . . . . . . . . . . . . . . . . . . . . . 152.4 Recurrent Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . 172.5 Neural Machine Translation . . . . . . . . . . . . . . . . . . . . . . . . 182.5.1 Open Vocabulary Problem . . . . . . . . . . . . . . . . . . . . 202.5.2 Multiple Predictions with Beam Search . . . . . . . . . . . . . 202.6 Automatically Generating Assert Statements . . . . . . . . . . . . . . 22i3 Systematic Literature Review in Deep Learning for Software Engineering 263.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.2 RQ Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313.3 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363.3.1 Searching for Primary Studies . . . . . . . . . . . . . . . . . . . 373.3.2 Filtering Through Inclusion Criteria . . . . . . . . . . . . . . . 393.3.3 Snowballing and Manual Additions . . . . . . . . . . . . . . . . 403.3.4 Exclusion Criteria and Quality Analysis . . . . . . . . . . . . . 403.3.5 Data Extraction and Analysis . . . . . . . . . . . . . . . . . . . 413.3.5.1 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . 423.3.5.2 Correlations . . . . . . . . . . . . . . . . . . . . . . . . 443.3.5.3 Association Rule Learning . . . . . . . . . . . . . . . . 443.3.6 Taxonomy Synthesis . . . . . . . . . . . . . . . . . . . . . . . . 463.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463.4.1 RQ1: What types of SE tasks have been addressed by DL-basedapproaches? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463.4.2 RQ2: How are software artifacts being extracted, prepared andused in DL based approaches for SE tasks? . . . . . . . . . . . 503.4.2.1 What types of data are being used? . . . . . . . . . . . 513.4.2.2 How is data being extracted and pre-processed into aformat that the DL model can appropriately learn from? 533.4.3 What type of exploratory data analysis is conducted to helpinform model design and training? . . . . . . . . . . . . . . . . 573.4.4 RQ3: What deep learning models are used to support SE tasks? 593.4.4.1 What types of model architectures are used to performautomated feature engineering of the data related tovarious SE tasks? . . . . . . . . . . . . . . . . . . . . . 59ii3.4.4.2 What learning algorithms and training processes areused in order to optimize the models? . . . . . . . . . 653.4.4.3 What methods are employed to combat overfitting andunderfitting of the models? . . . . . . . . . . . . . . . 683.4.5 RQ4: How well do DL tasks perform in supporting various SEtasks? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 723.4.5.1 What “baseline” techniques are used to evaluate DLmodels and what benchmarks are used for these com-parisons? . . . . . . . . . . . . . . . . . . . . . . . . . 733.4.5.2 How is the impact or automatization of DL approachesmeasured and in what way do these models promotegeneralizability? . . . . . . . . . . . . . . . . . . . . . 783.4.6 RQ5: What common factors contribute to the difficulty repro-ducing DL studies in SE? . . . . . . . . . . . . . . . . . . . . . 823.5 Guidelines for DL in SE . . . . . . . . . . . . . . . . . . . . . . . . . . 863.6 SE4DL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 893.7 Additional Classifications . . . . . . . . . . . . . . . . . . . . . . . . . 903.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 914 Learning Code Similarities 954.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 954.2 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 984.2.1 Code Fragments Selection . . . . . . . . . . . . . . . . . . . . . 994.2.2 Code Representation Extraction . . . . . . . . . . . . . . . . . 994.2.2.1 Identifiers . . . . . . . . . . . . . . . . . . . . . . . . . 1004.2.2.2 AST . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1004.2.2.3 CFG . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101iii4.2.2.4 Bytecode . . . . . . . . . . . . . . . . . . . . . . . . . 1014.2.3 Embedding Learning . . . . . . . . . . . . . . . . . . . . . . . . 1024.2.3.1 DL Strategy . . . . . . . . . . . . . . . . . . . . . . . . 1024.2.3.2 Graph Embedding Strategy . . . . . . . . . . . . . . . 1044.2.4 Detecting Similarities . . . . . . . . . . . . . . . . . . . . . . . 1064.2.5 Combined Models . . . . . . . . . . . . . . . . . . . . . . . . . 1064.3 Experimental Design . . . . . . . . . . . . . . . . . . . . . . . . . . . 1084.3.1 Datasets Selection . . . . . . . . . . . . . . . . . . . . . . . . . 1094.3.1.1 Projects . . . . . . . . . . . . . . . . . . . . . . . . . . 1094.3.1.2 Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . 1094.3.2 Experimental Procedure and Data Analysis . . . . . . . . . . . 1094.3.2.1 RQ1:How effective are different representations in de-tecting similar code fragments? . . . . . . . . . . . . . 1104.3.2.2 RQ2:What is the complementarity of different repre-sentations? . . . . . . . . . . . . . . . . . . . . . . . . 1114.3.2.3 RQ3:How effective are combined multi-representationmodels? . . . . . . . . . . . . . . . . . . . . . . . . . . 1124.3.2.4 RQ4:Are DL-based models applicable for detecting clonesamong different projects? . . . . . . . . . . . . . . . . 1134.3.2.5 RQ5:Can trained DL-based models be reused on dif-ferent, previously unseen projects? . . . . . . . . . . . 1144.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1154.5 Threats To Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1244.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1255 Deep Learning Assert Statements 1265.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126iv5.2 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1295.2.1 GitHub Mining . . . . . . . . . . . . . . . . . . . . . . . . . . . 1305.2.2 Method Extraction . . . . . . . . . . . . . . . . . . . . . . . . . 1315.2.3 Identifying Focal Methods . . . . . . . . . . . . . . . . . . . . . 1325.2.4 Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1335.2.4.1 Vocabulary . . . . . . . . . . . . . . . . . . . . . . . . 1335.2.5 Test-Assert Pairs and Abstraction . . . . . . . . . . . . . . . . 1345.2.6 Sequence to Sequence Learning . . . . . . . . . . . . . . . . . . 1375.2.7 Encoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1375.2.8 Attention Mechanism . . . . . . . . . . . . . . . . . . . . . . . 1385.2.9 Decoder and Copy Mechanism . . . . . . . . . . . . . . . . . . 1395.3 Experimental Design . . . . . . . . . . . . . . . . . . . . . . . . . . . 1415.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1475.5 Threats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1545.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1556 Conclusions and Future Work 158vACKNOWLEDGMENTSThis Dissertation and the work it encompasses would not have been possiblewithout the aid and support of many individuals. First, I would first like toexpress my gratitude to my adviser, Dr. Denys Poshyvanyk, for everything he hasdone for me throughout my Ph.D. studies. Denys is an incredible adviser, trustedmentor, compassionate person and dear friend. Denys believed in me, when I hadtrouble believing in myself and pushed me to strive toward goals I never thought Icould achieve. I am truly blessed to have the opportunity to learn from Denys andwould not be where I am today without him. I would also like to thank Dr.Angela Shiflet and Dr. George Shiflet for their guidance and impact in helping meachieve my goals of attending graduate school. They are a major inspirationtoward my passion in becoming an undergraduate mentor and educator. I wouldalso like to thank my distinguished committee members, Andreas, Robert,Zhenming and Massimiliano for their mentorship and feedback throughout thewriting of this dissertation.This dissertation is also a product of the effort and work done by my friends andintelligent collaborators. Special thanks to Michele for all of your patience andhard work on our research projects, you taught me a lot and I value your wisdom.Thanks to Kevin, in many ways I look up to you as an incredible researcher andteacher. Thanks David for your friendship, passion and dedication to our work.Thanks Carlos for always making the lab a great place to be. Thanks Nathan, itsencouraging to see the excitement of the next generation of SEMERU researchers.Lastly, thanks to Marty, Gabriele and Massimilano for your guidance andexpertise, it is a privilege to know such brilliant individuals.I want to thank my parents Mark and Roberta, my siblings Sarina, Gavin, and mygrandmother Dolores. Mom and Dad, words cannot express my love and gratitudefor all you did for me. It amazes me that two people could be so loving and caringwhile also sacrificing to give their children every opportunity to succeed. Sarinaand Gavin, the gifts you have and challenges you have overcome are an inspirationto me and to those around you. Thank you Grandma, your unwavering supportgave me a solid foundation to accomplish my goals. Thank you to Harris andKathy Settle, your love and encouragement is invaluable to me.Most importantly, I want to thank my lovely wife Laura. You leave me speechlessin the way you support, encourage, care for, motivate, challenge and love me. Icould have never completed my doctoral studies without you. You celebrated withme in my accomplishments and supported me when I was struggling. The sacrificesyou made allowed me to accomplish my dreams and I am forever thankful.viTo my wife Laura - All of my accomplishments rest in the love and support youhave given me. I am so grateful for the gift that you are, and I thank God everyday for the opportunity to share my life with his beloved daughter. I love you.viiLIST OF TABLES3.1 SLR Inclusion and Exclusion Criteria . . . . . . . . . . . . . . . . . . 393.2 SE Task Taxonomy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493.3 Metrics Used for Evaluation . . . . . . . . . . . . . . . . . . . . . . . 774.1 Project Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1084.2 Performances for different clone partitions . . . . . . . . . . . . . . . . 1154.3 Performances for different representations . . . . . . . . . . . . . . . . 1164.4 Complementarity Metrics . . . . . . . . . . . . . . . . . . . . . . . . . 1194.5 Spearman’s rank correlation . . . . . . . . . . . . . . . . . . . . . . . 1204.6 Performance of the CloneDetector . . . . . . . . . . . . . . . . . . . . 1214.7 Performance of the CloneClassifier . . . . . . . . . . . . . . . . . . . . 1214.8 Model Reusability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1235.1 Prediction Classification . . . . . . . . . . . . . . . . . . . . . . . . . . 1465.2 Types of Predicted Assert Statements . . . . . . . . . . . . . . . . . . 1515.3 Learning Based vs. Frequency Based . . . . . . . . . . . . . . . . . . . 152viiiLIST OF FIGURES1.1 Feature engineering an image of a tucan. . . . . . . . . . . . . . . . . 51.2 Traditional Programming vs. Software 2.0 Programming. . . . . . . . 72.1 Recursive Autoencoder . . . . . . . . . . . . . . . . . . . . . . . . . . 162.2 RNN Unfolded . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.3 Neural Machine Translation Model . . . . . . . . . . . . . . . . . . . . 192.4 Beam Search Example . . . . . . . . . . . . . . . . . . . . . . . . . . . 212.5 Software Testing Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . 233.1 The Components of Learning . . . . . . . . . . . . . . . . . . . . . . . 323.2 SLR Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383.3 Venue distribution of DL4SE . . . . . . . . . . . . . . . . . . . . . . . 413.4 SE Tasks Per Year . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473.5 Data Types With SE Tasks . . . . . . . . . . . . . . . . . . . . . . . . 513.6 Preprocessing Techniques by SE Task . . . . . . . . . . . . . . . . . . 553.7 DL Model Taxonomy . . . . . . . . . . . . . . . . . . . . . . . . . . . 603.8 DL Architectures by Task . . . . . . . . . . . . . . . . . . . . . . . . . 623.9 DL Architectures by Data Type . . . . . . . . . . . . . . . . . . . . . 643.10 Overfitting and Underfitting Overview . . . . . . . . . . . . . . . . . . 693.11 Overfitting Techniques per Task Type . . . . . . . . . . . . . . . . . . 703.12 Overfitting Techniques per Data Type . . . . . . . . . . . . . . . . . . 713.13 Benchmark Usage DL in SE . . . . . . . . . . . . . . . . . . . . . . . 753.14 Impact of DL4SE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79ix3.15 Evidence of Occam’s Razor . . . . . . . . . . . . . . . . . . . . . . . . 813.16 Contributing Factors to Reproducibility . . . . . . . . . . . . . . . . . 833.17 Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 933.18 Additional Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . 944.1 First iteration to encode a stream of tokens. . . . . . . . . . . . . . . . 1034.2 HOPE Embedding Technique . . . . . . . . . . . . . . . . . . . . . . . 1055.1 Overview of ATLAS Workflow . . . . . . . . . . . . . . . . . . . . . . . 1305.2 Overview of the abstraction process . . . . . . . . . . . . . . . . . . . 1365.3 The RNN bidirectional encoder decoder model . . . . . . . . . . . . . 1385.4 Copy Mechanism Example . . . . . . . . . . . . . . . . . . . . . . . . 1405.5 Edit Distance between Imperfect Predictions and Ground Truth (Trun-cated tail of higher edit distances) . . . . . . . . . . . . . . . . . . . . 1485.6 Example of a Perfect Prediction Made by the Abstract Model, and itsunabstracted mapping to actual code . . . . . . . . . . . . . . . . . . 1495.7 Seconds per Assert Generation . . . . . . . . . . . . . . . . . . . . . . 153xDeep Learning in Software EngineeringChapter 1Introduction1.1 OverviewSoftware engineering (SE) research investigates questions pertaining to the design, devel-opment, maintenance, testing and evaluation of software systems. As software applicationspervade a wide range of industries, both open- and closed-source code repositories havegrown to become unprecedentedly large and complex. This results in an increase of unstruc-tured, unlabeled, yet important data including requirements, design documents, sourcecode files, test cases and defect reports. Previously, the software engineering communityhas applied traditional machine learning (ML) techniques to identify interesting patternsand unique relationships within this data. Unfortunately, the process of implementing MLtechniques can be time consuming and is limited with regard to the representations thatcan be manually engineered from analyzed data. With recent improvements in compu-tational power and the amount of memory available in modern computer architectures,an advancement to traditional ML approaches has arisen called Deep Learning (DL). DLrepresents a fundamental shift in the manner by which machines learn patterns from databy automatically extracting salient features for a given computational task as opposed torelying on human intuition. DL approaches are characterized by architectures comprisedof several layers that perform mathematical transformations, according to sets of learnable2parameters, on data passing through them. These computational layers and parametersform models that can be trained for specific tasks by updating the parameters accordingto a model’s performance on a labeled set of training data. Given the immense amount ofdata in software repositories that can serve as training data, DL techniques have usheredin advancements across a range of tasks in software engineering research.The use of DL as a tool for software development can allow for effective and efficientsoftware production since it has the potential to extend to every part of the software lifecycle. However, the SE community has taken advantage of this tool without much regardfor its limitations. This is in part due to the mystical, black-box nature of DL algorithms,which are difficult to visualize and often include complex parameterization. The challengesof applying DL models differs from that of customary approaches since a majority of thecomplexity associated with the implementation is composed of selecting the appropriatemodel architecture and correctly evaluating the model’s performance. The selection andimplementation of these model architectures can be a convoluted task because they areheavily dependent on the large numbers of parameters, which can have significant bearingon model performance. Therefore, SE researchers have been met with frustration whentrying to determine when DL is beneficial, as well as how to apply the most effectivearchitecture. In this dissertation, we look to demystify the use of DL algorithms andprovide clarity for when a DL solution may be applicable. Our goal is not to limit the useof DL in SE research, but rather to identify the limitations so that software engineers canuse this tool to its fullest potential without exceeding its capabilities.The clear purpose for the use of DL in SE is automation. These models do not merelyautomate a SE task, rather, they use data to automate the generation of a solution tothe SE task. This shifts the burden of engineering a solution from the developer to themachine. This monumental shift in responsibility reduces the amount of human error andcan more accurately account for the patterns within the data that are undetectable froma developer’s perspective. The applications of DL to improve and automate SE tasksand the dire need of developer support for creating, testing, and maintaining DL projects3more broadly points to an apparent synergy between ongoing research in SE and DL. Thisdissertation will serve as an example of this synergy and will exemplify the power of DL inSE when applied within the proper constraints, thus facilitating the further usage of DLwithin the SE research community.1.2 Motivation - Why Deep Learning?The use of statistically based approaches to model software and its artifacts is a well estab-lished notion. The practicality of using these statistical approaches intrinsically involves theuse of natural language processing (NLP), which brings up the question "is software natu-ral?" Many SE papers have studied the idea of "natural" software [174, 77, 126, 225, 101]to justify or refute the use of natural language models on software. However, the intro-duction of canonical machine learning (ML) and eventually DL algorithms, shored up theweaknesses of NLP methods in two significant ways. The first is the ability to account forlonger-range dependencies of variable length data, while NLP models can only account forthe immediate context. The second is the capability of learning a richer notion of similar-ity across different contexts of data, while NLP algorithms are more rigid in their abilityto detect similar data [20]. Both limitations of NLP algorithms significantly hinder theability to effectively model software repositories. Therefore, a large influx of canonical MLand DL based strategies were applied to SE tasks to evaluate the increased performancefrom NLP.Prior to the use of DL strategies, canonical ML approaches were at the forefront ofapplications to SE tasks. These algorithms require the user to manually identify andcompile expressive features to describe the data, which the ML algorithm will then use togenerate meaningful representations. For example, consider the task of image classificationwhere the goal is to determine the type of an animal within a picture. Although not strictlya SE task, image classification highlights the principals of feature engineering that areinherent to the design of canonical ML based approaches. Designing a machine learning4algorithm with hand-crafted features to accomplish such a task can be cumbersome, as ahuman would have to extract the salient image features (e.g., lines, colors, etc) as shownin figure 1.1. These features will likely be important for recognizing patterns delineatingdifferent classes of data. Furthermore, even if effective features are discovered, the classdivisions of the data may be difficult to learn via simple (e.g. linear, logistic) functions andmay instead require a more complex kernel that could be difficult or effectively impossibleto properly learn without a richer representation of data features. This process, knownas feature engineering, can be lengthy and lead to a large amount of technical debt forreal-world ML systems [238].Figure 1.1: Feature engineering an image of a tucan.SE research can require the analysis of complicated, feature rich data, so the relianceon feature engineering is not always plausible. Therefore, there is a need for automatedapproaches which can extract as many, or as few, salient features as necessary to accuratelyclassify data. DL has shown to be an effective solution for the feature engineering problemwhile maintaining the ability to accurately represent large, heterogeneous, datasets in situ.Considering the complexity of software repositories, SE research needs approaches basedon the science of building machines to automatically discover discriminatory features asopposed to the traditional art of intuitive feature engineering. These approaches learn5kernels rather than simply applying kernels [45]. Along with the advancement towardautomatic feature engineering, DL algorithms transition from the use of shallow architec-tures to the use of hierarchical representations. This advancement in the complexity of thearchitecture allows SE researchers to better conceptualize software repositories. Thus, wecan more accurately model software artifacts toward the goal of automating SE tasks fordevelopers.While these DL techniques bring with them unprecedented advancements, they alsobring with them unprecedented and opaque complexity. Thus, SE researchers need togain a better understanding of where DL has been both effectively and ineffectively usedto support SE tasks in order to learn successful data collection, processing, architectures,representations used, as well as neglected tasks that serve as prime candidates for future DLapplications with large impact. The rapid adoption of DL based approaches has outpacedthe analysis of applicability and limitations. SE researchers have applied this new techniquebefore contemplating the applicability to the specific SE task or the limitations which canbe difficult to determine. There is a need within the SE community to stop and evaluatewhat SE tasks DL has been applied to, and more importantly, what challenges were facedwhen applying these complex architectures. With this evaluation, DL algorithms can bemore accessible to SE researchers with the understanding of how and when they can beapplied to SE tasks.1.3 ApplicationsThe purpose of the work presented in this dissertation focuses on two major goals, bothof which propel the use of DL for SE (DL4SE) research:1. To propagate the knowledge and experience from previous implementations of DL inSE that will further drive the analysis of complex SE repositories62. To demonstrate the analysis and automation of software engineering tasks for de-velopers through the use of DL algorithms, which removes the obstacle of manualfeature extractionTo exemplify the need for continued research in DL4SE, we present work in the domain ofsoftware maintenance and software testing. These domains reflect back on the idea thatDL can be applied to every faction of the software development life cycle. Although theseworks display advancement within their particular factions of the software developmentlife cycle, they also represent the idea of automating developer focused tasks through DLmodalities. We will show that through the numerous amount of software artifact dataavailable, we can build successful systems that learn high-level, invariant features usedspecifically for the tasks of clone detection and test code generation.Figure 1.2: Traditional Programming vs. Software 2.0 Programming.The overarching idea of these two areas contributes to the changing dynamic of softwaredevelopment and thus software engineering, coined by Andrej Karpathy as Software 2.0[143]. Software 2.0 represents a transition away from traditional programming and movestoward automated programming through the use of DL. Figure 1.2 shows how traditionalsoftware engineering has been built on an idea of decomposition through the instructions7of programming. We take a larger task, break it down into smaller tasks and solve thosesmaller tasks. Programs are engineered to represent how we decompose the problem intothese smaller, solvable tasks. A traditional program will accept data as input, then performa set of instructions to manipulate, identify, classify and arrange the data into a meaningfulsolution we call an output. DL turns this paradigm on its head. Now, rather than writinga set of instructions to interact with the data, we merely give the machine the data andour expected output for this data. In turn, the machine writes the program to manipulate,identify, classify and arrange the data. Now the output of the machine is the algorithmfor generating the desired output.Finally, we will highlight an emerging field of SE dedicated to the advancement andaid of developers implementing DL models. We refer to this field as SE4DL (SoftwareEngineering for Deep Learning). Although our work focuses on the realm of DL4SE, theflaws and limitations seen in this realm points to a need for effective software maintenanceand testing for DL architectures. Thus, we hope this dissertation serves as a mediumbetween SE and DL to demonstrate that not only does SE research benefit from thepowerful algorithms of DL, but that DL implementations can benefit from practices of SE.1.4 Contributions and OutlineTo accomplish the first goal set out by this dissertation, we design and conduct a systematicliterature review (SLR) of DL4SE. This SLR serves to quantify and qualify the use of DLapproaches to solve SE tasks. The intellectual contribution of this work is to determinenovel patterns and proceedings pertaining to the components of learning [14]. This analysisshould aid future SE researchers to systematically decompose and apply a DL approachto a SE task. Additionally, we note the challenges referenced by authors relating to thedata preparation and creation of the DL model. Together, these findings reveal a researchroadmap that identifies potential future research directions for DL in SE. Additionally,we provide guidelines for the implementations of these complex models along with some8intuition on which complex architectures could be best suited for different types of SEartifacts. Although not explicitly a part of the DL4SE SLR, we also report a number ofpapers related to SE4DL. We briefly explore how a variety of works have addressed someof the unique problems that DL implementations have. Particularly, this body of workaims to better visualize and understand the inner-workings of a DL algorithm as well asprovide a better testing structure to ensure the model’s accuracy.To accomplish the second goal set out by this dissertation, we design, build and em-pirically evaluate two DL based approaches which aim to aid developers in software main-tenance and software testing. In particular, these approaches automate the detection ofsimilar code fragments based on multiple representations of source code and the construc-tion of semantically and syntactically correct assert statements within test methods. Wedemonstrate the intellectual contributions of these works by:• Mining large, open-source, software repositories for distinct software artifacts.• Effectively synthesizing SE artifacts into feature rich inputs to our DL based ap-proaches.• Packaging a DL framework to train, test and inference our DL model.• Evaluating our DL model to ensure end-to-end automation that aids developers inthe tasks of clone detection and assert statement generation.In Chapter 2 we discuss the background related to DL as it is applied in this dis-sertation. We begin by introducing related works for SLRs and the resources currentlyavailable to SE researchers when applying DL solutions. Additionally, we review relatedwork pertaining to the task of code clone detection and test case generation. This providesthe reader with the necessary information to understand the DL models implemented inchapters 4 and 5.In Chapter 3 we perform a systematic literature review for the SE community, whichunveils a prominent research roadmap of DL4SE. This study involves SE related works from9major international conferences and journals to gather the most representative collectionof papers from all areas of SE. This SLR also addresses the challenges and methods ofimplementing and reporting on the use of DL based solutions in SE. This should serve as aresource for researchers in SE who wish to implement a DL model but are inquisitive aboutwhat problems can be addressed and what is vital when reporting their findings. Lastly,this work reveals the emerging field of SE we call SE4DL. We find that many researchershave identified problems with DL implementations and are using unique SE methods tobetter test and understand these complex models.In Chapter 4 we introduce our novel technique for the identification of similar codesnippets through the use of multiple code representations. This technique involves themining of source code from 10 compiled Java projects as well as 46 Java libraries todetermine if we can identify similar code snippets across different libraries. We thensubject the source code to four different types of abstractions that include identifiers,Abstract Syntax Trees, Control Flow Graphs and Bytecode. We conjectured that eachcode representation can provide a different, yet orthogonal view of the same code fragment,thus enabling a more reliable detection of similarities in code.In Chapter 5 we developed a novel technique for automatically generating assertstatements for test methods. We mined millions of Java test methods, which contain theuse of the Junit assert package. We then identified the particular focal method for eachassert statement within the test method. With this information, we made use of a neuralmachine translation (NMT) model to translate test methods without an assert statementinto semantically and syntactically correct assert statements. Our model, along with theimplementation of a copy mechanism, was able to automatically learn assert statementseven when the model was unable to naturally resolve certain tokens.In Chapter 6 we discuss future directions of research pertaining to the introducedtopic of software 2.0 as well as SE4DL. Both of these areas within software engineeringare heavily motivated by this dissertation and this work will serve as a medium for those10desiring to research in either of these areas. Lastly, we will summarize the contributionsand findings of this dissertation.In addition to the work presented in this dissertation, the author has worked on a varietyof research topics within software engineering over the course of his career as a doctoralstudent including: (i) Bug-fixes [262] (ii) Code-changes [260] (iii) Software Testing [265]and (iv) Software evolution [201].11Chapter 2Background2.1 Deep LearningIn this chapter, we look to expound upon the complex statistical tool of DL and how itapplies to the study of SE. We begin with a brief explanation as to how and why DLwas first applied to unlabeled software artifacts found in online repositories. Once theSE community discovered the representational power of these DL models, their use toautomate SE tasks dramatically increased. Next, we provide information regarding theuse and practicality of SLRs in the field of SE. Lastly, we discuss the implementationdetails and theory behind two DL approaches used in this dissertation. We anticipate thatthe background information provided here will foster a smooth transition into the workpresented in chapters 3, 4 and 5.The progression toward the use of DL stems from the ability to automatically generatefeatures. This removes the substantial manual effort required to generate the hand-craftedfeatures that are required for canonical ML approaches. DL models also distinguish them-selves from other statistically based models in their ability to capture long range depen-dencies within data and learn feature rich representations of software artifacts [20]. DLrepresentations can be viewed as an extension of the perceptatron, an idea that was firstproposed in 1965 [132]. However, the coined phrase of Deep Learning was not introduced12until 1986 [74]. Although the introduction of DL was fairly early, its debut for modelingcomplex software repositories was not presented until 2015 by White et al. [286]. Thiswork stimulated the use of DL to model software artifacts, demonstrating some of theadvantages over information retrieval and natural language models.The introduction of DL into SE brought about a series of works applying these modelsto a variety of tasks. White et al. [284] used DL, and in particular, representation learningvia a recursive autoencoder for the purpose of code clone detection. The authors representeach source code fragment as a stream of identifiers and literal types (from the AST leafnodes). Another work by Lam et al. [158] focuses on bug localization using both DLand Information Retrieval (IR) techniques. Rather than manually defining features, theIR technique, revised Vector Space Models (rVSM), collects these features which capturetextual similarity. Then, a deep neural network learns how to map terms from the bugreports to tokens within the source code. Allamanis et al. [17] proposed an approach forsuggesting meaningful class and method names. To accomplish this, identifiers in codeare assigned a continuous vector, which considers the local context as well as long rangedependencies by a neural log-bilinear context model. Then, identifiers which have similarvectors or embeddings will also appear to have similar contexts. However, in order tocapture the global context of source code tokens, features are engineered which requirescorrect configuration and representation when integrated into the model. Building uponthis previous work, Allamanis et al. [22] used an Attentional Neural Network (ANN)combined with convolution on the input tokens for assigning descriptive method names.Their approach allows for automatic learning of translation invariant features. Lastly, Guet al. [109] used DL to avoid feature engineering in order to learn API usage scenariosgiven a natural language query. The approach encodes a query or annotation of the codeinto a fixed length context vector. This vector helps to decode an API sequence whichshould correlate with the query. Thus, once the model is trained, a natural language querywill result in the correct API usage scenario for the context given in the query. We present13only a few DL approaches to demonstrate the breadth and depth of DL applications inSE.In the present day, there are a variety of different models and DL strategies to addressmany SE tasks. We perform a SLR to gain insight into which specific DL strategies havebeen implemented and provide some guidance on how to implement DL in SE for future SEresearch. Additionally, this dissertation presents two impactful DL approaches to detectcode similarity and generate assert statements within test methods. We will review thesetwo DL frameworks, which make use of recursive autoencoders, recurrent neural networksand neural machine translation.2.2 Systematic Literature ReviewsSystematic literature reviews (SLR) have been a staple for the medical community forthe process of summarizing and organizing critical aspects of medical research [1]. Sincethen, authors Kitchenham and Charters have adopted similar guidelines and procedures forconducting SLRs in the software engineering domain [11]. The purpose of SLRs is threefold:(i) to summarize existing evidence concerning a technology, (ii) to identify gaps or areas forcontinued investigation by researchers and (iii) to provide a framework or background fornew research endeavors. Our work encompasses the gathering and evaluation of empiricalresearch that uses DL in SE.Many SLRs have been performed in the past on a variety of SE topics including effortestimation [240, 288, 267], fault prediction [128, 117, 223], software testing [80, 229] andmany others [30, 98, 51]. These works attempt to summarize the landscape of researchwithin their particular domain at the current time. They map the current approachesand techniques applied to a specific SE task and identify key challenges for researchers tocontinue exploring. However, none of these SLRs investigate the use of a single constructapplied to multiple software engineering tasks. Rather, the software engineering task is14the focal point of the review and a thorough analysis of the approaches and techniques arereported.The study that has the closest relation to our work is by Allamanis et al. [19], whoperformed a SLR on machine learning (ML) and probabilistic models toward program-ming languages. The work the authors survey includes techniques that have approachedthe modeling of source code similar to the modeling of natural language. After the identi-fication of such approaches, the authors begin to summarize how these models have beenapplied to SE tasks such as clone detection and program synthesis. They then highlightthe unique challenges and limitations of these models when applied to the aforementionedSE tasks.In our SLR, we systematically analyze the work done in applying DL to a particularSE task. Similar to Allamanis et al., we look to identify techniques that have been usedas solutions to numerous SE problems. We specifically look for DL based implementationsand do not consider the broader body of work consisting of ML based approaches. SinceDL has quickly risen as a popular tool to model complex software artifacts, we wantedto give researchers a resource that would summarize the body of work already performed,identify important challenges and limitations when applying these models, and distill outfuture research opportunities to apply DL in SE. To the best of our knowledge, no SLRexists for these applications.2.3 Recursive AutoencodersIn this section, we present the architecture for recursive autoencoders which are usedextensively in chapter 4. Recursive auto-encoders are a type of deep believe network(DBN) that creates a representation of features from an input vector [246, 247]. In aDBN, the layers of nodes communicate with previous and subsequent layers, but thereis no communication between nodes of the same layer, which remains true for recursiveautoencoders. To better understand what a recursive autoencoder does, it is important15to recognize the function of this type of DL model. A recursive autoencoder will take asequence of tokens as input, and for many tasks in SE, this results in an input sequenceof source code tokens. These tokens are then abstracted and compressed into a smallerrepresentation, however, this representation can also be decoded to reconstruct the originalinput. An overview of this algorithm can be seen in figure 2.1 where x4 and x5 are combinedinto a n-dimensional feature vector z, which can be decoded into x̂4 and x̂5. The decodingprocess should result in the original input tokens represented by x4 and x5.Figure 2.1: Recursive AutoencoderA recursive autoencoder performs this compression at multiple different levels. Ratherthan encoding the entire sequence at one time, the sequence is compressed over a seriesof time steps. In figure 2.1 this results in the combination of two representations into thesingle encoding z at time step 1. Then, in the next time step, the algorithm combinesthe encodings z and x4 to create a new encoding. This eventually results in a single en-coding that can represent an entire sequence. The purpose of this model is to representdata with a smaller and unique feature vector. Then, feature vectors can be mapped into16an n-dimensional space and clustered based on their distance to other mapped vectors.Training a recursive autoencoder measures the amount of loss there is between the orig-inal input sequence and the decoded output sequence. This loss is minimized over manydifferent input encodings and subsequent decodings via a learning algorithm such as backpropagation. In chapter 4, we will revisit the specific implementation applied to the taskof clone detection.2.4 Recurrent Neural NetworksChapter 5 explores the use of recurrent neural networks for the purpose of learning thegeneration of assert statements in test methods. We look to provide background infor-mation regarding these complex models here. Recurrent neural networks (RNN) have, inmany ways, advanced the way that we represent and embed software artifacts. Unlike therecursive autoencoders, this type of DL model considers the context of previous tokensin order to predict the next token. Additionally, recurrent neural networks are a type ofsupervised learning, meaning that the dataset we use to train these models are labeled.These labels are what we train the model to predict when given a particular input sequence.The type of RNN this work uses is a long short-term memory network (LSTM) which wasdiscovered in 1997 [127]. This network outperformed multiple traditional models for taskssuch as speech and handwriting recognition [236, 171]. Due to its success, these modelswere quickly identified as a tool for advancements in machine translation [252].Unlike NLP models, which were the primary models used to represent source codebefore DL, the context of every previous token is considered. Figure 2.2 found in Whiteet al. [286] describes how a RNN unrolled provides meaningful context to the next tokenpredicted by the model. Here, the token ω(5) is being predicted by the model. The rednode γ is the previous token’s hidden state, which is comprised of the hidden state for everytoken before it. The current hidden state (grey node) computes its non-linear activationas a function of the input token (α) and the previous hidden state (γ). This process17results in the next predicted token. This type of model is very effective in representingsequential tokens that are dependent on the tokens before them. Therefore, these DLmodels are an excellent choice for modeling the software artifact of source code. Theapplicability of RNNs to software artifacts has been applied to many different SE tasks,including software traceability [111], API usage [109], bug-fixes [262], mutation operators[265], malware detection [168], test input generation [146] and code clone detection [284].In this work, we utilize the power of RNNs for the task of neural machine translation.Figure 2.2: RNN Unfolded2.5 Neural Machine TranslationNeural machine translation is the DL framework presented in chapter 5. Our NMT im-plementation is comprised of the RNNs discussed in the previous section. Specifically,NMT is an end-to-end deep learning approach that specializes in modeling sequential datain order to perform a specific task [204]. Therefore, it is commonly used for the tasksinvolving the need for automated translation, text summarization and image captioning.NMT was introduced by Kalchbrenner et al. [140], Sutskever et al. [253], and Cho etal. [64] who defined RNN models for machine translation. The most famous architecture18and the model we incorporate in chapter 5 is by Bahdanau et al. [32] published in 2015.NMT is comprised on an encoder that will encode a sequence of tokens into a fixed-lengthvector and a decoder that will translate that embedding back into translated tokens. Acommon example of this pipeline can be seen when translating Chinese to English. Figure2.3 visualizes this process of translation.Figure 2.3: Neural Machine Translation ModelTo begin the NMT translation process, it is important to understand the componentsthat comprise this complex framework. Let’s begin with the first component known as thebi-directional encoder. The bi-directional encoder is comprised of two RNNs. The firstRNN learns the sequence of tokens in a standard forward pass, while the second RNN learnsthe sequence of tokens in a reverse pass. These models then combine their final states andcreate a single embedding, which is fed to the decoder. The decoder is the next majorcomponent of the NMT framework and is comprised of another RNN. This RNN predictsthe set of output tokens, given the embedding created by the bi-directional encoder. Thedecoder not only considers the input embedding, but also a context vector and any tokenswhich have already been resolved by the decoder. The context vector represents the impactthat the input tokens have on the prediction of the output tokens. In figure 2.3 this is19shown by the lines with varying thickness. The thicker lines represent a greater influencea token in the embedding has on the current token the decoder is attempting to generate.This context vector is created by the attention mechanism incorporated into the model.All components of the model are trained together and are optimized by minimizing thenegative log-likelihood of the targeted terms, while the weights are updated using stochasticgradient decent.2.5.1 Open Vocabulary ProblemNMT suffers from a number of limitations which constrain its use and implementation.One such problem NMT faces is the open vocabulary problem. The NMT model can onlyconsider a finite amount of tokens that will populate the source and target sequences.NMT models generate output from a probability distribution over words. This means thata large number of words to consider significantly slows down the optimization of the model.The oxford English dictionary estimates there to be around 250,000 unique English words,and many of these are uncommon or obsolete [2]. However, when modeling source code,developers are not limited to a finite vocabulary. Therefore, to take advantage of the NMTframework when modeling source code, we must artificially limit the vocabulary used. Anumber of processes to limit the vocabulary have been adopted in a variety of SE works[62, 115, 265, 262].2.5.2 Multiple Predictions with Beam SearchOne limitation of the NMT model is the ability to extract different, possible translationsbased on a single input sequence. Traditionally, NMT models decode the output token ina greedy way, choosing the token with the highest probability at the current time step.However, this may not lead to the correct result in every case. To increase the model’saccuracy and effectiveness, we utilize a strategy named beam search [226]. When usingbeam search, rather than always choosing the token with the highest probability, thismethod keeps the top k hypothesis at each time step for what the decoded token could be.20Figure 2.4: Beam Search ExampleTherefore, rather than the model generating a single sequence of the most probable sourcecode tokens, the model can generate k sequences as potential translations. Formally, letHk be the set of k hypothesis decoded until time step t:Hk = {(α11 . . . α1t ), (α21 . . . α2t ), (αk1 . . . αkt )}At each time step t+1, there are V possible new tokens for each hypothesis. Therefore,we have a total of k · V potential hypotheses.Ct+1 :=k⋃i=1{(αi1, . . . , αit, 1), . . . , (αi1, . . . , αit, V )}The candidate sets are decoded such that we keep k sequences that represent thehighest probability candidates. This process happens at every time step until we reacha special end token that signifies the process has stopped. In our work, we consider ktranslations to potential assert statements. Since beam search is an approximate searchalgorithm, the larger the k the more likely the algorithm will generate a correct translation.However, there is no way to automatically determine the proper translation from the setof k hypothesis and thus puts a greater strain on the developer.21Consider the example shown in figure 2.4 where we show a beam width of k = 3.This example is taken in the context of our NMT model that predicts assert statements.Assuming the input to the NMT model is the test method in the upper-left corner ofthe figure, the beam search mechanism attempts to predict 3 possible assert statementtranslations. Beginning with the START token, beam search starts by generating the top-3 candidates for the first token (i.e., org, assert, (). At time step 2, beam search extendsall three hypotheses, as the top-3 highest probability tokens ( sum, ., 1 ) follows theprevious tokens found in each of the three beams. However, at time step 3, the third beamdoes not generate one of the top-3 highest probability tokens, so the beam is terminated(denoted by red circle). This process continues until the algorithm encounters an end ofsequence token. The resulting sequences can be seen by following the green nodes untilthe yellow terminating node is reached.2.6 Automatically Generating Assert StatementsIn this section, we provide context and motivation for the learned generation of an assertstatement for a particular test method which will be discussed in chapter 5. This worklies within the SE task of software testing, which is a vital component of the software lifecycle. Software testing is widely studied in software engineering and focuses on topics fromautomatic test code generation [29] to automatic generation of test data for software [153].However, software testing is an incredibly complex and systematic process that is oftenarticulated through the use of hierarchical testing levels. Figure 2.5 shows this hierarchywhere the first level is unit testing. This type of testing focuses on individual units ofsoftware with the intention of validating each unit as it is designed.Although unit tests are a fundamental step in the process of software testing, the abilityto test individual components of functionality is a crucial step toward more complex typesof software testing. Assert statements provide the ability to capture the logic and func-tionality of the underlying code base within unit tests. Therefore, generating meaningful22Figure 2.5: Software Testing Hierarchyassert statements is necessary to detect potential faults. Despite the importance of unittests, they can be incredibly time consuming and a laborious process for the developer. Inorder to reduce this burden on developers, automated tools such as Evosuite [94], Randoop[213] and Agitar [5], attempt to automatically generate entire test suites.While several automated tools such as Evosuite [94], Randoop [213] and Agitar [5]have been proposed to automatically generate test methods, these tools embed their ownmethods for synthesizing assert statements. Evosuite, one of the most popular automatedtest generation tools, uses a mutation-based system to generate appropriate assert state-ments. In particular, it introduces mutants into the software and attempts to generateassert statements able to kill these mutants. Evosuite also tries to minimize the numberof asserts while still maximizing the number of killed mutants [94].Randoop is another automated test generation tool that creates assertions with in-telligent guessing. This technique applies feedback-directed random testing by analyzingexecution traces of the statement it creates. Essentially, a list of contracts, or pieces oflogic that the code must follow, are used to guide the generation of assert statements.These contracts are very similar to user defined assert statements. However, the contractsonly provide the logic. The Randoop program creates a syntactically correct assert state-23ment that tests the user’s provided logic pertaining to the test method. Differently fromEvosuite and Randoop, Atlas applies a deep learning-based approach, with the goal ofmimicking the behavior of expert developers when writing assert statements.Recent research has evaluated the ability of state-of-the-art automated techniques fortest generation to capture real faults [239, 26]. They found that the quality of the generatedassert statements are one of the primary reasons that automatically generated tests failedto uncover real faults. Another recent study conducted by Shamshiri et al. [239] highlightsthe importance of high-quality, complex assert statements when detecting real faults. Theauthors compare the abilities of both exception and assert statements as fault findingmechanisms. They note that three automated approaches (Evosuite, Randoop, Agitar)detect more faults through the use of an assert statement, however, insufficiencies in theautomatically generated asserts led to many bugs going undetected [239]. Thus, this workmakes two important conclusions: (i) assert statements are an important component ofautomated test case generation techniques, as they are the main vehicle through whichfaults are detected; (ii) the current quality of assert statements generated by automatedtesting techniques are often not of high enough quality to detect real faults.A complimentary study performed by Almasi et al. [26], tested Evosuite and Randoop’sability to detect 25 real faults in an industrial software system. The study found that Evo-suite was able to detect 56.4% and Randoop was able to detect 38.0% of the faults withinthat system. Of particular note was the author’s qualitative evaluation, which showed thatnearly half of the undetected faults could have been detected with a more appropriate as-sert statement. Additionally, the authors solicited feedback from developers, asking themhow the generated tests could be improved. The general consensus among the respondentswas that automated testing strategies failed to generate meaningful assert statements. Theauthors also presented hand-written test methods to developers for comparison, and theycommented that "the assertions are meaningful and useful unlike the generated ones".These findings demonstrate a clear need for automated techniques that generatemeaningfulassert statements to complement existing test generation approaches.24One of the limitations that tools such as Evosuite, Randoop and Agitar have is thatthey rely on heuristics or "intelligent" randomness in order to generate assert statements.This type of generation does not take into account the learnable components of test andfocal methods, and thus leads to more simplistic asserts. Therefore, we leverage a NMT-based DL technique to generate asserts that can test the complexities contained within thecontext of the test and focal method. This strategy results in an assert which possessesthe ability to accurately evaluate the logic of the focal method, leading to more useful unittests and a higher quality test suite.25Chapter 3Systematic Literature Review inDeep Learning for SoftwareEngineering3.1 IntroductionSoftware engineering (SE) research investigates questions pertaining to the design, devel-opment, maintenance, testing and evaluation of software systems. As software applica-tions pervade a wide range of industries, both open- and closed-source code repositorieshave grown to become unprecedentedly large and complex. This results in an increase ofunstructured, unlabeled, yet important data including requirements, design documents,source code files, test cases and defect reports. Previously, the software engineering com-munity has applied traditional machine learning (ML) techniques to identify interestingpatterns and unique relationships within this data to automate or enhance many tasks typi-cally performed by developers. Unfortunately, the process of implementing ML techniquescan be a tedious exercise in careful feature engineering, wherein researchers experimentwith identifying salient pieces of data that can be leveraged to help solve a given prob-lem or automate a given task. With recent improvements in computational power and26the amount of memory available in modern computer architectures, an advancement totraditional ML approaches has arisen called Deep Learning (DL). Deep Learning repre-sents a fundamental shift in the manner by which machines learn patterns from data byautomatically extracting salient features for a given computational task, as opposed to re-lying upon human intuition. Deep Learning approaches are characterized by architecturescomprised of several layers that perform mathematical transformations on data passingthrough them. These transformations are controlled by sets of learnable parameters thatare adjusted using a variety of learning and optimization algorithms. These computationallayers and parameters form models that can be trained for specific tasks by updating theparameters according to a model’s performance on a labeled set of training data. Giventhe immense amount of unstructured data in software repositories that can serve to un-cover hidden patterns, DL techniques have ushered in advancements across a range oftasks in software engineering research including automatic software repair [263], code sug-gestion [107], defect prediction [276], malware detection [169], feature location [69], amongmany others [187, 271, 179, 285, 289, 114, 258, 182]. We refer to this emerging field ofresearch that leverages deep learning to improve or automate SE tasks as Deep Learningfor Software Engineering (DL4SE).Given the successes of DL approaches on benchmark computer vision (CV) and naturallanguage processing (NLP) tasks, they have become quite popular as a general form ofsoftware development in many industries. That is, developers and engineers are continuingto leverage DL-based systems to solve problems that are difficult to tackle analytically viatraditional algorithm design or programming (self-driving vehicles, facial recognition, etc.).Unfortunately, the very characteristics that make DL attractive and highly effective carrywith them trade-offs related to interpretability and testability. In other words, while thesemodels can often perform well for complex tasks, current challenges exist in interpretingwhy these models make the decisions they do, and in testing them to find where they fallshort. Furthermore, the definition of a “development” cycle for DL models is still emergent,and understanding the best practices and outstanding challenges is essential to ensure27that researchers and practitioners properly allocate resources to supporting DL developersand solving the most pressing challenges. For example, DL models are inherently datadriven, wherein most of the implementation effort is composed of selecting the appropriatemodel and acquiring or creating a large labeled dataset to train the model. However,the selection and implementation of these model architectures can be a complex taskdue to the relatively large number of parameters, which can have significant bearing onmodel performance that must be selected. Additionally, the collection and preparation ofa suitable dataset, and the subsequent representation of this data in a DL framework [164],can comprise a substantial effort in and of itself. Proper data collection, preprocessing andrepresentation can be crucial to the implementation of effective DL models as the level ofabstraction of the input data has the potential to reveal or hide latent features that themodel could learn. The above aspects show the complexities behind the engineering for DL-related projects, and clearly highlight the need for established best practices and intuitiveautomated developer tools that provide developers an effective platform for creating andreusing DL models. We refer to the area of research that aims to improve developmentpractices for DL-based systems as Software Engineering for Deep Learning (SE4DL). Whileresearch in SE has begun to move toward supporting DL systems, particularly in the areaof testing [258, 273, 145], a clear enumeration of current practices and open challenges isneeded to help guide the SE and DL research communities towards the most impactfulproblems.The clear applications of DL to improve and automate SE tasks and the dire needof developer support for creating, testing, and maintaining DL projects more broadlypoints to a clear synergy between ongoing research in SE and DL. However, in order toeffectively chart the most impactful path forward for research at the intersection of thesetwo fields, researchers need a clear map of what has been done, what has been successful,and what can be improved. For instance, researchers would benefit from knowing whichSE areas DL has been applied to and what types of models are optimal when analyzingdifferent types of SE data. There are many DL algorithms, all posing different trade-offs28and benefits within the context of their applications. SE researchers could gain immensebenefit from understanding past successes (and failures) of these applications to furtheradvance automation of new SE tasks. On the other hand, DL researchers and practitionerswould benefit from understanding the open challenges and engineering practices related tothe emerging development cycle of DL models. A better understanding of these existingchallenges, alongside a characterization of the DL development cycle is absolutely essentialto ensure more reliable and practical DL implementations in industry, and more fluidreproducible research in Deep Learning.In an effort to map and guide research at the intersection of DL and SE, we conducted asystematic literature review to identify and systematically enumerate the synergies betweenthe two research fields. We do this through the creation of a detailed taxonomy of pastwork on DL techniques applied to SE tasks, and subsequently identified open challenges andbest practices for applying DL techniques to SE-related data. Additionally, we analyzedthe impacts of these DL based approaches and discuss the difficulties encountered withreproducibility of these studies.We organized our work around four types of Research Questions (RQs) that are fun-damentally centered upon the components of learning. That is, we used the various com-ponents of the machine learning process as enumerated by Yaser Abu-Mostafa’s book,Learning From Data [12], to help ground the creation of our taxonomy and exploration ofthe topic. Our overarching interest was to identify best practices and promising researchdirections for applying DL frameworks to SE contexts, as well as to identify the directionof emerging work that applies principles from SE research to improve the DL develop-ment process. Clarity in these respective areas will give researchers the tools necessary toeffectively apply DL models to SE tasks.To answer the RQs, we focused primarily on the primary studies classified as DL4SE.We created a taxonomy of these studies that highlights important concepts and methodolo-gies applied to the types of software artifacts analyzed, the learning models implementedand the evaluation of these approaches. We discovered that while DL in SE has been29successfully applied to many SE tasks, there are common pitfalls and missing details whenimplementing these complex models. Therefore, in addition to the taxonomies that de-scribe how the components of learning have been addressed, we provide insight into com-monly missed steps and how to avoid them when considering a DL approach. From this,we can provide the SE community with important guidelines for applying DL models thataddress issues such as sampling bias and data snooping, overfitting and underfitting of thelearning models, and the over engineering of DL architectures. Our goal is to empower SEresearchers in their ability to create more robust and reliable DL approaches that addressinteresting SE tasks.With regards to SE4DL, we found initial research that has begun to address the follow-ing DL development tasks of (i) requirements engineering, (ii) design, (iii) implementation,(iv) testing, and (v) maintenance of software. Specifically, we found that recent researchhas focused on the testing and interoperability of these models. This stems from the in-herent abstract nature of DL models and their ability to model target functions whichare intractable for human minds. We discussed these works at length and attempted togive insight into the troublesome limitations when testing DL approaches. However, sincethese approaches do not actually implement a DL based approach, we did not includethem in our DL4SE classifications. Our entire SLR process can be found in figure 3.2,which explains the process for developing the RQs and describes the steps taken in orderto synthesize the necessary information gathered from the primary studies.The remainder of our survey is structured as follows: Section 3.2 discusses the synthesisof the RQs this SLR looks to answer, which are paralleled to the components of learning;Section 3.3 explains our systematic process for extracting and analyzing the primary stud-ies that comprise our taxonomy of DL in SE; Section 3.4.1 begins the generation of ourtaxonomy by analyzing the SE tasks that DL has been applied to; Section 3.4.2 analyzesthe type of data that DL approaches use and how that data is being represented to theDL model; Section 3.4.4 focuses on the learning models that are used to address SE tasks;Section 3.4.5 discusses the impacts and performance of DL approaches in SE; Section 3.4.630begins a discussion around the factors that contribute to the difficulty of reproducibilityof DL applications in SE; Section 3.5 provides future researchers with steps that should betaken and common pitfalls when applying DL in SE; Section 3.6 discusses a new emergingfield in SE, where SE concepts are applied to the field of DL engineering; Section 3.7 fo-cuses on the primary studies which were found by our search string but are not includedin the taxonomy of DL in SE; finally, Section 3.8 summarizes our results and highlightsinteresting, potential directions for the field of DL4SE.3.2 RQ SynthesisThe synthesis and generation of research questions is a quintessential step to any sys-tematic literature review (SLR). When studying how DL has affected SE, we wanted toformulate RQs that would naturally highlight a taxonomy of DL in SE, establish coherentguidelines for applying DL to SE tasks and address common pitfalls when implementingthese complex models. Therefore, we sought to allow the components of learning to guidethe development of our RQs and tailor those questions to specifically analyze DL used inthe context of SE. The components of learning are a ML formalization introduced by YaserAbu-Mostafa [13] as a way to enumerate the conditions for learning. We also wanted toaddress what components of learning the primary studies in SE are missing or are not ade-quately reporting on. Failure to adequately consider or report the components of learningwhen applying a DL approach in SE could lead to difficulty in reproducibility and threatenthe validity of the results.The first component of learning is an unknown target function, which represents therelationship that is desired to be learned. This component is closely linked to the taskthat the learning approach is being applied to. From analysis of the target function to belearned, one can infer the input and output of the model, the type of learning, hypotheticalfeatures to be extracted from the data and potential architectures to be used. To encompassthe essence of this component of learning, our first RQ was:31RQ1: What types of SE tasks have been addressed by DL-based approaches?In understanding what SE tasks have been analyzed, we were able to naturally presenta taxonomy of what tasks have yet to be explored using a DL based approach. We werealso able to infer why certain SE tasks may present unique challenges for DL models andthe target users of these DL approaches, given the SE task they address.Unknown Target Function!: # → %Training Examples#!, %! … #", %"Learning Algorithm(Final Hypothesis)(#) ≈ !(#)Hypothesis Set-Figure 3.1: The Components of LearningA natural next step from analyzing the unknown target function is to explore the databeing presented to the model to learn this unknown target function. Here, we primarilyfocused on studying the input and output training examples and the techniques used in DLapproaches to prepare the data for the model. An understanding of the training examplespresents greater insight into the target function while also giving further intuition aboutwhat the potential features to be extracted are and DL architectures that can be used toextract those features. A taxonomy of the data used, how it was extracted and how itwas preprocessed should encourage new ideas or techniques to better capture the featuresof the data points, resulting in a more accurate model of the target function. Thus, oursecond RQ was:32RQ2: How are software artifacts being extracted, prepared and used in DLbased approaches for SE tasks?Within this RQ we explored multiple facets of the extraction and preprocessing steps bybreaking this RQ down into three sub-question. The first sub-question discussed the dif-ferent types of data that are being used in DL based approaches. Given the plethora ofsoftware artifacts stored in online repositories, it is important to know which of those ar-tifacts are being analyzed and modeled with a DL architecture. Our second sub-questionaddressed how data is being extracted and pre-processed into a format that the DL modelcan appropriately learn from. Analyzing the extraction of the data will present researcherswith potential tools and techniques to mine similar types of data for different DL appli-cations within SE. Additionally, the representation of data is somewhat dependent on theDL architecture and its ability to extract features from that representation, which givesimportance to the discussion of the relationship between DL architectures and the datathey process. Our third sub-question explored what type of exploratory data analysis isconducted to help inform model design and training. It is important to note that DLalgorithms are data hungry, but the quality of the learned hypothesis is a product of thequality of data the model learns from. Therefore, we looked to explore techniques tomitigate problems associated with the data, such as sampling bias and data snooping.Next, we encompassed both the third and fourth components of learning, the learningalgorithm and hypothesis set, into a single research question because we felt that theywere highly related to one another. The learning algorithm is the process of navigating thehypothesis set for the best fit model to the data. These are primarily mathematical modelsthat use a probability distribution over the input to appropriately approximate the optimalhypothesis from the hypothesis set. The hypothesis set is a set of all hypotheses, based onthe learning algorithm, that the input can be mapped to. This set changes because it isa function of the possible outputs given the input space, and is dependent on the learning33algorithm’s ability to model those possible outputs. Together the learning algorithm withthe hypothesis set is referred to as the learning model, thus, our third RQ asked:RQ3: What deep learning models are used to support SE tasks?We also wanted to make an important distinction within the learning model to address thevariety of architectures used and the optimizations applied to those architectures in orderto model the appropriate hypothesis set. We broke down RQ3 into three sub-questions,which addressed the aforementioned attributes of the learning model separately. Firstly, weasked what types of model architectures are used to perform automated feature engineeringof the data within various SE tasks. Here, we looked at the variations of architectures torelate them to the data they analyzed and the task they addressed. We also used this RQto present how the type of architecture chosen to model a particular target function relatesto the types of features that are being extracted from the data. Secondly, we looked intothe different learning algorithms and training processes that are used in order to optimizethe models. This is where we addressed the variety of different learning algorithms whoseresponsibility is to properly capture the hypothesis set for the given input space. Thedifferent optimization algorithms and training processes used to tune the weights of themodel are an important step for finding the target hypothesis that best represents thedata. Lastly, we analyzed the methods used to combat overfitting and underfitting. Thisquestion addressed one of the major concerns with using DL algorithms and their tendencyto overfit or underfit the data they model. Within SE, we wanted to know what techniquesthe authors are implementing to combat these well understood limitations.Our fourth RQ addressed the component of learning known as the final hypothesis.The final hypothesis is the target function learned by the model that is used to predictthe target of previously unseen data points. Measuring the effectiveness of DL modelscan encourage the use of DL in areas of SE that DL has not yet been applied. We alsofelt that we could provide an indication toward the advantages of certain data formatting,34DL architectures or techniques that have been successful for certain SE tasks in the past.Thus, our fourth RQ was:RQ4: How well do DL tasks perform in supporting various SE tasks?The purpose of this RQ is to analyze the effectiveness of DL applied to SE tasks, how-ever, this can be a difficult and complex. We attempted to answer this overarching RQby first addressing the “baseline” techniques that are used to evaluate new DL models anddiscovering what metrics are used in those comparisons. An evaluation that contains acomparison with a "baseline" approach, or even non-learning based solution, is importantfor determining the increased effectiveness of applying a new DL framework. Additionally,the metrics used to quantify the comparison between DL approaches should be knownso that future implementations of DL in SE know how to quantify their increased effec-tiveness. Our second sub-question explored how DL based approaches are impacting theautomatization of SE tasks and in which way these models promote generalizability todifferent sets of data. Generalizability of DL approaches in SE is vital for the usability ofthese models. If the state of the art DL approach is only applicable within a narrowly de-fined set of circumstances, then there may still be room for improvement on this solution.Lastly, we wanted to explore common mistakes made by overengineering the DL model andanalyzing the trade-off of model complexity with accuracy. We look at DL architecturesthrough the lens of Occam’s Razor principal, which states that the least complex modelthat is able to learn the target function is the one that should be implemented [224].Our last RQ encompassed all of the components of learning by asking a discussionquestion related to the reproducibility of DL approaches in SE. SE as a field has made asignificant effort toward publishing works that are reproducible. However, DL models canbe very complex and we wanted to determine if there were important details left out ofthe primary studies we analyzed. Therefore, we asked the question:RQ5: What common factors contribute to the difficulty when reproducing DLstudies in SE?35Our goal was to identify common components of implementing a DL based approach whichare absent from the primary studies. Particularly, we addressed two important attributes:what primary studies are not reproducible and why, as well as what primary studies are notreplicable and why. Reproducibility is the ability to take the exact same model with theexact same dataset from the primary study and produce the same results. Replicability is tofollow the methodology described by the primary study such that a similar implementationcan be generated potentially on a different dataset. This RQ helps the SE community tounderstand what factors are being insufficiently described or absent all together within theDL approach, leading to difficulty in reproducing or replicating the primary study.Lastly, we looked to generate a set of guidelines, not only for the implementation ofDL in SE, but also how to report the model in a way that values reproducibility. Theseguidelines start with the identification of the SE task to be studied and provide a step bystep process through evaluating the new DL approach. Due to the high variability of DLapproaches and the SE tasks they are applied to, we only provide generalizable steps thatshould be performed for future implementations of DL in SE. In addition, we providedcheckpoints throughout this process that address common pitfalls or mistakes that futureSE researchers can avoid when implementing these complex models. Our hope is thatadhering to these guidelines will lead to future DL approaches in SE with an increasedamount of clarity and reproducibility.3.3 MethodologyIn this paper we performed a systematic literature review of deep learning in softwareengineering, which discusses the open issues involving the use of these complex models toaddress software engineering tasks. We applied a systematic methodology to our litera-ture review in order to uphold the integrity of our analysis and provide a reproducibilityaspect to this work. Within the field of SE, SLRs have become a standardized practice tocommunicate the past and present state of a particular area within SE. These standards36were developed by Kitchenham’s work, Guidelines for Performing Systematic LiteratureReviews in Software Engineering [149], which contains the guidelines and strategies usedin this review.The full breakdown of our methodology in conducting the SLR can be seen in figure3.2. As is described in Kitchenham’s guidelines, we synthesized research questions (RQs)before beginning the searching process, which helped to naturally guide the SLR procedureand focused the search only to primary studies that helped answer these RQs. From here,we performed the following steps when conducting our review:• Searched for primary studies• Filtered studies through our inclusion criteria• Performed snowballing and manual additions to the body of work• Applied our exclusion criteria and quality analysis of the remaining studies• Extracted relevant data and analyzed that data from the final set of studies• Synthesized and created a taxonomy of DL in SE3.3.1 Searching for Primary StudiesThe next step in the methodology was to begin searching for primary studies that would aidin the ability to properly address the synthesized RQs. We first began by identifying venueswe would like to search. We chose venues which have a high standard for peer reviewedresearch, as well as a high influence over the field of SE. Therefore, we considered researchfrom the SE conferences of ICSE, ASE, FSE, ICSME, ICST, MSR, and ISSTA. In addition,we considered research from the SE journals of TSE, TOSEM, EMSE, and IEEE Software.Lastly, we considered the top conferences dedicated to machine learning and deep learningimplementations. We did this in order to capture papers focused on the creation of a DLapproach, that also apply that approach to a SE task. The conferences we considered in the37Apply Search Methodology MSRISSTAICSTICSEEMSEIEEE SoftwareICSMEASETOSEMTSEICLRNIPSFSEICMLApply Inclusion CriteriaRQ Synthesis1,184 Papers127 PapersSnowballing145 PapersManual Additions148 PapersApply Exclusion CriteriaApply Quality, Manual Analysis 84 PapersData Extraction Taxonomy SynthesisFigure 3.2: SLR MethodologyML/DL field were ICML, ICRL, and NIPS. Once we established the venues to search, wedeveloped a search string to query each database for relevant research. The developmentof the search string was determined based on terms that relate to the RQs we wantedto answer. We empirically evaluated multiple search strings using a variety of derivedterms and found the string ("Deep" OR "Learning" OR "Neural") to return the mostrelevant primary studies. At this point, we were ready to begin querying the respectiveconferences and journals for primary studies. Most of the venues we searched were foundin four online databases: arXiv, IEEE Xplore, ACM Digital Library and SpringerLink.The fourth database we searched was Google Scholar, which was mainly used to access theprimary studies contained within the ML/DL conferences.The four major databases we considered provided a mechanism for advanced searchingthat can be used in conjunction with the search string. Although there are nuances amongsteach database, the method for gathering studies was consistent. The search string providedan initial filtering of the primary studies, then additional features of the advanced searchallowed us to add discriminatory criteria. These additional features are how we limited oursearch results by year and venue. In addition to the three major databases, we searched38Google Scholar for papers within ICML, ICRL and NeurIPS through the Publish or Perishsoftware [266]. To search these three conferences, we had to augment our search string withSE terms that would only return papers addressing a SE task. We gathered these termsfrom ICSE (International Conference on Software Engineering), since they are used astopics for technical track paper submissions. We iterated through each term and appendedit to our search string. The results of that search were manually inspected for relevantpapers to SE. After searching the databases with the specified search string, our initialresults yielded 1,184 relevant studies.3.3.2 Filtering Through Inclusion CriteriaIn this step of the SLR, we defined the inclusion criteria that determined which primarystudies would be included in our taxonomy. To an extent, part of our inclusion criteria wasalready used to extract the primary studies. The year and the venue were used in advancedsearches to filter out extraneous research that our search string returned. However, afterthe search phase concluded, we filtered our results based on a more strict set of inclusioncriteria. This ensured that the primary studies would be a useful addition to the taxonomyand would contain the necessary information to answer the RQs. A full list of the inclusionand exclusion criteria can be viewed in table 3.1.Table 3.1: SLR Inclusion and Exclusion CriteriaInclusion Criteria Exclusion CriteriaStudy is published in the year range 2009-2019. Study was published after January 1, 2009 or beforeApril 1, 2019.Study clearly defines a SE task that is addressed witha DL based approach. Study does not solve a SE task.Study was published in the predefined venues. Study is outside the scope of software engineering oris published in an unaccepted venue.Paper must identify and address the implementationof the DL framework.Paper does not fully implement, evaluate or addressthe implementation of a DL approach.39The primary studies gathered through the use of the search string, the snowballingmethod, or through manual addition were subjected to the same criteria for inclusion.Three of the primary authors divided the works and labeled them for inclusion based onreading the abstract and methodology of the paper. A second author then reviewed eachclassification. If the classifications between the two authors were in disagreement, thenall authors reviewed and discussed this work until a unanimous consensus was achieved.At the end of this phase of the SLR, we were able to include 127 primary studies forconsideration.3.3.3 Snowballing and Manual AdditionsOur next step in the process was snowballing and the inclusion of manually added papers.After extracting primary studies from the specified venues and subjecting those studiesto our inclusion criteria, we performed snowballing on the resulting studies. Snowballinghelped to ensure we were gathering as much related research as possible. Our snowballingprocess looked at every reference within the studies passing our inclusion criteria anddetermined if any of those references also passed our inclusion criteria. Lastly, using ourexpert knowledge of the field, we manually added any related works that were missedby our systematic process. We performed this manual addition for completeness of thesurvey and made note of which studies were manually added in our taxonomy. Fromperforming snowballing, we were able to add 18 new pieces of research to include in ourSLR. In addition, we manually added three relevant pieces of work that were not capturedby either our search string or snowballing. This resulted in 148 primary studies.3.3.4 Exclusion Criteria and Quality AnalysisAt this stage, we applied our exclusion criteria to determine if there were any primarystudies which were inadequate to be a part of our taxonomies. This involved a significantmanual analysis to closely analyze how each study incorporated the use of DL when ana-lyzing a SE task. Particularly, we found many studies that only used a DL based approach40as a means for evaluation. We also found instances where DL was discussed as an ideaor part of the future work of a study. We therefore excluded these works to ensure thatevery paper we analyzed implemented and evaluated a full DL approach to address a SEtask. This resulted in 84 studies, all of which were included in the data extraction andtaxonomy synthesis phases. In figure 3.3 we show the breakdown of venues that our final84 studies came from.ASE FSE ICLR ICML ICSE ICSME MSR NIPS TSE EMSE ISSTA ArXivFigure 3.3: Venue distribution of DL4SE3.3.5 Data Extraction and AnalysisOur next step in the SLR methodology was to extract the necessary data from the primarystudies. This step involved the development of an extraction form where the authorsagreed about specific information to extract from reading the papers. We would then usethis information to create complete taxonomies of DL in SE. Each author was tasked withextracting data from a subset of the 84 primary studies classified as DL4SE. However,41the results of the extraction phase were confirmed by at least two additional authors toensure that all important details were gathered and that papers were properly representedby the taxonomy. We then performed data analysis of our extracted data to determinethe types of relationships between features. We explain our process of data analysis in thenext section.3.3.5.1 Data AnalysisOur final task was to analyze the data extracted from the SLR process. We performedExploratory Data Analysis (EDA) in conjunction with Confirmatory Data Analysis (CDA),in order to obtain descriptive statistical attributes of our dataset. The mined facts andassociations allowed us to ascertain and support the conclusions of this SLR.The Systematic Literature Review of DL4SE required formal statistical modeling torefine the answers for the proposed research questions and to help formulate new hypothe-ses to be addressed in the future. Hence, we introduced DL4SE-DA, a set of statisticalprocesses and data mining pipelines that uncovered hidden relationships among the liter-ature reporting a DL based approach in SE. Such hidden relationships were collected andanalyzed to illustrate the state-of-the-art DL techniques employed in the SE context.Our DL4SE-DA is a simplified version of the classical Knowledge Discovery in Databases,or KDD [92]. The KDD process extracts knowledge from a DL4SE structured database.This structured database was the product of multiple iterations of data gathering andcollection from the inspected literature. The KDD involves five stages:1. Selection. This stage was led by the taxonomy process explained in the beginningof this section. After collecting all the papers and creating the taxonomies, weorganized the data into 35 features or attributes that you find in the repository. Infact, we manually engineered features from the DL4SE papers. Some of the featureswe considered were: venue, year published, type of paper, metrics, data-scale, typeof tuning, learning algorithm, SE data, and so on.422. Preprocessing. We applied a preprocessing technique that transformed the featuresinto the correct type (nominal), removed outliers (papers that do not belong to theDL4SE), and re-inspected the papers to extract missing information produced by thenormalization process. For instance, we normalized the feature “metrics” into “MRR”,“ROC or AUC”, “BLEU Score”, “Accuracy”, “Precision”, “Recall”, “F1 Measure”, and“Other Metrics”. “Other Metrics” refers to unconventional metrics found during theextraction. Similarly, the same normalization was applied to other features like“SE Data” and “Reproducibility Types”. This separation into more detailed classescontributes to a better understanding and classification of the paper by the datamining tasks or methods.3. Transformation. In this stage, we omitted the use of any data transformationmethod. We performed a Principal Component Analysis (PCA) to reduce 35 featuresinto 2 components for visualization purposes.4. Data Mining In this stage, we used two distinct data mining tasks, CorrelationAnalysis and Association Rule Learning. We decided that the goal of the KDDprocess should be oriented to uncover hidden relationships on the extracted featuresCorrelations and Association Rules. A clear explanation is provided in the subsection“Data Mining Tasks for the SLR of DL4SE”.5. Interpretation/EvaluationWe used the Knowledge Discover to automatically findpatterns in our papers that resemble “actionable knowledge”. This actionable knowl-edge was generated by conducting a reasoning process on the data mining outcomes.This reasoning process produced an argument support analysis formally found in ouronline repository [280].We used RapidMiner [10] as our software tool to conduct the data analysis and theprocedures and pipelines for using this tool are published in our repository. Before carryingout any mining tasks, we decided to address a basic descriptive statistics procedure for each43feature. These statistics exhibit basic relative frequencies, counts, and quality metrics. Werevised 4 quality metrics: ID-ness, Stability, Missing, and Text-ness. ID-ness measures towhat extent a feature resembles an ID, where as the “title” is the only feature with a highID-ness value. Stability measures to what extent a feature is constant (or has the samevalue). Missing refers to the number of missing values. Finally, Text-ness measures towhat extent a feature contains free-text. The results for each feature could be navigatedonce the SRL process was ran in RapidMiner. We employed two well-established datamining tasks to better understand our data: Correlations and Association Rule Learning.3.3.5.2 CorrelationsDue to the nominal nature of the SLR features, we were unable to perform classic statisti-cal correlations on our data (i.e. Pearson’s correlation). However, we adapted an operatorbased on attributes’ information dependencies. This operator is known as “mutual infor-mation” [190]. Similar to covariance or Pearson’s correlation, we were able to representthe outcomes of the operator in a confusion matrix.The mutual information measures to what extent one feature knows about anotherone. High mutual information values represent less uncertainty; therefore, we built argu-ments such as “the deep learning architecture used on a paper is less uncertain (or morepredictable) given a particular SE task” or “the reported architectures within the papersare mutually dependent with the Software Engineering task”.The difference between correlation and association rules depends on the granularitylevel of the data (e.g., paper, feature, or class). The correlation procedure was performedat the feature level, while the association rule learning was performed at the class orcategory level.3.3.5.3 Association Rule LearningFor generating the association rules, we employed the classic FP-Growth algorithm (FPstands for Frequent Pattern). The FP-Growth computed frequently co-occurring items44in our transactional dataset (see DL4SE-Dataset.csv in [280]) with a minimum support of0.95. These items comprise each class per feature. For instance, the feature “Loss Function”exhibited a set of classes (or items) such as “NCE”, “Max Log Likelihood”, “Cross-Entropy”,“MSE”, “Hinge Loss”, “N/A-Loss”, and so on. The main premise of the FP-Growth can besummarized as: if an item set is frequent (i.e. MSE, RNN), then all of its item subsets arealso frequent (i.e. MSE and RNN).Once the FP-Growth generated the item sets (e.g. MSE, Hinge Loss), the algorithmstarted to check the item sets’ support in continuous scans of the database (DB). Suchsupport measures the occurances of the item set in the database. The FP-Growth scans theDB, which brings about a FP-tree data structure. We recursively mined the FP-tree datastructure to extract frequent item sets [118]. Nonetheless, the association rule learningrequires more than the FP-tree extraction.An association rule serves as an if-then statement (or premise-conclusion) based onthe frequent item set pattern. Let’s observe the following rule mined from our dataset:“Given that an author used Supervised Learning, we can conclude that their approach isirreproducible with a support of 0.7 and a confidence of 0.8”. We observe the associationrule has an antecedent (i.e., the item set Supervised Learning) and a consequent (i.e.,theitem set Irreproducible). These relationships are mined from item sets that usually havea high support and confidence. The confidence is a measure of the number of times thatpremise-conclusion statement is found to be true. We tuned both parameters to be greaterthan 0.7.We refute the idea that association rule learning avoids spurious correlations. Regard-less, we organized the rules into an interconnected net of premises/conclusions to find ex-planations around techniques and methodologies reported on the papers. Any non-logicalrule is disregarded as well as rules that possess a lower support and confidence.The results of our exploratory data analysis and source of the information provided inthis SLR can be found in our online repository [280].453.3.6 Taxonomy SynthesisIn the next section, we discuss the resulting taxonomies created from the primary studiesfound through our systematic approach. We use these taxonomies in order to answerthe RQs presented in the beginning of this work. These RQs naturally define a largertaxonomy that researchers can use to determine what types of SE tasks can be betterstudied using certain DL based approaches, as well as look for future applications of DLto model complex software artifacts. We oriented the results of our RQs to provide anunderstanding about the process of applying a DL based approach in SE.3.4 Results3.4.1 RQ1: What types of SE tasks have been addressed by DL-basedapproaches?In this RQ, we explored what types of SE tasks DL based approaches are applied to.DL models are excellent at identifying an appropriate target hypothesis given complex,hierarchical, abstract representations of data points. This makes them useful models toapply to SE tasks, as these models can automatically extract useful, hierarchical featureswithin SE data that developers or researchers could not find manually. These features canbe used to establish relationships between the facets of input data and target variables.Those relationships can then be used in the prediction of a target, given an input datapoint that was previously unseen by the model.Out of the 84 papers we analyzed for this SLR, we found 23 separate SE tasks wherea DL based approach had been applied. Figure 3.4 gives a visual breakdown of how manySE tasks we found within these 84 primary studies across a 10 year period. Unsurprisingly,there was very little work done between the years of 2009 and 2014. However, even afterthe DL breakout paper of AlexNet by Krizhevsky et al. [156] in 2012 showing the useful-ness of DL in image classification, no work on SE tasks took place until 3 years later in460 5 10 15 20 25 30 35 40 45Source CodeNatural Language DescriptionsBug ReportLabeled UI ComponentsRepository MetadataLDA TopicsTime MeasurementsPPBE TasksImages or VideoExecution TracesContextual Flow GraphsProgram Input-Output ExamplesKarel DSL GridAbstract Intents in Android ApplicationsSymbolic EquationsCertificatesCode ComprehensionSource Code GenerationSource Code Retrieval & TraceabilitySource Code SummarizationBig-Fixing ProcessCode SmellsSoftware TestingNon Code Related Software ArtifactsClone DetectionSoftware Energy MetricsProgram SynthesisImage To Structured RepresentationSoftware SecurityProgram RepairSoftware Reliability / Defect PredictionFeature LocationDeveloper Forum AnalysisProgram TranslationSoftware CategorizationFigure 3.4: SE Tasks Per Year2015. The first SE tasks to use a DL technique was that of Code Comprehension, SourceCode Generation, Source Code Retrieval & Traceability, Bug-Fixing Process, and FeatureLocation. Each of these tasks uses source code as their primary form of data, which is dueto its large abundance in online repositories. This is a result of the Software Engineeringcommunity’s shift toward becoming more open sourced, allowing for access to millions ofsoftware projects through popular hubs such as GitHub, Gitlab, and Bitbucket. Access toa large amount of data and a well understood task is important for DL4SE, since in orderfor DL to have an effective application two main components are needed: i) a plethoraof data to support the training of multi-parameter models capable of extracting complexrepresentations and ii) a task that can be addressed with some type of predictable target.One of the major benefits of DL implementations is the ability for automatic feature ex-traction. However, this requires the ability to find data that has a distinct relationshipwith the predicted target variable.It wasn’t until 2017 that DL was being used extensively in solving SE tasks as shownin Figure 3.4, with a large increase in the number of papers, more than doubling over47the previous 2 years. During this period, there was also a more diverse set of SE tasksattempting to be solved, such as Code Smells, Software Security, and Software Categoriza-tion. However, there are three main SE tasks that have remained active across the years:Code Comprehension, Source Code Retrieval & Traceability, and Program Synthesis1. Themost popular of the three being Program Synthesis, composing a total of 20 papers outof the 84 we collected. We suspect that a variety of reasons contribute to the multipleapplications of DL in program synthesis. First and foremost, is that the accessibility todata is more prevalent. Program synthesis is trained using a set of input-output examples.This makes for high quality training data, since one can train the DL model to generate aprogram, given the set of formal specifications. The second largest reason is having a clearand established relationship between the training examples and the target program. Theserelationships are what deep learning methods thrive on as they can adequately model thisrelationship without the need for hand-crafted features.We display the full taxonomy in Table 3.2, which associates the cited primary studypaired with its respective SE task. Interestingly, we encountered a few tasks that maynot initially seem as though they are SE related. For example, tasks such as image tostructured representation, software security and software energy metrics do not initiallyappear directly related to SE. However, upon further analysis we found these tasks to behighly related to SE and therefore were included in our taxonomy.When performing our exploratory data analysis, we found the the SE task was the mostinformative feature we extracted, meaning that it gave us the greatest insight into whatother features may have been associated. In particular, we found that SE tasks had strongcorrelations to data preprocessing (1.80), the loss function used (1.45) and the architectureemployed (1.40). Therefore, we can predict that there are DL framework components thatare better suited to address specific SE tasks. Throughout the remaining RQs, we look to1We recognize that Program Synthesis does not appear in Y19, however since we did not collect papersfor all of Y19, we are confident that Program Synthesis papers would appear.48Table 3.2: SE Task TaxonomySE Task PapersCode Comprehension [167, 43, 219, 122, 161, 18, 21, 162]Souce Code Generation [142, 123, 282, 287, 110, 54]Source Code Retrieval & Traceability [58, 112, 108, 76, 159, 52]Source Code Summarization [58, 271, 23]Big-Fixing Process [166, 202, 48, 76, 159]Code Smells [179]Mobile Testing [183]Traditional Software Testing [102, 71, 242]Non Code Related Software Artifacts [136, 184, 237, 65, 67, 66]Clone Detection [170, 285, 232, 99, 176, 261]Software Energy Metrics [230]Program Synthesis [100, 50, 294, 78, 250, 203, 227,178, 36, 79, 49, 270, 61, 125, 28,300, 241, 88, 89, 173]Image To Structured Representation [75, 55, 200]Software Security [119, 297, 53, 99, 120, 72]Program Repair [46, 274, 120, 264, 181]Software Reliability / Defect Prediction [116, 277, 281]Feature Location [70]Developer Forum Analysis [56, 289, 175, 191]Program Translation [60]Software Categorization [27]Code Location Within Media [211]Developer Intention Mining [129]expand upon the associations we find to better assist software engineers in choosing themost appropriate DL components to build their approach.Although the applications of DL based approaches to SE related tasks is apparent, thereare many research areas of interest in the SE community as shown in ICSE’20’s topics ofinterest2 that DL has not been used for. Yet, many of these topics have no applicabilityfor a DL based solution, such as Cloud computing, human and social aspects of softwareengineering, parallelization, distributed and concurrent systems, etc. These topics areeither out of scope for DL based automation or require more stringent methods that are2https://conf.researchr.org/track/icse-2020/icse-2020-papers#Call-for-Papers49not stochastic. Still, some of the interesting topics that we believe can see benefit from DLapproaches have yet to be explored by the research community or are underrepresented.Topics of this unexplored nature include refactoring and program analysis, whereas topicswhich are underrepresented include software testing, both traditional systems and mobile,software documentation, feature location, and defect prediction. Some possible reasonswhy certain SE tasks have yet to gain traction in using DL could be due to the following:• There is a lack of available, clean data in order to support such DL techniques,• The problem itself is not well defined, such that a DL model would struggle to beeffectively trained and optimized• No current architectures are adequately fit to be used on the available data.3.4.2 RQ2: How are software artifacts being extracted, prepared andused in DL based approaches for SE tasks?In this research question, we analyzed the type of data that is modeled by the DL ap-proaches in SE. We wanted to address a few aspects of how SE data is used, how the datais extracted or preprocessed, the scale of the data being modeled and the type of learningbased on the data available. All four of these points are a crucial step in understandinghow the approach is able to model specific software artifacts. These points also open upa discussion about future research toward the improvement of data manipulation and pre-processing steps to increase accuracy of the DL approaches. In many cases, data must becleaned or filtered in such a way that limits the ability for the approach to model a desiredrelationship. Since DL models are frequently seen as a black-box in terms of explainabilityand interoperability, it is important that the relationships between input and output areaccurately reflected within the dataset.503.4.2.1 What types of data are being used?When analyzing the types of data that were being used in DL based approaches, weprovided a high level classification, along with some intuition, as to why some types ofdata were used for particular tasks. We found that overwhelmingly the most commontype of data being used is source code at different granularities. Within this SLR, wefound source code being used at the binary level, code snippet level, method level, classlevel and project level. The high volume of approaches that learn from source code doso for a number of reasons. The first is that source code is plentiful and can be foundfrom a number of different online repositories. This availability aids in the fact that DLmodels are extremely data hungry in order to learn a representative target function. Thesecond reason is that a majority of SE tasks revolve around the generation, understanding,debugging and documentation of source code, which makes the desire to automate thesetasks for developers extremely high.0 5 10 15 20 25 30 35 40 45Source CodeNatural Language DescriptionsBug ReportLabeled UI ComponentsRepository MetadataLDA TopicsTime MeasurementsPPBE TasksImages or VideoExecution TracesContextual Flow GraphsProgram Input-Output ExamplesKarel DSL GridAbstract Intents in Android ApplicationsSymbolic EquationsCertificatesCode ComprehensionSource Code GenerationSource Code Retrieval & TraceabilitySource Code SummarizationBig-Fixing ProcessCode SmellsSoftware TestingNon Code Related Software ArtifactsClone DetectionSoftware Energy MetricsProgram SynthesisImage To Structured RepresentationSoftware SecurityProgram RepairSoftware Reliability / Defect PredictionFeature LocationDeveloper Forum AnalysisProgram TranslationSoftware CategorizationFigure 3.5: Data Types With SE Tasks51We find that out of the 111 uses of data in DL4SE papers, 49 of them are attributedto source code. The reason that there are 111 uses of data, which is larger than the 84papers we analyze, is because many primary studies use multiple different data types inconjunction to learn from. Although source code is the primary type of data that DLmodels attempt to learn from, we found that the type of data used is heavily dependenton the SE task. Thus, the SE tasks that focus on the comprehension, generation, summa-rization and testing of source code will frequently use some granularity of source code tocomprise their dataset. However, there are many SE tasks that address problems wheresource code may not be the most representative type of data to learn from. As an example,the SE tasks of program synthesis primarily uses input and output examples to comprisetheir dataset. This type of data incorporates attributes, which more strongly correlatesto the relationship the engineer desires the model to learn. This pattern continues for SEtasks that can learn from textual data, software artifacts and repository metadata. Ourexploratory data analysis also highlighted these points. Our analysis demonstrated thatthe main data employed in DL4SE papers are input-output (I/O) examples, source code,natural language, repository metadata and visual data. In fact, these categories of datatypes comprised 78.57% of the distribution we found in this SLR. In conjunction with thisfinding, execution traces and bug reports represent ∼ 10% of the distribution of data. Theremaining ∼ 12% is comprised of a variety of different data types. The full breakdown ofdata types can be viewed in figure 3.5.Although we find a variety of different data types used, the data must be accessiblein large quantities in order to extract and learn the relevant target hypothesis. Withan increase in the number of online repositories, the ability to access these repositorieshas increased over time. This can partially explain the dramatic increase in DL papersaddressing SE tasks, which was a trend discussed in 3.4.1. Interestingly, the growth ofonline repositories does not only increase the amount of accessible source code but alsoother types of software artifacts. For example, analyzing SE tasks such as software security,program translation, etc. have only recently been addressed, in part due to the increasing52amount of accessible online repository data/metadata. We hope to see this trend continuewith new benchmarks and datasets that will be released with future publications.When exploring online repositories, one attribute of the data that can be problematicfor the use of some DL approaches and should be considered is that the data is frequentlyfound unlabeled, meaning that there is no inherent target to associate with this data. Forunsupervised learning techniques this is not problematic, however, for supervised learningtechniques this type of data is not usable without first establishing a target label for thedata. The use of unlabeled data from online software repositories was foreshadowed inprevious work [282]. Our findings corroborate this study, as many labeled and unlabeleddatasets are used within DL based approaches to address SE tasks. The use of labeledversus unlabeled data can be ascertained based on the type of DL model chosen. However,we find that researchers mined unlabeled data and label it themselves in order to use aparticular DL architecture. With the increase in the number of online repositories and thevariety of unlabeled data within those repositories, we expected the number of DL basedapproaches analyzing software artifacts to increase.Throughout our study, we noticed a few SE artifacts that have not yet been analyzedthrough a DL based approach. Specifically, we noticed that software requirements, soft-ware dependencies and software licensing have not been mined or studied using a DLarchitecture. We emplore the SE community to continue to mine and analyze different SEartifacts for the automatization of SE tasks. The continuation of this process will lead tomore datasets and benchmarks for future DL model comparisons.3.4.2.2 How is data being extracted and pre-processed into a format that theDL model can appropriately learn from?In 3.4.2.1, we analyzed what types of data were being used to model complex relationshipsbetween software artifacts and the target output function as it relates to a specific SEtask. In this RQ, we looked for the mechanism behind the extraction and preprocessing ofthis data. DL models extract complex, hierarchical features based on the input data. This53data is subjected to a number of hidden layers and hidden units, which are responsible forthe feature engineering process. The learning associated with these models is the processof tuning the weight matrices between the layers. This tuning enables the extraction of thenecessary features to appropriately model the target hypothesis [104]. In many instances,the preprocessing of data can be handled by tools designed to format input to DL models.These tools are necessary because DL models cannot accept raw data mined from theonline repositories. Rather, the data is subjected to a preparation and formatting processbefore given to the DL model. This is an extremely important step for those applyingDL to SE, since the process can dramatically affect the performance of the model and theresulting target hypothesis. For example, some primary studies represent source code in anabstracted format. This abstraction will inherently remove some of the complex featureswithin the dataset. This process has both pros and cons. Examples of pros are thatabstraction addresses some of the learning problems, such as the open vocabulary problemand generalizability, that can arise when applying a DL model to a SE task. However, alimitation of the abstraction process is that it removes some complex, hierarchical featuresfrom the dataset, which can limit the model’s ability to accurately predict the target insome cases.The pros and cons of any preprocessing step must be carefully considered before beingimplemented. The authors should understand what inherent limitations the preprocess-ing step can impose on the model. Just as importantly, the preprocessing process mustbe documented and explained in order for a study that uses this technique to be repli-cated or used by developers. If one were to use a DL model without also replicating thepreprocessing steps, the results of the experiment would be inconclusive.In light of the importance of preprocessing in DL based approaches, we looked togenerate a taxonomy of preprocessing and formatting techniques. We also looked to analyzethe relationships between the SE tasks, types of data and the preprocessing steps taken.Unsurprisingly, the preprocessing and formatting techniques of data can be highly specific.This is in part due to the features that the authors look to ascertain from their dataset.54The desired features should help drive the decision on what preprocessing and formattingtechniques to use. A preprocessing pipeline that removes or overshadows the desiredfeatures from the dataset will ultimately lead to a poor performing model. A breakdownof the preprocessing and formatting techniques used in the primary studies we analyzedcan be found in figure 3.6.05101520253035Byte Pair EncodingTokenizationNeural EmbeddingExecution Trace VectorizationOne Hot Encoding VecoterTransformation of Execution FeaturesSequence Splitting of PDF Objects into a…LLVM to Contextual Flow GraphsExtracting Abstract Intents into a VectorLookup TablesVectorized Code MetricsIssue Report FeaturesPrefix Tree AcceptorSource Tree and Target Tree to Binary…IDA Pro Feature ExtractionGenerate Input - Output VectorsDirected GraphChange Sequence Semantic InformationFrame Extraction with PyTubeCode Comprehension Souce Code GenerationSource Code Retrieval & Traceability Source Code SummarizationBig-Fixing Process Code SmellsSoftware Testing Non Code Related Software ArtifactsClone Detection Software Energy MetricsProgram Synthesis Image To Structured RepresentationSoftware Security Program RepairSoftware Reliability / Defect Prediction Feature LocationDeveloper Forum Analysis Program TranslationSoftware Categorization Code Location Within MediaDeveloper Intention MiningFigure 3.6: Preprocessing Techniques by SE TaskIt is important to understand that figure 3.6 shows generally how the preprocessing andformatting techniques break down by SE task. However, within the same preprocessingtechnique, there can be small nuances across different studies. There is rarely a standardmethod for the ability to preprocess the data in such a way that will fit the desired DLmodel. Therefore, we suggest looking into the primary studies for more details pertainingto a specific preprocessing or formatting technique of interest. However, there are somedominant techniques researchers use in order to prepare their data to be analyzed by a DLmodel. Specifically, the use of tokenization and neural embeddings are popular for a varietyof different tasks. This was expected, since we knew that source code was the primary data55type used for DL models in SE. Tokenization of that source code is an effective process forpreparing different granularities of code to be fed into a DL architecture.Even more popular than the use of tokenization was the use of neural embeddings(57.14% of the distribution). This technique can use canonical machine learning techniquesor other DL architectures to process the data, meaning that the output from these othermodels are then used as input to an additional DL architecture. We found that Word2Vecand recurrent neural networks (RNNs) were the most popular types of models for thepreprocessing of data. Again, this relates to the high amount of sequential data that isused in SE that DL models can take advantage of. Source code, natural language andother repository metadata can have a sequential nature, which can be captured by thesetechniques. In both cases, the outputs of these models are a series of vectorized numericalvalues, which capture features about the data they represent. These vectors do not requiremuch additional manipulation before they can be used again as input data to another DLmodel.Although tokenization and neural embeddings are the most popular type of prepro-cessing techniques, there are many more that are required for a different types of data orSE tasks. The ability to accurately represent the data to the DL model is what providestraining data for the algorithm to learn from. Any alteration to this process can result inthe learning algorithm focusing on different features, leading to an entirely different finalhypothesis.Our exploratory data analysis discovered that the steps taken in preprocessing werestrongly dependent on the DL architecture used (0.97). Intuitively, this should make sensesince the data must be transformed into a vector that the DL architecture can accept. Ifthe preprocessing steps and architecture are disjointed, then the model will lack the abilityto accurately and adequately represent the final hypothesis. Therefore, we encouragefuture researchers to analyze the preprocessing steps taken by previous authors given theDL architecture they employed. Vice versa, if certain preprocessing steps need to be56taken, future researchers should analyze what DL architectures coincide with the resultingvectorized format of the data.3.4.3 What type of exploratory data analysis is conducted to help informmodel design and training?DL models are incredibly efficient at finding patterns in a large set of data without theintervention of the DL engineer. However, these models have one significant flaw that isfrequently overlooked in the SE community: the quality of the data significantly impactsthe quality of the learned target function. The analysis and quality of data analyzed hasbeen shown to have adverse effects on machine learning models in the task of code clones[16], however, the issues discussed in this work transcend the particular SE task. Giventhese limitations, it is the responsibility of the researcher to analyze and evaluate the data,such that it appropriately represents the relationship between input and output. We referto problems within the dataset as confounding variables, which can explain the results ofthe learned final hypothesis. In this RQ, we analyzed our primary studies to determine ifprecautions were taken to limit the number of confounding variables. Primarily, we wereinterested to see if there exists any sampling bias or data snooping when applying theseDL models to SE tasks.First, we wanted to describe the issues involving sampling bias and data snooping. Wefound that when DL is applied to SE, there were many instances where there were nomethods to protect against these confounding variables. This leads to doubt surroundingthe results of the primary study and can severely limit the conclusions that can be drawnfrom the learned target hypothesis. Sampling bias is a result of collecting data from adistribution incorrectly or in such a way that it does not accurately represent the distri-bution due to a non-random selection process [216]. The repercussions of sampling biasnormally results in a target hypothesis that is only useful for a small subset of the actualdistribution of the data. Authors can unknowingly make claims about their predictedtarget function, which overestimate the actual capabilities of their model, when sampling57bias is not appropriately considered. The best mechanism for reducing sampling bias is tolimit the amount of criteria and filtering the data is subjected to. Additionally, extractingas much useful data as possible will help to effectively train the model and will reducesampling bias.Data snooping is the process of using data more than once for the purpose of train-ing and testing the model. This means that the data the model was trained on is alsoincorporated into the testing and evaluation of the model’s success. Since the model hasalready seen the data before, it is unclear whether the model has actually learned thetarget hypothesis or simply tuned itself to effectively reproduce the results for previouslyseen data points (overfitting). The overall result of data snooping is an overinflated accu-racy from the model. Therefore, it is important that software engineers apply methods toreduce data snooping in order to protect the integrity of their results. This can be donethrough a simple exploratory data analysis or the removal of duplicate data values withinthe training and testing sets.To evaluate the level at which methods to mitigate sampling bias and data snoopingwere already being done, we noted instances within the primary studies where the authorsdid some type of exploratory data analysis before training. Ideally, the authors performedsome type of exploratory data analysis in order to identify if the dataset that has beenextracted is a good representation of the target distribution they are attempting to model.Additionally, data exploratory analysis should provide insight into data snooping of themodel by determining if the training dataset is mutually exclusive from the validation andtesting datasets.There was a noticeable number of primary studies (over 13) that did not mention orimplement any methods to mitigate sampling bias or data snooping. However, this numberis a bit misleading as we only analyzed primary studies for any sign of exploratory analysison their dataset. This exploratory analysis ranged from basic statistical analysis to an in-depth study about the dataset that looked to remove sampling bias and data snooping. Ourprimary objective within this RQ was to bring attention to the oversight of confounding58problems within the data. DL models are entirely dependent on the data to properly modelthe target hypothesis, thus we wanted to encourage future software engineers to heavilyanalyze their dataset and their methods for extraction. We also note that as time hasprogressed, the primary studies we analyzed generally included more details about theirdata exploration process. We hope to see this trend continue as more DL implementationsare used to address SE tasks.3.4.4 RQ3: What deep learning models are used to support SE tasks?In 3.4.4 we looked to investigate the uses and applications of DL models to represent SEartifacts. We primarily focused on two key aspects of a deep learning framework: thearchitecture and the learning algorithm. Analyzing a DL architecture can grant a lot ofinsight into the types of features the authors anticipate will be extracted from the dataset.Additionally, we wanted to empirically determine if certain architectures pair with specificSE tasks. Additionally, we wanted to know what types of diversities existed within the DLarchitectures and why those idiosyncrasies were important when considering the specificSE task at hand. However, we not only considered the DL architectures, which providedinsight to the automated feature engineering inherent to DL, but we also analyzed thelearning algorithms used in conjunction with those architectures. Specifically, we wanted tocreate a taxonomy of different learning algorithms and determine if there was a correlationbetween the DL architectures, the learning algorithms and the SE tasks.3.4.4.1 What types of model architectures are used to perform automatedfeature engineering of the data related to various SE tasks?In 3.4.4.1, we wanted to discuss the kinds of DL models software engineers are using toaddress SE tasks. The DL architectural choice is heavily dependent on the type of databeing analyzed and what types of features the researcher desires to automatically extract.For example, the use of a recurrent neural network (RNN) would automatically extractfeatures related to the sequentialness of the data being analyzed. Likewise, the use of a59convolutional neural network would look for capturing the spatial relationship betweenpixels of visual data. Thus, knowing the type of architecture allows us to infer the typesof features that will be automatically extracted.Our main objective in 3.4.4.1 was to see the variety of different architectures applied tocertain SE tasks. We wanted to know if there are some architectures that are more widelyused and of those models implemented, which have shown to have success in modeling thedata. From this analysis, we can determine if there are other DL architectures that havenot been applied to a specific SE tasks, thus providing the SE community with potentialfuture research directions. However, we first take note of all DL architectures found in theprimary studies, which can be seen in the taxonomy created in figure 3.7.Figure 3.7: DL Model TaxonomyLooking into the initial breakdown of the taxonomy, we found that the use of recurrentneural networks (RNNs) was highly prevalent. RNNs excel in modeling sequential datasince the architecture is formulated such that a weight matrix is responsible for represent-60ing the features between the sequence of inputs [104]. In particular, source code is highlysequential and the prediction of the next source code token can be highly reliant on thesurrounding tokens. Since the most used type of data we found in 3.4.2.1 was source code,it made sense to see many of our primary studies implementing a RNN to model theirdata. Another architecture we saw frequently applied to SE tasks was the encoder-decoderarchitecture. In this architecture, a datum is translated into a feature rich, latent repre-sentation, which is then translated back into altered raw data. Encoder-decoder networksare used for a variety of tasks, but they excel at understanding sequential data. Impor-tantly, RNNs are normally a vital component of an encoder-decoder model, however, thesemodels can also incorporate feed forward neural networks, convolutional neural networksand more. The popularity of this architecture aligns with the findings in 3.4.2.2, whereneural embeddings were a popular form of preparing the data for the DL model. Theencoder’s responsibility is to translate the raw data into a latent representation that thedecoder is capable of understanding and decoding into the target. We frequently foundthat neural networks serve as the encoders for this architecture that take raw data andgenerate a neural embedding for the decoder. Therefore, since neural embeddings weresuch a popular preprocessing technique for data formatting and preparation, it aligns withthe high prevalence of the encoder-decoder DL architecture. In our exploratory data anal-ysis, we found that SE tasks greatly influences the architecture adopted in an approach.The mutual information value between the features of a SE task and a DL architecture is1.40. We also note that the SE research landscape has primarily focused on textual basedproblem, which includes source code. This explains why RNNs are used in 40.48% of thepapers analyzed in this SLR. The encoder-decoder architecture was also seen frequently( 19% of papers), where they take advantage of using RNNs for both the encoder anddecoder portion of the model.We can glean some of the popular types of architectures through our basic taxonomy,however, due to the close relationship between the automated features extracted and theDL architecture, we wanted to explore what architectures are popular within specific SE61tasks. We found a large variety of different networks that have been applied to different SEtasks as seen in figure 3.8. As expected, most of the SE tasks having to do with source codegeneration, analysis, synthesis, traceability and repair, make use of RNNs and encoder-decoder models. This is because source code has many features that are embedded in itssequential nature and these features can be easily captured by RNNs and encoder-decodermodels. Likewise, SE tasks involving the use of images or media data have convolutionalneural networks commonly applied to them.051015202530354045AutoEncoderEncoder-Decoder CNNRNNNeural GPU FNNGraph Neural Network (GNN)Typed-directed encoders (TDE)Restricted Boltzman MachinesTransformerLog-BilinearRecursive Neural NetDeep Reinforcement LearningGANsDeep Belief NetworksSiameseCode Compensation Souce Code GenerationSource Code Retrieval & Traceability Source Code SummarizationBig-Fixing Process Code SmellsSoftware Testing Non Code Related Software ArtifactsClone Detection Software Energy MetricsProgram Synthesis Image To Structured RepresentationSoftware Security Program RepairSoftware Reliability / Defect Prediction Feature LocationDeveloper Forum Analysis Program TranslationSoftware Categorization Code Location Within MediaDeveloper Intention MiningFigure 3.8: DL Architectures by TaskWe wanted to bring attention to some of the less popular types of DL architectures, suchas: siamese networks, deep belief networks, graphical neural networks and autoencoders.Many of these architectures have only been applied to a few tasks but it is important tonote that these architectures have only recently gained any interest. We believe that thesenetworks have a lot of potential when applied to SE tasks and could lead to new researchdirections. We also felt it important to mention the lack of deep reinforcement learningwithin the SE community. Deep reinforcement learning excels at modeling decision-making62tasks. One could argue that deep reinforcement learning is perfect for SE since SE iscomprised of decisions about code structure and algorithm implementation. This is a fairlyopen area of DL in SE that has not really been explored. Only the SE task of softwaresecurity had a DL framework that involved the use of deep reinforcement learning.In addition to the discussion around the DL architectures and their relations to partic-ular SE tasks, we wanted to briefly discuss some trends in the explicit and implicit featuresextracted from these models. As we discussed in 3.4.2.2, it is common for data to be fedinto DL models only after being subjected to certain preprocessing steps. However, insupervised learning, once that data has been preprocessed, the DL model automaticallyextracts implicit features from the preprocessed data in order to associate those featureswith a label or classification. In unsupervised learning, the model extracts implicit featuresfrom the preprocessed data and groups similar datum together as a form of classification.We refer to the preprocessing steps as highlighting certain explicit features, since thesesteps frequently perform dimensionality reduction while maintaining important features.In our analysis we found the most common techniques for highlighting explicit features tobe tokenization, abstraction, creating a neural embedding or vectorizing a latent represen-tation. These techniques attempt to highlight explicit features that are uniquely tailoredto the data being analyzed. Once the data is fed into the model itself, the model is respon-sible for extracting implicit features to learn a relationship between the input data andtarget hypothesis. These features are what the weights of a neural network are intendedto represent and are used to model the target hypothesis. For example, an LSTM is goingto find temporal dependencies of tokenized data such that it can use those features tounderstand the sequential nature of the data and use that to predict the next token ina sequence. A Tree-LSTM can learn the temporal dependencies of tree based structures,such as a program dependency graph (PDG). This allows the model to understand howthe nodes of the tree relate to one another, thus allowing the model to predict additionalnodes or similar tree-like structures. The extraction of explicit and implicit features are63the reasons that the model is capable of modeling a target function, which can be used topredict the targets of unseen data points.0102030405060Source CodeNatural LanguageBug ReportLabeled UI ComponentsRepository MetadataLDA TopicsTime MeasurementsPPBE TasksImages or VideoExecution TracesContextual Flow GraphsProgram Input-Output ExamplesKarel DSL GridAbstract Intents / FiltersSymbolic EquationsCertificatesAutoEncoder Encoder-DecoderCNN RNNNeural GPU FNNGraph Neural Network (GNN) Typed-directed encoders (TDE)Restricted Boltzman Machines TransformerLog-Bilinear Recursive Neural NetDeep Reinforcement Learning GANsDeep Belief Networks SiameseFigure 3.9: DL Architectures by Data TypeFigure 3.9 shows a breakdown of DL architectures by the type of data they are learningfrom. This relationship between data and architecture is important since the architecturesis partially responsible for the type of implicit feature being extracted. For example, imagesand other visual data are commonly represented with a CNN. This is because CNNs areespecially proficient at modeling the spatial relationships of pixel data. There are manyother trends seen when associating data types and DL architectures. We hope that futureresearchers can perform exploratory data analysis and discover the architectures that areoptimal when extracting implicit features from the data. These features can then be usedto model the target function of interest.643.4.4.2 What learning algorithms and training processes are used in order tooptimize the models?In addition to the variety of DL models that can be used within a DL based approach,the way in which the model is trained is very reliant on the learning algorithm chosen.The taxonomy associated with the learning algorithm is extremely diverse and can beclassified in three primary ways: by the weights associated with the model and how theyare adjusted, by the way the overall error associated with prediction is calculated, andby the optimization algorithm, which can systematically adjust the parameters of thelearning algorithm as training progresses. Learning algorithms for a DL approach in SEare primarily used without any alteration or adjustment and are based in mathematicalprincipals that adjust the weights of the model. Since researchers in SE are primarilyinterested in the use of DL models, rather than adjusting the algorithms that power them,a large majority of the studies we analyzed share in the use of out-of-the-box learningalgorithms to power their DL framework. The largest commonality among the studies wasthe incorporation of the gradient descent algorithm as a way to adjust the weights withinthe deep neural network. The breakdown of learning algorithms throughout our SLR canbe found in the following sentences. We found 65/84 of the primary studies used someversion of gradient descent to train their DL model. The remaining studies used gradientascent (2/84), VAE (1/84), policy based learning (3/84), a context free grammar learningalgorithm (1/84), and many studies did not explicitly specify their learning algorithm inthe paper (12/84). Our exploratory data analysis revealed that papers published in recentyears (2018 and 2019) have begun to employ learning algorithms that differ from gradientdescent. Instead, they incorporate a reward policy or gradient ascent.In conjunction with analyzing how the DL approaches adjust their weights, we alsoquantified how these approaches calculate the loss of the model’s prediction when com-pared to the actual target. We found that there are a variety of ways that DL basedimplementations calculate error. However, we did find that a majority of the papers we65analyzed used cross entropy as their loss function (19/84), which was only paired withgradient descent algorithms. Other common loss functions that were used with gradientdescent algorithms were negative log likelihood, maximum log likelihood, and cosine loss.There were a number of papers which did not provide any indication about the loss func-tion within their learning algorithm. We did find that when the primary study was notusing gradient descent as a way to adjust the weights associated with the DL model, theerror functions used became a lot more diverse. For example, the work done by Ellis etal. learned to infer graphical programs from deep learning hand-drawn images. They usedgradient ascent rather than descent as their learning algorithm and also used surrogate like-lihood function as a way to calculate the error of the model [89]. We found that the lossfunction influences the learning algorithm with a mutual dependence of 0.72 bit. However,the SE community omits this parameter in 27.38% of the papers we analyzed. We foundthat approaches that implement reinforcement algorithms are based on a developed policy,which calculates the error associated with the action taken by the model and adjusts theweights to either reinforce or punish that action.Lastly, we looked at the optimization algorithms to determine if there was any note-worthy pattern or intuition as to why certain optimization algorithms were used. What wediscovered is that the choice of optimization algorithm is somewhat agnostic to the model,the weight adjustment algorithm and the error function. In many cases, the optimizationalgorithm was not reported within the primary study (54.76% of the time). Unfortunately,we were not able to ascertain whether optimization methods were never implemented orwere just omited from the paper for space purposes. However, we did analyze the pa-pers that did provide this information and found that there are four major optimizationalgorithms: AdaGrad, AdaDelta, RMSProp and Adam. Below, we briefly address eachoptimization algorithm in order to point out potential situations in which they should beused.Adagrad is an algorithm that adapts the learning rate based on the impact that theparameters have on classification. When a particular parameter is frequently involved66in classification across multiple inputs, the amount of adjustment to those parameters islower. Likewise, when the parameter is only associated with infrequent features, thenthe adjustment to that parameter is relatively high [87]. A benefit of AdaGrad is thatit removes the need for manual adjustment of the learning rates. However, the way thatAdaGrad calculates how much to adjust the parameters is by accumulating the sum of thesquared gradients. This can lead to summations of the gradient that are too large, whichrequires the learning rate to become extremely small.AdaDelta was formulated out of AdaGrad in order to combat this limitation. Ratherthan consider all the sums of the past squared gradients, AdaDelta only considers thesum of the past squared gradients limited to a fixed size. Additionally, this optimizationalgorithm does not require a default learning rate as it is defined by an exponentiallydecaying average of the calculated squared gradients up to a fixed size [293].RMSprop is the next optimization algorithm, however, this algorithm has never beenpublished or subjected to peer review. This algorithm was developed by Geoff Hinton andfollows the similar logic of AdaDelta. The way in which RMSprop battles the diminishinglearning rates that AdaGrad generates is by dividing the learning rate by the recent averageof the squared gradients. The only difference is that AdaDelta uses the root means squarederror in the numerator as a factor that contributes to the adjustment of the learning ratewhere RMSprop does not.Adam, the last of our optimization algorithms discussed, also calculates and uses theexponentially decaying average of past squared gradients similar to AdaDelta and RM-Sprop. However, the optimization algorithm also calculates the exponentially decayingaverage of the past gradients. Keeping this average depending on gradients rather thanjust the squared gradients allows Adam to introduce a term which mimics the momentumof how the learning rate is moving. It can increase the rate at which the learning rate isoptimized [148]. We see all four types in DL approaches applied to SE tasks, and did notidentify any particular patterns or associations worth noting.673.4.4.3 What methods are employed to combat overfitting and underfittingof the models?The two major problems associated with using any type of learning based approach,whether that be canonical machine learning or deep learning, is overfitting and under-fitting. Both overfitting and underfitting deal with the idea of generalization, i.e., howwell does a trained ML/DL model perform on unseen data. Overfitting is the process of amodel learning to fit the noise in the training data extremely well, yet not being able togeneralize to unseen data. The model has learned such a proficient approximation for thisnoisy data that it has become ungeneralizable to unseen data. However, this is not a goodapproximation of the true target function that the model was intending to learn. Under-fitting involves having a learning model that is unable to approximate the seen trainingdata in addition to unseen data points. This can occur when the model lacks the necessarycomplexity, is overly constrained, or has not had the sufficient training time to appropri-ately approximate the target hypothesis. Figure 3.103 gives an overview of some generalways to combat overfitting and underfitting. The figure also addresses what parts of anML/DL approach is affected by these techniques. In this RQ, we were interested in thespecific type of ways researchers are combating these two problems in the context of SEtasks.Outlined in [13], the use of a validation set is a commonly used method for detecting ifa model is overfitting or underfitting to the data, which is why it is very common to splitdata into training, validation and evaluation sets. The splitting of data helps to ensure thatthe model is capable of classifying unseen data points. This can be done while performingtraining, to ensure that overfitting is not occurring. We see this done in almost everypaper analyzed in our SLR. However, more pertinent and powerful techniques to preventoverfitting were seen less frequently. Specifically, we were looking for any regularization3Generated in part from multiple blog posts: https://elitedatascience.com/overfitting-in-machine-learning, https://hackernoon.com/memorizing-is-not-learning-6-tricks-to-prevent-overfitting-in-machine-learning-820b091dc42, https://towardsdatascience.com/dont-overfit-how-to-prevent-overfitting-in-your-deep-learning-models-63274e552323, https://elitedatascience.com/bias-variance-tradeoff68AugmentingIncreasingBalancingEmsemble (Bagging)Decrease ComplexityRegularizationPretrainingEarly StoppingRegularizationOverfittingModel Data TrainingEmsemble (Boosting)Increase ComplexityDifferent ArchitecturePretrainingOptimal CapacitySearch PruningUnderfittingFigure 3.10: Overfitting and Underfitting Overviewtechniques or additional methods the authors implemented to prevent the model fromoverfitting to the training data. As shown in our overview figure, there were three maintypes of regularization. The first regularizes the model, which includes things such asadding Dropout layers [248] or Batch Normalization [131]. The second regularizes thedata itself, either through adding more data or cleaning the data already extracted. Thethird type of regularization is applied to the training process, which modifies the lossfunction with L1 regularization, L2 regularization or incorporates early stop training.As we’ve done with previous components of DL approaches, we wanted to determine ifthe SE task had any affect on the techniques to combat overfitting. Figure 3.11 analyzesthe relationship between DL approaches and the techniques that combat overfitting. Thisfigure shows that there are some techniques that are much more commonly applied toSE tasks than others. For example, dropout was the most commonly used regularizationtechnique and is used in a variety of DL approaches that address different SE tasks. Thistrend holds true for early stop training, data cleaning and L1/L2 regularization. However,the trend is somewhat broken in custom methods implemented to combat overfitting. It isunclear exactly what these custom methods are, but we found their highest prevalence in69the SE task of program synthesis. We suspected that this is due to the highly specializedprograms that are generated from input-output examples. The specialized data may requirea specific technique that is not considered one of the common regularization methods. Inaddition to the aforementioned techniques, we found some unique approaches to combatingoverfitting including the use of deep reinforcement learning instead of supervised learning[61], gradient clipping, lifelong learning [100], changing the loss function [49], pretraining[272, 242], and ensemble models [136].Figure 3.11: Overfitting Techniques per Task TypeAfter analyzing overfitting techniques in relation to SE tasks, we decided to also analyzethe relationship between overfitting techniques and data type. We found a similar patternin that there are a variety of techniques to combat overfitting regardless of the data type.The only exception to this pattern was seen when analyzing natural language, where L1/L2regularization was predominately the most used regularization technique. The messagefrom both figure 3.11 and 3.12, is that the techniques to combat overfitting do not have astrong association with either of these two features. Therefore, we can assume that thesetechniques can work in tandem with a variety of SE tasks and data types to prevent themodel from overfitting. The most concerning trend we took away from these two figures70is the Did Not Discuss category. Given the importance of combating overfitting whenapplying a DL approach, it is troublesome that so many primary studies did not makemention of using these techniques. Although we cannot be sure whether this is a lack ofimplementation or merely reporting the use of these techniques, we hope to see this trenddissipate as DL in SE continues to gain popularity.Figure 3.12: Overfitting Techniques per Data TypeFor the prevention of underfitting, there are not a lot of current techniques that can beeasily applied to the model and training process. However, underfitting greatly dependson the target function and the amount of time needed to appropriately model the targethypothesis. Underfitting normally requires finding the model’s optimal capacity. Theoptimal capacity is the inflection point where the model starts to overfit to the trainingdata and performs worse on the unseen validation set. While there are not many techniquesto combat this issue, there are a few key insights to prevent underfitting the data. Thefirst is to maximize training time without losing the generalizability of the model. Asmentioned previously, this happens when the training process shows increased performanceon the training data, but stagnant or decreased performance on the validation set. Othertechniques include the use of a more complex model or a model more suited for the target71function, which can be determined by varying the number of neurons, varying the numberof layers, trying different DL architectures, pretraining the model, and search pruning thesearch space. From our SLR, the most commonly used underfitting techniques appliedwere search pruning the search space of the model [270, 61] and pretraining [272, 242].Surprisingly, more than 25% of papers did not discuss any techniques used to combatoverfitting or underfitting of their models. Combating this issue is a delicate balancingact, as attempting to prevent one can begin to cause the other if not done correctly. Forexample, having a heavily regularized learning model to prevent overfitting to a noisydataset can lead to an inability to learn the target function, thus causing underfitting ofthe model. This is also possible while attempting to prevent underfitting. An increase inthe number of parameters within the architecture to increase the complexity of the modelcan cause the model to learn a target function that is too specific to the noise of the trainingdata. Therefore, the incorporation of techniques to address overfitting and underfitting iscrucial to the generalizability of the DL approach. Additionally, reporting on techniquesthat combat overfitting and underfitting is important toward the goal of creating replicableand reproducible DL approaches in SE. Due to the span of learning components that thesetechniques affect, knowing which techniques were implemented is an important aspect toreport on.3.4.5 RQ4: How well do DL tasks perform in supporting various SEtasks?In this RQ, we explored the impact that DL has had in SE. It is difficult to quantify andverify the results of the primary studies we analyzed, however, we can infer the impact ofthese approaches based on the metrics used to measure the performance of the models andthe increased automation the approach brings to the SE task. In some regards, this RQ istask specific because performance is based on the difficulties and progress made within theSE task being addressed. We wanted to provide researchers with some intuition about howthe current limits of DL in SE can be extended for particular SE tasks. In many instances,72an increase in performance can have drastic implications on the situations in which theDL model can be applicable. Likewise, it is important to know what aspects of a SE taskhave already been automated by DL models.We also wanted to provide a comprehensive listing of available baselines and bench-marks used for the evaluation of deep learning models in SE. By making future researchersaware of the baselines they can compare to, we hope to foster the promotion of compar-ative evaluations when looking at different DL approaches addressing similar SE tasks.Additionally, it is helpful to know what benchmarks exists and are made available for par-ticular SE tasks. Benchmarks allow for an objective comparison between models lookingat the same data. The use of benchmarks also helps to mitigate problems correspondingwith sampling bias and data snooping. Creating a known listing of available benchmarksshould result in a twofold benefit: the increased analysis and scrutiny of those benchmarksto ensure high quality data for comparison, and to promote future benchmarks in SE taskswhere they do not exist.3.4.5.1 What “baseline” techniques are used to evaluate DL models and whatbenchmarks are used for these comparisons?Within this RQ, we were interested in evaluating the applicability and effectiveness of DLmodels applied to SE tasks. Looking at the baseline techniques used for comparison, aswell as the evaluation metrics, we can establish a sense of how well a particular approachcompares to another within the same task. Specifically, we found that there are certainbaselines and metrics that are common to use for comparison within specific SE tasks.A failure to compare against these common baselines can call into question the impactand superiority of a DL approach over its predecessors. Additionally, we noticed thatthere are not many well documented resources for researchers to find common baselinesand benchmarks used in previous approaches. Thus, we highlighted the need for the SEcommunity to compile and make available these common baselines and benchmarks.73We looked at our primary studies in DL4SE and noted that baseline approaches areextremely individualized, even within the same SE task. Some DL4SE papers do notcompare against any baseline approaches while others compare against 3-4 different models.Therefore, we included the listing of baselines that each paper compared against in oursupplemental material. We found that many of the baseline approaches were canonicalmachine learning models or very simple neural networks. We suspected the reason for thisis in part due to DL4SE being a relatively new field, meaning that there were not manyavailable DL based approaches to compare against. Additionally, the use of canonicalmachine learning models as baselines demonstrates the need and benefit of applying DLto the specific SE task. However, as DL4SE begins to mature, we hope to see the trendtransition to evaluations that include comparisons against previously implemented DLapproaches.Additionally, we find a reoccurring trend of model implementations being unavailableto the public. This, in part, explains why there are so many highly individualized, baselineapproaches. Since researchers do not have access to common baselines used for comparison,they are forced to implement their own version of a baseline. It is possible that baselineapproaches are implemented incorrectly or configured in such a way that gives them adisadvantage over the newly created approach. Thus, it can be difficult to fully rely onthe results when compared to baseline approaches since many papers did not include anyinformation about the baselines themselves. Additionally, a unique implementation of thesame baselines could lead to confounding results when looking at improvement over thosebaselines. We want to encourage future DL4SE studies to include the details surroundingthe implementation of baseline approaches or to include the implementation in an onlinerepository.The use and proper implementation of baseline approaches is essential to a quality eval-uation of a DL model. However, the data used when comparing two different approachesis equally important. As we discussed in 3.4.2.2, DL models are very dependent on a highquality dataset free of sampling bias and data snooping. There is a need within the SE740510152025Code ComprehensionSouce Code GenerationSource Code Retrieval & TraceabilitySource Code SummarizationBig-Fixing ProcessCode SmellsSoftware TestingNon Code Related Software ArtifactsClone DetectionSoftware Energy MetricsProgram SynthesisImage To Structured RepresentationSoftware SecurityProgram RepairSoftware Reliability / Defect PredictionFeature LocationDeveloper Forum AnalysisProgram TranslationSoftware CategorizationCode Location Within MediaDeveloper Intention MiningSelf GeneratedPrevious BenchmarkSelf Generated & AvailableFigure 3.13: Benchmark Usage DL in SEcommunity to objectively evaluate these common benchmarks to ensure that comparisonsbetween approaches are not skewed due to underlying attributes of the data. Likewise, ifa new dataset or benchmark is introduced for evaluation, it is important that the authorscan verify the quality of the dataset, such that reviewers can determine if it is fit to beused for comparison with other approaches. As new benchmarks are introduced for spe-cific SE tasks, we can reduce the amount of exploratory data analysis, since we will havemore data vetted by reviewers. Since DL models themselves are unbiased mathematicalmodels, any bias or discrimination demonstrated by the DL approach can be attributedto the preprocessing techniques or the dataset itself. This emphasizes the importance ofreviewers vetting benchmarks that are used in DL4SE studies.Within our online repository, we included a list of all the benchmarks and baselinesused for each paper within our SLR [280]. The reason we did not include the full list hereis due to a number of unique benchmarks synthesized specifically to evaluate a particularapproach. To solidify this point, we recorded the number of primary studies that used apreviously curated benchmark as opposed to ones that curated their own benchmark. We75noted that there is an overwhelming amount of self generated benchmarks. Additionally,we classified self generated benchmarks into those that are publicly available and thosethat are not. Unfortunately, we found a large majority of self generated benchmarks maynot be available for public use. The full breakdown of benchmarks used in the primarystudies can be seen in figure 3.13. This trend within DL4SE is worrying as there arefew instances where DL approaches can appropriately compare against one another withavailable benchmarks. We hope that this SLR helps future researchers by providing themwith an understanding about which benchmarks are available for an evaluation of theirapproach within a specific SE task. Additionally, we urge future researchers to make selfgenerated benchmarks publicly available, which will provide a much needed resource notonly for comparisons between approaches, but also for available data that a DL frameworkcan use to train on.Although the use of previously established benchmarks is not the norm, we did wantto highlight a few benchmarks we saw used multiple times within the primary studies.For the SE task of clone detection, we found that the dataset BigCloneBench was usedfrequently to test the quality of the DL frameworks. Also, for the task of defect prediction,we saw uses of the PROMISE dataset as a way to compare previous DL approaches thataddressed defect prediction in software. We believe that these two benchmarks are a greatexample of how benchmarks can be used in the future. These are popular, well document,and tested benchmarks that have been used to compare DL approaches within the sameSE task.The aspect of vetted benchmarks from the primary studies in DL4SE was the leasttransparent. In particular, details involving the baselines used in the evaluation werescarce or nonexistent. Without the implementation details of the baselines and a detailedanalysis of the benchmarks, it is very difficult to objectively compare DL models in theirability to address certain SE tasks.Understanding baseline approaches and accessing appropriate benchmarks is a crucialpart of evaluating any DL based approach in SE. However, we also explored the common76Table 3.3: Metrics Used for EvaluationMetrics StudiesMMR [142, 58, 123, 159, 70, 107, 52]BLEU Score [59, 136, 272, 110, 54, 120]METEOR Score [58, 271]Precision [263, 179, 170, 119, 159, 18, 285, 289, 276, 112, 76, 232][122, 65, 67, 28, 107, 162, 175, 72, 129, 281, 261, 211, 199]Recall [180, 170, 119, 18, 289, 276, 112, 76, 232, 122, 65, 67][28, 48, 162, 175, 176, 72, 129, 281, 52, 261, 211]F1-Measure [180, 119, 297, 18, 289, 23, 277, 65, 67, 161, 162, 72][129, 281, 261]Accuracy [166, 202, 123, 170, 100, 75, 167, 119, 50, 43, 294, 78][250, 203, 46, 227, 219, 56, 289, 178, 79, 76, 49, 270][61, 125, 232, 122, 28, 55, 162, 175, 60, 99, 274, 21][300, 241, 89, 173, 66, 129, 211, 27, 120]CIDER [272, 300]ROC [297, 232, 99, 281]Root Mean Squared Error [116, 237]Model Perplexity [282, 287]Timing [285, 36, 88, 89, 191]AUC [232, 65, 67, 99, 72, 281]Macro-averaged Mean Absolute Error (MMAE) [65, 67]Mean Absolute Error [65, 66]Top K Generalization [241, 200]Top K Model-Guided Search Accuracy [241, 181]Solved Tasks [242, 88, 281]Other [136, 182, 230, 102, 272, 203, 297, 287, 125, 67, 65, 107][53, 71, 274, 66, 261, 200]metrics used to measure the performance of DL models applied to their respective SEtasks. In many instances, the metrics chosen to analyze the model are common to thetype of learning. Therefore, many of the supervised learning methods have metrics thatanalyze the resulting hypothesis, such as the accuracy, true positives, F1 measure, precisionand recall. These metrics are used to compare the supervised learning algorithms withthe outputs representing the target hypothesis. Intuitively, the type of metric chosento evaluate the DL based approach is dependent upon the data type and architectureemployed by the approach. Figure 3.3 describes the distribution of metrics found in thisSLR. We included the top used metrics to give researchers and indication about whichmetrics are popular for evaluation. The other category is comprised of the less popularmetrics including: likert scale, screen coverage, total energy consumption, coverage ofedges, rouge-L, CIDER, jicard distance, minimum absolute difference, Kruskal’s υ, crossentropy, macro-averaged mean cost-error, matthews correlation coefficient, f-rank, top-k generalization, top-k model-guided search accuracy, median absolute error, spearmans77rank and confusion matrix. We included these for completeness, but it is important to beaware that these metrics can be specialized toward evaluating the model’s ability to predictthe specific target outputs. Interestingly, we found that many supervised DL approachesomit the ROC or AUC evaluations with a condience level of 0.87. This metric should beincluded in the comparison with baseline approaches to demonstrate the improvement ofthe current approach.3.4.5.2 How is the impact or automatization of DL approaches measured andin what way do these models promote generalizability?The impact of a particular approach in research is somewhat subjective and can be sus-ceptible to bias. Therefore, we did not look to validate or invalidate any of the researchpresented in this SLR. We attempted to objectively look at the claimed impacts that theauthors of these primary studies have made as a way to determine what value the ap-plication of a DL model had for the SE task addressed. In many cases, DL approachescontribute a variety of advantages both in correctness and efficiency of addressing a SEproblem. Therefore, we give a brief overview of the potential implications that the primarystudies included within this SLR can have. These implications are primarily pulled fromthe claims of the authors about the impacts of their approach.We classified each primary study into seven categories, which represents the majorcontribution of the work. The result of this inquiry can be seen in figure 3.14. We providedthis analysis to show future researchers that DL can contribute to the advancement of theSE field in multiple ways. However, the most common contribution of a DL4SE study isthe ability to increase automation or efficiency. Interestingly, SE tasks that had multipleDL implementations frequently claimed another novelty such as an advancement in thearchitecture or increased performance over their predecessor.Our exploratory data analysis of this RQ revealed that the automation impact is de-pendent on the SE task deduced from a high mutual information of 0.976 bits. This meansthat there is a strong association between the SE task and the automation impact of the78approach. We found three primary objectives that the implementation of a DL model ismeant to address: (i) in 75% of SE tasks observed, a DL approach was implemented withthe main goal of increasing automation efficiency; (ii) in 45.8% of the SE tasks observed,a DL approach was implemented with the main goal of advancing or introducing a novelarchitecture; (iii) in 33% of the SE tasks observed, a DL approach was implemented withthe main goal of replacing human expertise. We discuss this taxonomy so that futureresearchers can think about the main objective they intend their DL approach to address.This objective can be used to help shape the evaluation and the metrics used to quantifyeffectiveness.19345 617120510152025303540Open VocabularyIssueIncreasedPerformanceOverPredeccesorIncreasedAutomation /EfficiencyIncreasedUnderstanding ofthe TopicSolvingPreviouslyUnsolvableProblemsAdvancedArchitecture /NoveltyReplacingExpertiseFigure 3.14: Impact of DL4SEWithin our SLR, we saw many DL implementations for a variety of tasks. Yet, wenoticed that the primary studies did not put as strong an emphasis on the generalizationof a model as opposed to the other aspects of the DL approach. The generalization of amodel is very important as it outlines the conditions in which the model can be applied tothe task. If a model lacks generalizability, then it may not be applied to data outside of the79training and testing set. This can make the DL approaches in SE too specific for developersworking on production software. Thus, in this RQ, we analyzed how these primary studiesensured the generalizability of their model. To do this, we looked for evidence that theauthors considered Occam’s Razor, as well as the efficiency of their model.Before continuing into our analysis, we wanted to address Occam’s Razor and howit relates to DL approaches in SE. Occam’s Razor was originally translated to "Entitiesshould not be multiplied beyond necessity" and is quoted from William of Occam in thelate Middle Ages [235]. In the current age, this concept has been applied to the fieldof machine learning to address the over engineering of learning models resulting in lessgeneralizability. Currently, Occam’s Razor is separated into two Razors: 1) "Given twomodels with the same generalization error, the simpler one should be preferred becausesimplicity is desirable", 2) "Given two models with the same training-set error, the simplerone should be preferred because it is likely to have lower generalization error" [85]. Whenspecifically applied to deep learning, we were interested in determining if the SE taskcould have been solved with a less complex model. Unfortunately, it was very difficult toprove Occam’s Razor in the context of a primary study focused on a SE task. Therefore,we looked for evidence that Occam’s Razor was considered by the authors but did notexplicitly analyze how the complexity of the model was fully evaluated. In figure 3.15 webreak the primary studies into three groups: 1) those that compare against less complexmodels and analyze the results; 2) those that manipulate the complexity of their own modelby testing a variety of layers or nodes per layer; 3) those that did not have any Occam’sRazor consideration.Although a majority of the primary studies do consider Occam’s Razor, there are stillalmost 25% of DL4SE studies that do not consider Occam’s Razor. It is possible that thestudies we analyzed did not include details about how they considered the complexity oftheir model when applied to the specific SE task. However, these details are vital to includewithin the primary study, since they provide evidence that DL is appropriate to apply tothe SE task. Without this consideration, it is possible that a canonical machine learning80No Consideration Varied The Model Model ComparisonFigure 3.15: Evidence of Occam’s Razormodel or a simple statistical based approach may be the optimal solution. Interestingly, inabout 15% of the primary studies, the author’s considered Occam’s Razor by adjusting thecomplexity of the model being evaluated. This is done by varying the number of layers,the number of nodes, the size of embeddings, etc., which all contribute to the complexityof the approach. The downside to this method is that there is no way to determine ifthe extraction of complex hierarchical features is more complex than what is necessary toaddress the SE task. The only way to properly answer this question is to compare againstbaseline approaches that are not as complex. In our DL4SE studies, this often lookedlike a comparison with a canonical ML approach. This further bolsters our point madein 3.4.5.1, that baseline approaches are essential to establish the validity of a DL basedapproach.813.4.6 RQ5: What common factors contribute to the difficulty reproduc-ing DL studies in SE?This SLR relies on the details given by the authors of DL approaches applied to SE tasks.As we progressed through our review, we wanted to highlight the importance of reportingkey components for the purpose of reproducibility. Reproducibility is of great importanceto the research community and as DL becomes a more popular tool to model softwareartifacts, it is important that papers include the necessary details so that reproducingthe study is possible. DL models can also be extremely complex and highly customizableto fit the needs of the particular SE task. We wanted to emphasize this point sinceeven small, nuance changes can lead to drastic affects in the approach’s performance.These slight alterations do not necessarily refer to the DL model. They also apply to theextraction and preprocessing of the data, the learning algorithm, the training process andthe hyperparameters, which all have a significant contribution to the overall success of aDL approach. In this RQ, we surveyed the work of our SLR to determine the importantconcepts and details that need to be reported when implementing a DL based approach.The first and easiest analysis to perform was determining the ability to replicate theDL4SE approaches. To replicate an approach means to re-implement the approach exactlyas the authors originally did. Due to the variability that each approach can have, the dataand source code should be publicly available in order to deem the work replicable. Wefound that out of the 84 primary studies we analyzed, only 28 provided enough informationthrough an online repository to fully replicate the approach. This means that the vastmajority of DL4SE studies either did not provide the implementation of the approachor did not provide the dataset to train and test the approach. We also analyzed thereproducibility of DL4SE studies. We define reproducibility as the ability for someonewithin the field of DL4SE to read the study and implement the approach with slightdifferences either in the implementation or data representation. In this case, small detailscan be missing but the steps to reproduce the approach, it its entirety, should be present.82We found that 43 out of the 84 studies we analyzed were capable of being reproduced. Outof the 43 studies that can be reproduced, only 18 of those studies were also replicable. Ourgoal here was to further analyze what factors were responsible for the lack of reproducibilityamongst the remaining 41 primary studies. We found that there were ten major factorsthat contributed to the lack of reproducibility. The breakdown of this analysis for theprimary studies are show in in figure 3.16.Learning Algorithm Hyperparameters Architecture Input Metrics Training ProcessImplementation Data Extraction Preprocessing Data Filtering Online RepositoryFigure 3.16: Contributing Factors to ReproducibilityIn figure 3.16 we show areas where DL approaches in SE may be lacking the necessarydetails to reproduce or reimplement the approach. The first two areas that contribute tothe difficult of reproducibility pertain to the learning algorithm and the hyperparameters.The learning algorithm is the way in which the parameters of the model are tuned to appro-priately estimate the target function. We found that there were missing details pertainingto either the method of tuning the weights, the way in which the error was calculated orthe optimization algorithm. All three aspects of the learning algorithm can make reim-83plementation of the approach difficult and prone to errors. Without the proper learningalgorithm, it is likely that any attempt to reproduce the study will vary from the originallyreported results. This is because there are a vast amount of combinations of these threeaspects to the learning algorithm, but all contribute significantly to the reported resultsof the study. In addition to the learning algorithm, the hyperparameters are also crucialas they can have an affect on the learning algorithm and the model architecture. Conse-quently, if the hyperparameters are not reported, then it is impossible to know how manyparameters contributed to the estimation of the target function, since hyperparameterscontrol the number of layers and the number of nodes per layer. Additionally, the waythat the learning rate is adjusted is ultimately what controls the parameters that estimatethe target function. An incorrect learning rate can lead to incorrect parameters, which inturn leads to a completely different target function that is modeled.Although the details pertaining to the learning algorithm and the hyperparameters isimportant, they are not the only details that a DL approach would need to report in orderto be replicated or reproduced. Particularly, the data and the way it was extracted, filteredand formatted would need to be known in order for a study to be reproduced. DL modelsare data driven, meaning that they extract features from the data without any humanintervention. In order for the study to be reproducible, three pieces of information needto be accessible. The first is the extraction details. In order for a study to be reproduced,the data must be accessible which means either the dataset must be readily available orthe details about how the data was extracted need to be reported. The second piece ofinformation that is needed is the preprocessing details. Once the data is extracted, it needsto be formatted into a representation that the DL model can accept as input. Additionally,the way that the data is represented to the model, in some part, controls the features thatare able to be extracted. So if the data is represented contrary to the way it was donein the original study, then the results of the reproduced study would be invalid. Thelast attribute of the data that needed to be known was the filtering details. Data can beinherently noisy and authors will frequently need to filter out noisy data in order for the84DL model to learn an effective relationship between the input and the target. This processcan involve the removal of certain data points or an entire section of the original datasetbased upon a criteria that the authors set. We discovered that 44/84 primary studies aremissing crucial details about the filtering of their data. Reporting these details related tothe filtering of data should not only include the filtering criteria, but should also explainthe steps and methodology taken to remove this data from the dataset.We performed exploratory data analysis pertaining to the reproducibility of the studiescontained within our SLR. Here, we attempted to establish patterns for why papers inSE struggle with irreproducibility. We discovered that SE papers not using visual data,repository metadata, natural language data, or I/O examples have a strong association withirreproducibility (0.70) with a relatively high confidence of 0.82. We also found that notusing visual data, repository metadata, or I/O example is strongly associated with lackingextractions details (0.71) with a high confidence of 0.82. Lastly, we found that not usingvision data or repository metadata was associated with a lack of data preprocessing details(0.74) with a confidence of 0.81. We describe these associations to bring awareness to theissues with reproducibility. From these facts found within our exploratory data analysis,we created guidelines so that future researchers know what information to include whenreporting on their DL based approach in SE.Lastly, we wanted to promote the idea of open sourced models and datasets. This isnot only the best mechanism for reproducibility, but also grants access to datasets andDL approaches to future SE researchers. We have mentioned before that DL models areextremely data hungry. However, there are not many publicly available datasets containingsoftware artifacts to train DL models on. This not only inhibits the use of DL in SE, but italso makes comparative studies with different DL models difficult. The lack of data hurtsthe evaluation of DL based approaches because there are very few baselines to compare anewly synthesized model to. This can lead to claims regarding the effectiveness of a DLimplementation that can be difficult to refute or verify. Therefore, we encourage futureresearchers to make the datasets and DL models publicly available. We believe that this85will drive a greater quality of research and allow for verifiable comparisons of DL basedapproaches in SE.3.5 Guidelines for DL in SEIn this section, we provide guidelines for the future use of DL within SE. We first generateda flow diagram that provides a logical pathway through the entire process of applying DLin SE. The diagram, seen in figure 3.17, provides the steps that need to be taken, but werefer back to the sections of our SLR for the specifics behind addressing each componentof learning. In addition to the flow diagram we also compiled additional guidelines, whichcan be treated as a sort of checklist, containing certain pitfalls or points to consider whenworking through the components of learning. This can be seen in figure 3.18. We do notclaim this as a comprehensive guide, as each implementation of these complex models havetheir own nuances. However, we do cover many of the essential aspects of applying DL toSE and make future researchers aware of the context in which DL can be a useful tool.After the implementation of the DL model, the reporting process should be completeand thorough. We want to reiterate what we found in figure 3.16 to help guide futureresearchers through the reporting phase of their research. The reporting process shouldbegin with data extraction. Here, the researchers should note any potential limitations orbiases within the data extraction process. DL models are limited in scope by the data theyanalyze, therefore, any restriction on data extraction or additional data filtering should bewell documented. The next phase of the reporting process should address the preprocessingsteps taken to prepare the data for the DL model. Steps within this process should beexplicitly explained to the point where replication is possible. This would include anyabstractions, filters, vocabulary limitations or normalizations that the data is subjectedto. In addition, the researchers should report how they vectorize the resulting data, whichincludes the format of the vector or matrix subjected to the DL model.86After reporting on the data extraction, the data filtering, and its preprocessing steps toprepare for the DL model, the authors should begin addressing the DL framework. This in-cludes information about the learning algorithm, DL architecture, hyperparameterization,optimization methods, and training process. There are many common learning algorithmsused, with the most prevalent being variants of gradient descent. It is important that thelearning algorithm be discussed, as this can significantly affect the results of the model. Inaddition to the learning algorithm, the optimization algorithm should also be mentioned.The optimization algorithm describes the process of adjusting the parameters of the modelwhich then results in learning. Researchers should also fully elaborate on the DL architec-ture used in their approach. This elaboration should include things such as the number ofnodes, number of layers, type of layers, activation function, etc. The approach cannot bereplicated without all necessary information pertaining to the architecture and should beavailable to the reader. Lastly, authors should report the hyperparameters and trainingprocess. The hyperparameters directly affect the results of the model as their values di-rectly control the process of the model’s learning. Hyperparameters are adjusted and setby the author, therefore the process of finding the optimal hyperparameterization, as wellas their final values should be explicitly stated. The reporting process should also includedetails pertaining to the training process of the DL model. This can include the number oftraining iterations, the time taken to train, and any technique that would help to preventoverfitting of the model.In addition to the patterns discovered in this work, our findings corroborate many ofthe issues discussed in the study by Humbatova et al. [130] This study analyzed realfaults discovered in DL systems from GitHub. They found three major areas of errorwhen implementing DL systems: errors in the input, the model and the training process.Interestingly, they interviewed developers to determine the severity and effort required toaddress many of these pitfalls. They found that developers thought the most severe errorsrelated to the proper implementation of the optimizer, entering in the correct input dataand correctly implementing the models with the right number of layers. However, they87also learned that developers find these three severe errors took a relatively low amount ofeffort to fix [130].Our research demonstrates the components of learning within DL4SE work that havebeen incorrectly implemented or not implemented at all. When taking into considerationthe findings of Humbatova et al., we hypothesize that the amount of effort needed to fixthese problems when applying the DL framework could be the reason it is not performed.For example, the adjustment and appropriate values of hyperparameters can be an im-pactful problem when applying a DL model. However, developers rated this issue to bethe third highest in the amount of effort needed to address it. Likewise, we see gettingthe training data and training process correct are ranked number two and number one,respectively, for the most amount of effort required to address the issue. We believe thatfollowing these guidelines and looking at previous implementations of DL in SE will aidfuture researchers in knowing how to quickly and effectively address the issues that arisewhen applying these complex models.Our intention with these guidelines is to bring attention to crucial details about apply-ing DL in SE. The figures and explanation included here is not intended to be a compre-hensive list of all information to include, rather, it is intended to serve as a basic checklist.Every DL based approach has its own idiosyncrasies that may or may not be importantto include when reporting on a DL approach in SE. However, all details mentioned in ourguidelines should be readily available in order to encourage reproduction and replicationof the experiments done. We also encourage authors to promote transparency of their ap-proach by making all datasets, models and implementation scripts available via an onlinerepository. This will lead to increased quality of future DL research in SE and allow formore meaningful comparisons of different DL approaches addressing similar SE tasks.883.6 SE4DLWhen searching for papers that could be classified as DL4SE, we ran across some pri-mary studies that give way to the new rising field of software engineering for deep learning(SE4DL). This field has seemingly come to fruition from the issues discovered with au-tonomous vehicles. Since DL models extract complex, hierarchical features automaticallythrough a sequence of nonlinear transformations, developers of DL models are not exactlysure what features are being valued to make the appropriate classification or prediction.This is why DL models are frequently referred to as a black-box. Data is fed into them,mathematical concepts drive the learning and training of these models and they are ableto perform classification with a high level of accuracy. This dramatically changes the tra-ditional paradigm of creating software, which is based on the software life cycle. Previoussoftware engineering analyzes a problem, designs a program which attempts to solve thatproblem, the program is then implemented, tested and maintained. The introduction ofdeep learning renders this paradigm of SE useless because the model is what is respon-sible for generating the program that will solve the task at hand. The developer of theDL based approach is responsible for giving the algorithm the input and its associatedtarget or output. The model then learns the appropriate target function, meaning thatit essentially generates the program to model the input and output examples. Since thedeveloper is not privy to the way in which the DL algorithm generated the program, it isdifficult to test and maintain the program using the traditional software engineering lifecycle paradigm. Specifically, the area of testing software is particularly challenging. Thecurrent methodology for testing DL models is to feed the model data and evaluate theaccuracy on its ability to predict the appropriate target. However, software which has veryminimal room for error, such as that for an autonomous vehicle, cannot be confident in amethod which can only be tested in an ad hoc manner. There is a need for a systematicmethodology that instills confidence in a model, which provides assurance that the modelcorrectly estimates a target function represented by the data.89Currently, this type of systematic methodology does not exist. However, software engi-neers have developed tools and ideologies that can be used to test these black-box models.Thus, a new field has arisen in SE4DL where the concepts learned from software testingand maintenance are being applied to the development of software based on DL algorithms.The process of our SLR captured some of the primary studies that have worked in apply-ing SE ideals to the models generated by DL algorithms. These works focus on problemssuch as concolic testing for deep neural networks, addressing the lack of interoperabilityof DL systems by developing multi-granularity testing criteria, generating unique testingdata through the use of generative adversarial networks, detecting bugs in the implemen-tations of these DL models, seeded mutation of input data to create new testing data,specifically detecting bugs when using certain ML frameworks such as TensorFlow anddetecting erroneous behaviors in autonomous vehicles powered by deep neural networks[251, 186, 295, 91, 188, 292, 299, 113, 90, 147, 198, 296, 258]. These pieces of work areonly the beginning of a growing field that will attempt to understand and interpret theinner-workings of DL models.3.7 Additional ClassificationsIn the midst of performing the SLR, we encountered studies which passed our initialinclusion criteria, but were eventually excluded based on their lack of a DL implementation.However, there has been debate about what type of implementation qualifies as a DL basedapproach. This SLR maintains the definition that deep learning models must automaticallyextract complex, hierarchical features from the data it is fed. This implies that the datamust be subjected to multiple, nonlinear transformations by passing it through multiplehidden layers within a neural network. This type of model would exclude algorithmsthat we would consider to belong to one of the following categories: artificial intelligence,canonical machine learning and representational learning. This hierarchy and definition ofdeep learning was taken from Goodfellow et al. [104]. Despite the limitations we set on90our SLR, we wanted to make the community aware that there are a variety of learningbased approaches that have been applied to the field of SE.There were two common types of papers we encountered when performing our SLR thatwe felt deserved a brief introduction and explanation as to why they were not included.The first is primary studies which use Word2Vec or some variation of Word2Vec in orderto embed some type of sequential data. While we agree that Word2Vec is an unsupervisedlearning algorithm, it fails to extract complex hierarchical features of the input data. Thismodel is a two layered neural network that excels at creating neural embeddings for wordsor other sequential data. This type of model learns the proper embeddings to accuratelyrepresent a set of words as a set of feature vectors, which are in a numerical format thata DL model could understand. We frequently see this type of embedding technique usedin conjunction with DL models but do not consider it a DL model on its own. Theother type of learning approach we encountered was the use of canonical machine learningalgorithms. These types of algorithms require manual feature engineering and therefore, donot rely on nonlinear transformations to learn complex hierarchical features automatically.Although these types of algorithms do require a more substantial manual effort, theyperform relatively well when data is extremely scarce or the developer wants to controlwhat specific features the model considers when being fed data. Canonical ML methodsare frequently used in SE and are applied to a variety of tasks, but the models do not fitthe criteria to be considered deep learning and are excluded from the study3.8 ConclusionsThis work performed a SLR on the primary studies related to DL4SE from the top soft-ware engineering research venues. Our work heavily relied on the guidelines laid out byKitchenham et al. for performing systematic literature reviews in software engineering.We began by establishing a set of research questions that we wanted to answer pertainingto applications of DL models to SE tasks. We then empirically developed a search string91to extract the relevant primary studies to the research questions we wanted to answer. Wesupplemented our searching process with snowballing and manual additions of papers thatwere not captured by our systematic approach but were relevant to our study. We thenclassified the relevant pieces of work using a set of agreed upon inclusion and exclusion cri-teria. After distilling out a set of relevant papers, we extracted the necessary informationfrom those papers to answer our research questions. Through the extraction process andthe nature of our research questions, we inherently generated a taxonomy which pertainsto different aspects of applying a DL based approach to a SE task.Through the process of our SLR, we discovered the presence of papers using a Word2Veclike algorithm as well as implementations of canonical ML algorithms. In addition, wediscovered evidence of an emerging field in software engineering: software engineeringfor deep learning (SE4DL). SE4DL applies fundamental concepts from software testingand maintenance to redefine the paradigm of developing software using DL methodologiesrather than the traditional software life cycle. Our hope is that this SLR provides futureSE researchers with the necessary information and intuitions for applying DL in new andinteresting ways within the field of SE. The concepts and relationships described in thisreview should aid researchers and developers in understanding where DL can be appliedand what the necessary considerations are when applying these complex models.92 Application of Deep Learning to SE Task Identification of potential features that can be used to establish target function Identify a specific SE Task you wish to analyze. Are there enough online repositories that can provide data to model a hypothesis? Deep Learning may be applicable to this problem May need to consider an alternative mode of analysis. Develop a data extraction technique to gather specific data that can be used to model a target function of interest. Yes No Consider the scope and generalization of the data you’re extracting. Did you extract data from a wide variety of places? Is there any bias within your extraction methods? Is there enough data to warrant a DL model? After data extraction, begin the preprocessing and formatting steps to yield the training examples. What are the abstract features you are trying to extract? Are you limiting sampling bias within the training and testing sets? Are you checking for data snooping between the training and testing sets? Determine if you are working with labeled or unlabeled data, as well as the nature of the data (text based, vision-based). This will heavily affect the type of DL architecture you may want to implement. Separate the dataset into a training set, a validation set and a testing set. Generate a representation for each training example. Ensure that the representation captures the features essential for modeling the target function. At this point you should have completed data exploratory analysis. It is important that you analyze the attributes of your dataset as it provides insight into what type of data examples the target function will model. Additionally, one can check for problematic issues such as class imbalance of the data. Determine the best architecture to highlight the features that are best for modeling the target hypothesis. The architecture is what is responsible for the automated feature engineering. After determining a generalized architecture, you will want to decide on a learning algorithm and training process. This can involve the manipulation of hyperparameters and the efficiency of the model. When considering the architecture, learning algorithm and training process, there are two important concepts to be aware of. The first is the efficiency of the DL model chosen. This can partially be addressed by the consideration of Occam’s Razor. Comparing your DL model against less complex models (canonical ML and statistical models) can provide evidence that your model is the most efficient model possible. The second concept to consider is the possibility for overfitting and underfitting. To avoid these problems, techniques such as early stop training, hyperparameter searching, data balancing, data augmentation and regularization should be implemented. The final step is to determine the hypothesis that best generalizes the target function. This target function is generated by the model learning the features extracted from the training examples. This may include the comparison of less complex models. Lastly, the evaluation and comparisons with other approaches should be completed. This involves the identification and comparison of previously used benchmarks within the specified SE task. Researchers should consider common metrics to evaluate how generalizable and accurate the model is. 1 2 1 3 1 4 1 6 2 8 6 9 6 10 6 6 126 6 5 1 7 1 11 6 6 Figure 3.17: Guidelines93 Additional Guidelines for SE Task • Software engineering tasks are strongly associated with several other features. In particular, there is a strong association with data preprocessing, loss function and architecture [ref]. • The SE task must contain enough analyzable data to warrant the use of a DL architecture. • Deep learning excels at identifying key features that help represent the relationship between the input and output. Ensure that you can identify this relationship within the input and target output of your data. • Deep learning has been applied to numerous SE tasks in the past. Refer to RQ1 to identify which tasks have been analyzed with DL. Additional Guidelines for Data Extraction • The most prevalent extraction technique was scraping online repositories or forums for source code, meta data or developer written statements. • Data extraction should be done using the widest variety of potential sources. Otherwise, the limitation of sources should be documented. • Be careful to inspect and remove instances of bias within the extraction methods, if there are biases, document the reasoning for the bias. Additional Guidelines for Data Preprocessing • Data preprocessing is strongly associated with the specified SE task. Therefore, refer to RQ1 to identify the unique associations between SE task and preprocessing technique implemented. • The most prevalent preprocessing techniques were tokenization and the generation of neural embeddings. • Neural embeddings are generated using neural networks, which highlight and abstract features about the dataset into a vector format. • Tokenization is used primarily with source code or natural language. All other relationships of preprocessing techniques to data type can be found in RQ2 of our SLR. • The task of data preprocessing will frequently incorporate some type of abstraction, which inherently limits the applications and scope of the approach. These limitations should be thoroughly explored and documented. • The state of the data after preprocessing will be fed into the DL architecture. Ensure that the vector is in the proper format and features have been appropriately represented. Additional Guidelines for DL Architecture and Learning Algorithm • In SE we find that the most prevalent type of DL architecture is Recurrent Neural Networks (RNNs) and Encoder-Decoder models. This is likely because source code is so heavily studied, which RNNs and Encoder-Decoder models excel in. • Many architectures in SE have not been thoroughly explored. For instances of which DL architectures have been applied to specific SE tasks, refer to RQ 3 of our SLR. • We find that the DL architecture is heavily influenced by the SE task being studied. Specific architectures excel at capturing specific data types better than others. For example, CNNs naturally capture the relationships of images where as RNNs naturally capture the features of sequential textual data. • DL architecture and loss function have a significant association with one another. We find that gradient descent is the most employed learning algorithm. • Do not forget to implement and report on the parameter optimization methods used to optimize the DL model. • To limit the model from overfitting, most authors performed a series of techniques. The most popular techniques found were early stop training, dropout and L1/L2 Regularization. Additional Guidelines for Evaluation • Many DL4SE works compile their own benchmarks for testing their DL-based approach. Many of these benchmarks are available for future comparisons with other DL-based approaches. • Not many implementations of other DL based approaches are publicly available. A compiled list of publicly available approaches can be found in RQ4 of our SLR. • The three most common metrics we find used in DL4SE papers is recall, F1 measure, precision and accuracy. However, there are many other metrics used, which can be viewed in supplemental material. Figure 3.18: Additional Guidelines94Chapter 4Learning Code Similarities4.1 IntroductionSource code analysis is a rock-solid foundation of every software engineering (SE) task.Source code analysis methods depend on a number of different code representations (ormodels), which include, source code identifiers and comments, Abstract Syntax Trees(ASTs), Control-Flow Graphs (CFGs), data flow, bytecode, etc. These distinct repre-sentations of code provide different levels of abstraction, which create explicit and implicitrelationships among source code elements. The importance of a specific code representa-tion is dependent upon the SE task. For example, identifiers and comments encode domainsemantics and developers’ design rationale, making them useful in the context of featurelocation [81, 228, 82, 84] and software (re)modularization [40, 39, 38]. Additionally, pro-grams that have a well-defined syntax can be represented by ASTs, which, in turn, can besuccessfully used to capture programming patterns and similarities [205, 209, 25]. Sincethere are numerous SE tasks, such as static and dynamic program analysis, change historyanalysis, automated testing, verification, program transformation, clone detection etc., itis important to rely on different available code representations so that different source coderelationships can be properly identified. Yet, many existing solutions to these SE tasksare based on “hardcoded” algorithms rooted in the underlying properties of the specific95code representation they use, which in order to work properly, also need to be adequatelyconfigured [214, 278, 215].While SE researchers regularly use machine learning (ML) or Information Retrieval(IR) techniques to solve important SE tasks, many of these techniques rely on manuallydefined or hand-crafted features. These features then allow for ML-based SE applica-tions. For example, distinct identifiers are typically used as features in concept locationapproaches [83], APIs (a reserved subset of identifiers) are used as features in approachesfor identifying similar applications [193, 269], and AST pattern similarities are used toenable clone detection and refactoring [197, 97, 134]. However, it should be noted thatall the features are selected via “an art of intuitive feature engineering” by SE researchersor domain (task) experts. While there are many successful and widely adopted ML-basedapproaches that use different code representations to support SE tasks, their performancevaries depending on the underlying datasets [214, 215].Improvements in both computational power and the amount of memory in moderncomputer architectures have enabled the development of new approaches to canonical ML-based tasks. The rise of deep learning (DL) has ushered tremendous advances in differentfields [165, 195]. These new approaches use representation learning [44] — a significantdeparture from traditional approaches — to automatically extract useful information fromthe disproportionate amount of unlabeled data in software. The value in circumventingconventional feature engineering when designing ML-based approaches to SE tasks is two-fold. Firstly, learning transformations of the data reduces the cost of modeling code, sincesoftware (and various code representations) stores a lot of data to improve the effectivenessof learning. Secondly, performance is increased because generally, learning algorithms aremore efficient than humans at discovering correlations in high dimensional spaces.While the number of applications of DL to SE tasks is growing [124, 96, 160, 206, 24,15, 160, 157, 42, 106, 275], one recent example by White et al. [283] shows that DL caneffectively replace manual feature engineering for the task of clone detection. Existing clonedetection approaches leverage algorithms working on manually specified sets of features,96and include, for example, text-based [137, 139, 138, 86, 231] string-based [34, 33, 35],token-based [141, 172, 233, 234], AST-based [290, 41, 154, 134], graph-based [152, 155,177, 97, 57], or lexicon-based [192] approaches. Instead, DL-based clone detection does notrequire any manual specification of features, and is comparable to existing clone detectionapproaches [283]. While White et al. did not manually specify any features, they did usea stream of identifiers and constant code tokens as a way to represent the code.In this chapter, we posit a fundamental question of whether an underlying code repre-sentation can be successfully used to automatically learn code similarities. In our approach,we employ the representations of identifiers, ASTs, CFGs, and Bytecode and use DL al-gorithms to automatically learn necessary features (rather than “hardcoding” algorithmsor crafting features in ML-based approaches), which in turn, can be used to support SEtasks. Moreover, we also study whether combinations of the models trained on diversecode representations can yield more accurate and robust support for the SE task at hand.Our conjecture is that each code representation can provide a different, orthogonal viewof the same code fragment, thus allowing a more reliable detection of similar fragments.Being able to learn similarities from diverse code representations can also be helpful inmany practical settings, where some representations are simply not available (e.g., whenonly compiled code is available or ASTs cannot be built due to compilation errors) or whensome of the representations are compromised (e.g., code obfuscation would prevent usingidentifier-based approaches). Many real-world SE tasks, such as third-party library detec-tion [189, 31] and code provenance [73] often deal with code having limited representationsavailable that needs to be analyzed.While we conduct our experiments on a specific SE task — clone detection — our goalis not to develop an ultimate clone-detection approach, but rather show that effective DL-based solutions can be assembled from diverse representations of source code. Specifically,the noteworthy contributions of this work are as follows:971. Deep Learning from different code representations - We demonstrate that it is in-deed possible to automatically learn code similarities from different representations,such as streams of identifiers, AST nodes, bytecode mnemonic opcodes, as well asCFGs. We evaluate how these different representations can be used in a DL-basedapproach for clone detection. We argue that our results should be applicable to anySE approach that relies on analyzing code similarities.2. Assembling a Combined model - We demonstrate that combined models can be auto-matically assembled to consider multiple representations for SE tasks (in our example,code clone detection). In fact, we show that the combined model achieves overallbetter results when compared to stand-alone code representations.3. Inter-project similarities - We also demonstrate that the proposed models can beeffectively used to compute similarities, not only in the context of a single project,but also to analyze code similarities among different (diverse) software projects (e.g.,detecting clones or libraries across multiple projects).4. Model reusability and transfer learning - We demonstrate that we can learn multi-representation models on available software systems and then effectively apply thesemodels for detecting code similarities on previously unseen systems.5. Open science - We release all the data used in our studies [7]. In particular, weinclude all the datasets, inputs/outputs, logs, settings, analysis results, and manuallyvalidated data.4.2 ApproachOur approach can be summarized as follows: First, code fragments are selected at thedifferent granularities we wish to analyze (e.g., classes, methods). Next, for each selectedfragment, we extract its four different representations (i.e., identifiers, AST, bytecode, andCFG). Code fragments are embedded as continuous valued vectors which are learned for98each representation. In order to detect similar code fragments, distances are computedamong the embeddings. Finally, we demonstrate how we effectively combine differentrepresentations for the task of code clone detection and classification.4.2.1 Code Fragments SelectionGiven a compiled code base formed by source code files and compiled classes, code frag-ments are selected at the desired level of granularity: classes or methods. We start bylisting all the .java files in the code base. For each Java file, we build the AST rootedat the CompilationUnit of the file. To do this, we rely on the Eclipse JDT APIs. Weuse the Visitor design pattern to traverse the AST and select all the classes and methodscorresponding to TypeDeclaration and MethodDeclaration nodes. We discard interfacesand abstract classes since their methods do not have a bytecode and CFG representation.While it is possible to extract the other two representations (identifiers and ASTs), wechose to discard them so that we only learned similarities from code which exhibits allfour representations. In addition, we filter out fragments (i.e., classes or methods) smallerthan ten statements. Similar thresholds have been used in previous work for minimal clonesize [279]. Furthermore, smaller repetitive snippets are defined as micro-clones. Syntacticrepetitiveness below this threshold has simply been considered uninteresting because itlacks sufficient content [37]. For each code fragment ci ∈ {Classes∪Methods} we extractits AST node ni and its fully qualified name (signature) si. Identifier and AST represen-tations are extracted from an AST node ni, while the bytecode and CFG representationsare queried using the fully qualified name si.4.2.2 Code Representation ExtractionWe used the following representations of the source code: (i) identifiers; (ii) ASTs; (iii)bytecode; and (iv) CFGs. We then performed extraction and normalization as our prepos-sessing steps for each representation. In this section we describe these two steps for eachselected representation.994.2.2.1 IdentifiersIn this representation, a code fragment is expressed as a stream (sentence) of identifiersand constants from the code. Similar representation is used by White et al. [283].Extraction. Given the code fragment ci and its corresponding AST node ni, we considerthe sub-tree rooted at the node ni. To extract the representation, we select the terminalnodes (leaves) of the sub-tree and print their values. The leaf nodes mostly correspond tothe identifiers and constants used in the code.Normalization. Given the stream of printed leaf nodes, we normalize the representa-tion by replacing the constant values with their type (< int >, < float >, < char >,< string >).4.2.2.2 ASTIn this representation, a code fragment is expressed as a stream (sentence) of the nodetypes that compose its AST.Extraction. Similarly to what was described above, the sub-tree rooted at the node niis selected for a given code fragment ci. Next, we perform a pre-order visit of the sub-treeprinting, for each node encountered, its node type.Normalization. We remove two AST node types: SimpleName and QualifiedName.These nodes refer to identifiers in the code, and were removed because: (i) they representlow-level nodes, which are less informative than high-level program construct nodes inthe AST (e.g., VariableDeclarationFragment, MethodInvocation); (ii) they account for∼46% of the AST nodes leading to a very large yet repetitive corpus; (iii) we target an ASTrepresentation able to capture orthogonal information as compared to the identifier repre-sentation. The latter is formed for∼77% of terms belonging to SimpleName/QualifiedNamenodes.1004.2.2.3 CFGIn this representation, a code fragment is expressed as it’s CFG.Extraction. To extract the CFG representation, we rely on Soot [8], a popular frame-work used by researchers and practitioners for Java code analysis. First, we extract thefully qualified name of the class from the signature si of the code fragment ci. We use itto load the compiled class in Soot. For each method in the class, the CFG G = (V,E)is extracted, where V is the set of vertices (i.e., statements in the method) and E theset of directed edges (i.e., the control flow between statements). In particular, the noderepresents the numerical ID of the statement as it appears in the method. Since the CFGis an intra-procedural representation, we see the method-level representation as a graph,while the class-level representation is a forest of graphs (CFGs of its methods).Normalization. The CFG represents code fragments at a high-level of abstraction, there-fore, no normalization is performed.4.2.2.4 BytecodeIn this representation, a code fragment is expressed as a stream (sentence) of bytecodemnemonic opcodes (e.g., iload, invokevirtual) forming the compiled code.Extraction. Let ci be the code fragment and si its signature. If ci is a method, weextract the fully qualified name of its class from si, otherwise si already represents thename of the class. Then, the bytecode representation is extracted using the commandjavap− c− private < classname > passing the fully qualified name of the compiledclass. The output is parsed, allowing for the extraction of the class- or method-levelrepresentation.Normalization. In the normalization step we remove the references to constants, keepingonly their opcodes. For example, the instruction putfield#2 is normalized as putfield.We also separate the opcodes stream of each method with the special tag < M >.1014.2.3 Embedding LearningFor each code fragment c ∈ {Classes ∪Methods}, we extract its representations: rident,rAST , rbyte, and rCFG. We learn a single embedding (i.e., vector) for each representation,obtaining: eident, eAST , ebyte, and eCFG. An embedding represents a code fragment ina multidimensional space where similarities among code fragments can be computed asdistances.We use two strategies to learn embeddings for the aforementioned representations. Foridentifier, AST and bytecode representations, we use a DL-based approach that relies onrecursive autoencoders [244, 245]. For the CFG representation we use a graph embeddingtechnique [212].4.2.3.1 DL StrategyLet C be the set of all code fragments and R the corpus comprised by the representationsof the code fragments in a representation (rident, rAST , rbyte). For each code fragmentc ∈ C, its representation r ∈ R is a sentence of the corpus R. The sentence r is a streamof words r = w1, w2, . . . , wj , where wi is a term of the particular representation (i.e., anidentifier, AST node type or bytecode opcode). The corpus is associated with a vocabularyV containing the unique terms in the corpus.To learn an embedding for a sentence r, we perform two steps. In the first stage,we learn an embedding for each term wi (i.e., word embeddings), which comprises thesentence. In the second stage, we recursively combine the word embeddings to learn anencoding for the entire sentence r. We now describe these two stages in details.In the first stage we train a Recurrent Neural Network (RtNN) on the corpus R, wherethe size of the hidden units is set to n, which corresponds to the embedding size [196]. Themodel, trained on the corpus R, generates a continuous valued vector, called an embedding,for each word wi ∈ V .102Figure 4.1: First iteration to encode a stream of tokens.The second stage involves training a Recursive Autoencoder [244, 245] to encode ar-bitrarily long streams of embeddings. Fig. 4.1 shows the recursive learning procedure.Consider the sentence r ∈ R formed by seven terms {w1, . . . , w7}. The first step maps thestream of terms to a stream of n−dimensional embeddings {x1, . . . , x7}. In the examplein Fig. 4.1, there are six pairs of adjacent terms (i.e., [xi;xi+1]). Each pair of adjacentterms [x`;xr] is encoded by performing the following steps: (i) the two n-dimensional em-beddings corresponding to the two terms, are concatenated into a single 2n-dimensionalvector x = [x`;xr] ∈ R2n; (ii) x is multiplied by a matrix ε = [ε`, εr] ∈ Rn×2n; (iii) a βiasvector βz ∈ Rn is added to the result of the multiplication; (iv) the result is passed to anonlinear vector function f : z = f (εx+ βz).The result z is an n-dimensional embedding that represents an encoding for the streamof two terms, corresponding to x. In the example in Fig. 4.1, x` and xr correspond to theembeddings x5 and x6 respectively, which in turn, correspond to the terms w5 and w6. Inthis steps the autoencoder performs dimensionality reduction. In order to assess how goodz encodes the pair [x`;xr], the autoencoder tries to reconstruct the original input x fromz in the δecoding phase. z is δecoded by multiplying it by a matrix δ = [δ`; δr] ∈ R2n×nand adding a βias vector βy ∈ R2n: y = δz + βy.103The output y = [x̂`; x̂r] ∈ R2n is referred to as the model’s reconstruction of theinput. This model θ = {ε, δ, βz, βy} is called an autoencoder, and training the model involvesmeasuring the Error between the original input vector x and the reconstruction y:E(x; θ) = ||x` − x̂`||22 + ||xr − x̂r||22 (4.1)The model is trained by minimizing Eq. (4.1). Training the model to encode streams withmore than two terms requires recursively applying the autoencoder. The recursion can beperformed following predefined recursion trees or by using optimization techniques, such asthe greedy procedure defined by Socher et al. [243]. The procedure works as follows: in thefirst iteration, each pair of adjacent terms are encoded (Fig. 4.1). The pair whose encodingyields the lowest reconstruction error (Eq. (4.1)) is the pair selected for encoding at thecurrent iteration. For example, in Fig. 4.1, each pair of adjacent terms are encoded (e.g.,dashed lines) and the pair of terms w5 and w6 is selected to be encoded first. As a result, inthe next iteration, x5 and x6 are replaced by z and the procedure repeats. Upon derivingan encoding for the entire stream, the backpropagation through structure algorithm [103]computes partial derivatives of the (global) error function w.r.t. θ. Then, the error signalis optimized using standard methods.4.2.3.2 Graph Embedding StrategyTo generate the embeddings for CFG representations we employ the graph embeddingtechnique HOPE [212] (High-Order Proximity preserved Embedding). We rely on thistechnique for two main reasons: (i) HOPE has been shown to achieve good results ingraph reconstruction [105]; (ii) unlike other techniques such as e.g., SDNe, HOPE embedsdirected graphs (as needed in the case of CFGs).Given a graph G = (V,E), HOPE generates an embedding for each node in the graph.Next, a single embedding is generated for the whole graph performing mean pooling onthe node’s embeddings. HOPE works by observing a critical property of directed graphs104known as asymmetric transitivity. This property helps to preserve the structure of directedgraphs by identifying correlations of directed edges. For example, assume three distinct,directed paths from v1 to v5 . Each of these paths increases the correlation probabilitythat there exists a directed edge from v1 to v5 , however, the lack of directed paths fromv5 to v1 decreases the probability of there being a direct edge from v5 to v1.Figure 4.2: HOPE Embedding TechniqueHOPE looks to preserve asymmetric transitivity by implementing the highly correlatedmetric known as the Katz proximity. This metric gives weights to the asymmetric transi-tivity pathways, such that they can be captured through the embedding vectors. HOPElearns two embedding vectors; the source vector and the target vector for each vertex.These vectors are then assigned values based upon the weights of the edges and their na-ture (source or target). For example, consider the graph in Fig. 4.2. Each of the solidlined edges are directed edges and the dotted edges capture the asymmetric transitivity of105the graph. The vector created for v1 as the target vector will be 0 since no paths lead to v1as their target. However, the value assigned to v5 target vector will be much higher sincemany paths end with v5 as their target. HOPE creates and learns the embeddings of thegraph by using Singular Value Decomposition for each vertex. Then, through mean poolingon the nodes embeddings, a single embedding for the entire graph can be generated.4.2.4 Detecting SimilaritiesLet E be the set of embeddings learned for all code fragments by a particular representation(i.e., Eident, EAST , Ebyte, ECFG). We compute pairwise Euclidean distances between eachand every pair of embeddings ei, ej ∈ E. The set of distances D are normalized between[0, 1] and a threshold t is applied to the distances in order to detect similar code fragments.4.2.5 Combined ModelsEach of the four models we built are trained on a single representation and identifiesa specific set of similar code fragments. Such models can be combined using EnsembleLearning, an ML paradigm where multiple learners are trained to solve the same problem.In contrast to ordinary machine learning approaches, which try to learn one hypothesisfrom training data, ensemble methods try to construct a set of hypotheses and combinethem [298].A simple class of Ensemble Learning techniques are the algebraic combiners, wheredistances computed by several, single-representation models, are combined through analgebraic expression, such as minimum, maximum, sum, mean, product, median, etc.. Forexample, a weighted average sum of the distances computed from each representation canbe computed. Formally, given two code fragments a and b, and their n representations,one can compute a dissimilarity score ds as follows:ds(a, b) =1nn∑i=1widi(a, b)106Where di(a, b) is the distance between a and b computed using the i-th representation,and wi is the weight assigned to the i-th representation. Weights can be set based on theimportance of each representation and the types of similarities we wish to detect.Single-representation models can also treated as experts and combined using voting-based methods. Each single-representation model expresses its own vote about the simi-larity of two code fragments (i.e., are similar vs are not similar) and these decisions arecombined in a single label. Different strategies can be used depending on the goal: (i)(weighted) majority of voting; (ii) at least one vote (max recall); (iii) all votes needed(max precision).The aforementioned strategies are non-trainable combiners. However, given a set ofinstances for which the oracle is available, ensemble learning techniques can be employedto perform training. Random Forest is one of the most effective ensemble learning methodfor classification, which relies on decision tree learning and bagging [47]. Bagging (i.e.,bootstrap aggregating), is a technique that generates bootstrapped replicas of the trainingdata. In particular, different training data subsets are randomly drawn with replacementfrom the entire training dataset. Random Forest, in addition to classic bagging on thetraining instances, also selects a random subset of the features (feature bagging). Eachtraining data subset is used to train a different deep decision tree. For a new instance,Random Forest averages the predictions by each decision tree, effectively reducing theoverfitting on the training sets.In the following, we show the effectiveness of combining similarities learned from dif-ferent representations training two models in the context of clone detection (i.e., a modelclassifying clone candidates as true or false positives) and classification (i.e., a model thatother than discerning clone candidates in true or false positives, also classifies the truepositives in their own clone type).Similarities from different code representations can be useful not only to detect clones,but also to classify them into types. For example, clone pairs with very high AST similarity,107but low identifiers similarity are likely to be Type-II clones (possibly parameterized clones,where all the identifiers have been systematically renamed).4.3 Experimental DesignTable 4.1: Project StatisticsProject #Classes #Methods Total LOCant-1.8.2 1,608 12,641 127,507antlr-3.4 381 3,863 47,443argouml-0.34 1,408 8,465 105,806hadoop-1.1.2 3,968 21,527 319,868hibernate-4.2.0 7,119 46,054 431,693jhotdraw-7.5.1 765 6,564 79,672lucene-4.2.0 4,629 23,769 412,996maven-3.0.5 837 5,765 65,685pmd-4.2.5 872 5,490 60,739tomcat-7.0.2 1,875 15,275 181,023The goal of this study is to investigate whether source code similarity can be learnedfrom different kinds of program representations, with the purpose of understanding whetherone can combine different representations to obtain better results, or use alternative rep-resentations when some are not available. More specifically, the study aimed to addressthe following research questions (RQ):RQ1 How effective are different representations in detecting similar code fragments?RQ2 What is the complementarity of different representations?RQ3 How effective are combined multi-representation models?RQ4 Are DL-based models applicable for detecting clones among different projects?RQ5 Can trained DL-based models be reused on different, previously unseen projects?1084.3.1 Datasets SelectionWe make use of two datasets: (i) Projects: a dataset of ten Java projects; (ii) Libraries: adataset comprised of 46 Java libraries.4.3.1.1 ProjectsThis dataset is comprised of ten compiled Java projects extracted from the Qualitas.classCorpus [256]. We rely on this dataset because it is publicly available and the projects havebeen already compiled. This (i) avoids any potential problem/inconsistency in compilingprojects, and (ii) ensures reproducibility. The selection of ten projects aimed at obtaininga diverse dataset, in terms of application domain and code size. Table 4.1 reports statisticsof the dataset that we use in all our research questions except RQ4.4.3.1.2 LibrariesThis dataset is comprised of 46 different Apache commons libraries [6]. We selected allthe Apache commons libraries, for which we were able to identify both binaries and sourcecode of the latest available release. We downloaded the compressed files for binaries andsource code. Within the binaries, we located the jar file, which represents the library, andextracted the .class files. The compressed source code files were simply decompressed.The list of considered libraries is available in our replication package. We use this datasetin RQ4 for inter-project clone detection.4.3.2 Experimental Procedure and Data AnalysisWe discuss the experimental procedure and data analysis we followed to answer eachresearch question.1094.3.2.1 RQ1:How effective are different representations in detecting similarcode fragments?Given the projects dataset P = {P1, . . . , P10} and the four code representations R ={R1, R2, R3, R4}, we extract for each code artifact c ∈ {Classes ∪ Methods} its fourrepresentations r1, r2, r3, r4. Then, for each project Pi we train the representation-specificmodel on the code representations at class-level. With the trained models, we subsequentlygenerate the embeddings for both classes and methods in the project. Therefore, thecode artifact represented as r1, r2, r3, r4 will be embedded as e1, e2, e3, e4, where ei is theembedding of the i-th representation. We set the embedding size of Identifiers, AST andBytecode to 300, while the CFG embedding size to four. The latter is significantly smallerthan the former because (i) CFGs are abstract representations which do not require largeembeddings; (ii) in order to converge towards an embedding representation, HOPE (andinternally SVD) needs the embedding size to be smaller than the minimum number ofnodes or edges in the graph. Pairwise Euclidean distances are computed for each pair ofclasses and methods of each system Pi. The smaller the distance, the more similar the twocode artifacts.In the next step, for each code representationRi, we query the clone candidates from thedataset P at class and method level. To query the clone candidates, we apply a thresholdon the distances to discern what qualifies as clones. We use the same two thresholds atclass- and method-level (Tclass = 1.00e − 08 and Tmethods = 1.00e − 16, similar to [283])for each representation. While, ideally, these thresholds should be tuned for each projectand representation, we chose the same thresholds to facilitate the comparison among thefour representations. Once we obtained the two sets of candidates CandClassesi andCandMethodsi for each representation Ri, we performed the union of the candidate sets ofthe same granularity among all the representations: CandClasses =⋃4i=1CandClassesi(the same applies for CandMethods).110For each candidate c ∈ CandClasses (or CandMethods) we generate a tuple tc ={b1, b2, b3, b4} where bi = Trueiffc ∈ CandClassi (i.e., if c is identified as a clone byRi) and bi = False otherwise. tc can assume 24 = 16 possible values (i.e., tc = {FFFF,FFFT, . . . , TTTT}). However, the combination FFFF does not appear in our datasetsince CandClasses and CandMethods are sets containing the union of the clones identifiedby all representations, thus ensuring the presence of at least one True value. Therefore,there are 15 unique classes of values for tc. We use these sets to partition the candi-dates in CandClasses and CandMethods. Next, from each candidate clones partition,we randomly draw a statistically significant sample with 95% confidence level and ±15%confidence interval. The rationale is that we want to evaluate candidates belonging todifferent levels of agreement among the representations.Three authors independently evaluated the clone candidate samples. The evaluatorsdecided whether the candidates were true or false positives and, in the former case, the clonetype. To support consistent evaluations, we adapted the taxonomy of editing scenariosdesigned by Svajlenko et al. [255] to model clone creation and to be general enoughto apply to any granularity level. Given the manually labeled dataset from each of thethree evaluators, we computed the two-judges agreement to obtain the final dataset (e.g.,a candidate clone was marked as a true positive if at least two of the three evaluatorsclassified it as such). In order to statistically assess the evaluators agreement, we computethe Fleiss’ kappa [93]. In particular, we compute the agreement for True and False positiveas well as the agreement in terms of Clone Types. On the final dataset, precision andrecall were computed for each candidate clones partition (e.g., TFFF , FTFF , etc.) and,overall, for each representation in isolation. We quantitatively and qualitatively discussTP/FP pairs for each representation, as well as the distribution of clone types.4.3.2.2 RQ2:What is the complementarity of different representations?To answer RQ2 we further analyze the data obtained in RQ1 to investigate the complemen-tarity of the four representations. First, we compute the intersection and the difference of111the true positive sets identified with each representation. Precisely, the intersection anddifference of two representations Ri and Rj are defined as follows:Ri ∩Rj =|TPRi ∩ TPRj ||TPRi ∪ TPRj |% and Ri \Rj =|TPRi \ TPRj ||TPRi ∪ TPRj |%where TPRi represents true positive candidates identified by Ri. We compute the percent-age of candidates exclusively identified by a single representation and missed by all theothers:EXC(Ri) =|TPRi \⋃j 6=i TPRj ||⋃j TPRj |%Then, to understand whether these code representations are orthogonal to each other,we collect all the distance values of each representation computed for each pair of codefragments (classes or methods) in the dataset. Given these distance values, we computethe Spearman Rank Correlation [249] between each pair of representations, to investigatethe extent to which distances computed from different representations on the same pairsof artifacts correlate.4.3.2.3 RQ3:How effective are combined multi-representation models?To answer RQ3 we evaluate the effectiveness of two combined models: CloneDetector,which classifies candidate clones as true/false positives, and CloneClassifier, which providesinformation about the type of detected clone (Type-1, 2, 3 or 4). While we are aware thatis its possible to determine a clone type by comparing the sequence of tokens, the purposeof this classification is to show the potential of the proposed approach to provide completeinformation about a clone pair. The two models are trained using the manually labeleddataset obtained in RQ1, with the distances computed by each representation used asfeatures and the manual label used as target for the prediction. To train the two models werely on Random Forest employing the commonly used 10-fold cross-validation technique topartition training and test sets. We evaluate the effectiveness of the two models computing112Precision, Recall and F-Measure for each class to predict (e.g., clone vs not-clone for theCloneDetector).4.3.2.4 RQ4:Are DL-based models applicable for detecting clones among dif-ferent projects?The goal of RQ4 is two-fold. On the one hand, we want to instantiate our approach in arealistic usage scenario in which only one code representation is available. On the otherhand, we also want to show that the DL based model can be used to identify inter-projectclones. Indeed, the latter is one of the major limitations of the work by White et al. [283],where given the potentially large vocabulary of identified-based corpora, the approach wasevaluated only to detect intra-project clones.We instantiate two usage scenarios both relying on the same Library dataset, and ontraining performed on the binaries (bytecode from .class files). In the first scenario, asoftware maintainer has to analyze the amount of duplicated code across projects belongingto their organization. We use the entire Library dataset and filter for inter-project clonecandidates only (i.e., candidates belonging to different projects). Clone candidates areevaluated by inspecting the corresponding Java files from the downloaded code.In the second scenario, a software developer is using a jar file (i.e., compiled library)in their project. The developer has no information about other libraries that could beredistributed with the jar file, since only compiled code is present. For provenance and/orlicensing issues, the developer needs to address whether the jar file j imports/shadows anyof the libraries in a given dataset L.To perform this study we select weaver-1.3 as the jar j and the remaining 45 librariesin the Library dataset as L. We identify similar classes between j and L. Then, toassess whether the identified classes have actually been imported in j from library x ∈ L,we downloaded the j’s source code and analyzed the building files (weaver-1.3 relies onMaven, thus we investigated the .pom files) to check whether the library x is imported asa dependency in j.1134.3.2.5 RQ5:Can trained DL-based models be reused on different, previouslyunseen projects?One of the major drawbacks of DL-based models is their long training time. This trainingtime could be amortized if these models we able to be reused across different projectsbelonging to different domains. The major factor that hinders the reusability of suchmodels is the possible variability in the vocabulary for new, unseen projects. For example,a language model and a recursive autoencoder trained on a given vocabulary would notbe able to provide adequate results on a vocabulary containing terms not previously seenduring the training phase. Such a vocabulary should be cleaned, either by stripping off theunknown terms, replacing them with special words (e.g., < unkw >) or using smoothingtechniques. These solutions negatively impact the performance of the models.While in principle, with enough training data and available time, any of the aforemen-tioned representation-specific models could be reused on a different dataset (i.e., unseenproject), in practice some representation-specific models are more suitable than others forreuse. In particular, models trained on representations with a limited or fixed vocabularyare easier to be reused on different projects. In our study, AST and Bytecode represen-tations both have a fixed vocabulary, limited respectively by the number of different ASTnode types and bytecode opcodes.To answer RQ5 we perform a study aimed at evaluating the effectiveness of reusing ASTmodels. We evaluate the effectiveness, showing that a reused model identifies a similar listof clone candidates as compared to the original model trained on the set projects. We selectthe AST representation which was trained on one of the biggest project in the dataset,RQ1 (i.e., lucene). We use this AST model to generate the embeddings for the remainingnine projects in the dataset. Using the generated embeddings we compute the distancesand query the clone candidates using the same class- and method-level thresholds used inRQ1. Then, let LR and LO be the lists of candidates returned from the reused model and114from the original model, we compute the percentage of candidates in LR that are in LOand vice versa.We also show that the combined model CloneDetector, trained on clone candidatesbelonging to a single project (hibernate), can be effectively used to detect clones in theremaining nine systems of the Projects dataset.4.4 ResultsRQ1: How effective are different representations in detecting similar code frag-ments? Table 4.2 shows the results in terms of precision for different candidate clone par-titions, where each clone partition includes clones detected only by a given combinationof representations. For example, the first partition FFFT represents the clone candidatesidentified by the Bytecode representation, but not by the other three representations. Notethat for the partition with ID = 10 (TFTF ) no class-level clones have been identified.Table 4.2: Performances for different clone partitionsID Iden AST CFG Byte Precision %Methods Classes1 F F F T 5 492 F F T F 9 583 F F T T 88 734 F T F F 79 635 F T F T 95 936 F T T F 100 1007 F T T T 100 1008 T F F F 95 1009 T F F T 100 10010 T F T F 100 -11 T F T T 100 10012 T T F F 100 10013 T T F T 100 10014 T T T F 100 10015 T T T T 100 100115Table 4.2 shows that most of the partitions exhibit a good precision, with peaks of100%. The notable exceptions are the partitions with ID 1 and 2, referring to clonecandidates detected only by the Bytecode and by the CFG representations, respectively.We investigated such false positives and qualitatively discuss them later.Table 4.3 shows the overall results for method- and class-level aggregated by repre-sentation. The table contains the raw count of True Positives (TP), False Positives (FP)and clone types identified by each representation. Estimated precision and recall is alsoreported in the last two columns of the tables. Note that the results of the single represen-tation Ri (e.g., Identifier) reported in 4.3 refer to aggregated candidates of table 4.2 wherethe representation Ri is the True (e.g., clone partitions ID 8-15).Table 4.3: Performances for different representationsMethodsRepresentation FP TP Type I Type II Type III Type IV Precision RecallIden 1 201 151 15 35 0 100% 52%AST 11 292 138 132 19 3 96% 75%CFG 43 178 69 81 19 9 81% 46%Byte 46 222 89 77 49 7 83% 57%ClassesRepresentation FP TP Type I Type II Type III Type IV Precision RecallIden 0 120 23 51 46 0 100% 40%AST 18 188 18 121 44 5 91% 63%CFG 24 120 7 65 41 7 83% 40%Byte 34 217 23 115 77 2 86% 73%The Identifier-based models achieve the best precision, both when working at method-and class-level. AST-based models provide the best balance between precision and recall.In terms of types of clones, Identifier-based models detect the highest number of Type I,AST-based models identify the highest number of Type II, Bytecode models identify thehighest number of Type III, and CFG models detect the highest number of Type IV. Thissuggests that the four representations are complementary, and will be addressed further inRQ2.116The data presented in this section is based on the agreement of three evaluators. TheFleiss’ kappa shows substantial agreement in terms of TP/FP (93% for methods and 65%for classes) and a moderate/substantial agreement in terms of clone types classification(75% for methods and 57% for classes). In the following breakdown, we discuss the resultsachieved by models using different representations.Identifiers. The identifiers-based model achieves the best precision both in method-and class-level results. However its estimated recall is respectively 52% and 40%, meaningthat the model fails to detect a significant percentage of clones. While the recall couldbe increased by appropriately raising the threshold (i.e., by adopting a more permissivethreshold) at the expense of precision, we found that this would still not be enough to detectmost of the clones detected by other representations. Indeed, we manually investigatedseveral TP clone candidates identified by other representations and missed by the Identifier-based model. We found that the distances computed by the Identifier-based model forsuch clones were several orders of magnitude higher than a reasonable threshold (e.g., theones we use or the ones used by White et al. [283]). Moreover, a close inspection at thevocabulary of the candidates showed that they share a small percentage of identifiers, whichin turn, makes them hard to detect with such a representation. Clearly, clone candidateswhere pervasive renaming or obfuscation has been performed, would be very difficult todetect with this model.Bytecode & CFG. From Table 4.2, we can notice that candidates detected only byBytecode or only by CFG models (partitions with ID 1 and 2) tend to be false positives.The precision for such partitions is 5% and 9% at method-level and 49% and 58% at class-level. However, when both Bytecode and CFG models detect clones not detected by theAST and Identifier-based models (partition with ID 3), they achieve reasonable levels ofprecision (88% and 73%).Bytecode and CFG representations have a very high degree of abstraction. Within theCFG, a statement is simply represented by a node. Similarly, the bytecode is formed by117low level instructions, which do not retain the lexical information present in the code (e.g.,a method call is represented as a invokevirtual or invokestatic opcode). Such a levelof abstraction appears to be imprecise for fine granularities such as methods, where thecode fragments might have similar structure but, at the same time, perform very differenttasks. Better results are achieved at class-level, where false positives are mainly due to JavaBeans, thus classes having a very similar structure, i.e., class attributes with getters andsetters acting on them performing similar low-level operations (storing/loading a variable,creating an object, etc.). While these classes are false positives in terms of code clones,they still represent a successful classification in terms of learning structural and low-levelsimilarities from the code. The general lesson learned is that the use bytecode and CFG ina combined model yield an acceptable precision. Indeed, if bytecode is available (compiledcode) CFGs can be extracted as well.Table 4.3 also shows that Bytecode and CFG are the representations that detect themost Type IV clones in our dataset. Again, this is likely due to their high level of abstrac-tion in representing the code.AST. AST-based models seem to have the best overall balance between precision andrecall. These models can identify similar code fragments, even when their set of identifiers(i.e., vocabulary) is significantly different and share almost no lexical tokens. This hasbeen confirmed by our manual investigation of the candidates. We report several examplesof candidates identified exclusively by the AST model in our online appendix.RQ2: What is the complementarity of different representations? RQ2:What is the complementarity of different representations? Table 4.4 reports thecomplementarity metrics between the different representations. In particular, on the leftside of table 4.4 we report the intersection of sets of true positive candidates detected bythe four representations. For example, 40% of the true positives are detected by both theAST and Identifier models. The relatively small overlap among the candidate sets, at both118method and class granularity (no more than 51%), suggests that these representationscomplement each other.Table 4.4: Complementarity MetricsMethodsIntersection % Difference % Exclusive %R1 ∩R2 Iden AST CFG Byte R1 \R2 Iden AST CFG Byte Ri EXC(Ri)Iden 40 21 36 Iden 17 43 29 Iden 5% (21)AST 42 44 AST 43 46 38 AST 9% (33)CFG 36 CFG 36 12 24 CFG 1% (4)Byte Byte 35 18 39 Byte 1% (2)ClassesIntersection % Difference % Exclusive %R1 ∩R2 Iden AST CFG Byte R1 \R2 Iden AST CFG Byte Ri EXC(Ri)Iden 33 14 42 Iden 19 43 8 Iden 3% (8)AST 31 51 AST 48 49 19 AST 9% (26)CFG 34 CFG 43 20 14 CFG 7% (21)Byte Byte 49 30 52 Byte 7% (21)The middle section of table 4.4 shows the difference in the sets of true positive can-didates detected by the four representations. The reported values show the percentageof true positive clones detected by a certain representation and missed by the other. Forexample, 48% of the true positive clones identified by the AST model at class level are notidentified by the Identifier-based model.Finally, the left section of table 4.4 shows the percentage and number of instances (inparenthesis) of true positive candidates identified by each representation and missed by allthe others. From these results, we note that there are candidates exclusively identified bya single representation. Note that the number of instances and percentages are computedonly on the manually validated sample.Table 4.5 shows the Spearman’s Rank correlation coefficient (ρ) for the different rep-resentations. While all the computed correlations are statistically significant, their lowvalue suggests that distances computed with different representations do not correlate andprovide different perspectives about the similarity of code pairs. The strongest correlation119Table 4.5: Spearman’s rank correlationρ(R1, R2) Ident AST CFG ByteIdent 0.094 0.120 0.069AST 0.157 0.031CFG 0.046Bytewe observe is between the AST and the CFG representations, which is still limited to 0.157(basically, no correlation).For example, the classes MessageDestination and ContextEjb were detected only bythe AST representation. They both extend ResourceBase and offer the same functional-ities. These classes share only a few identifiers (therefore not detected by the Identifiermodel) and differ in some if-structures, making them more difficult to detect by CFG andBytecode representations. This and other examples are available in our online appendix [7].RQ3:How effective are combined multi-representation models? Table 4.6 re-ports the results of the CloneDetector at class- and method-levels. When identifying cloneinstances, the model achieves 90% precision and 93% recall at class, and 98% precision97% recall at method level. The lower performance at class-level could be due to thesmaller dataset available (i.e., 483 methods vs 362 classes). The overall classification per-formance (i.e., the weighted average of the Clone/Not Clone categories) is also lower forthe class-level (∼85% F-Measure for classes, and ∼96% for methods).We also trained the model by considering a subset of the representations. In our onlineappendix, we provide results considering all possible subsets of features. We found thatsingle-representation models tend to have worse performance than the combined model,which was trained on all representations. However, the combinations with Identifiers+AST+{CFGor Bytecode} obtain results similar to the overall model trained on all representations.Table 4.7 shows the results of the CloneClassifier at class- and method-levels. At classlevel, the CloneClassifier has overall F-Measure of 68%, with average precision of 67% andrecall of 68%. At method level, the model obtains an overall F-Measure of 84%, with anaverage precision and recall of 84% across the different categories (i.e., Not Clone and the120Table 4.6: Performance of the CloneDetectorMethods ClassesPrecision % Recall % F-Measure % Precision % Recall % F-Measure %Clone 98 97 98 90 93 91Not Clone 90 91 90 61 52 56Weighted Avg. 96 96 96 85 86 85four types of detected clones). As expected, the category with the lowest precision andrecall is the Type IV clones for both method and class level, while other clone types achievea precision ≥ 74% and a recall ≥ 75%. Single-representation models obtain significantlyworse results in the classification task (with a bigger gap with respect to what observedfor the CloneDetector). All the results are available in our online appendix.Table 4.7: Performance of the CloneClassifierMethods ClassesPrecision % Recall % F-Measure % Precision % Recall % F-Measure %Not Clone 89 94 91 59 61 60Type I 89 88 88 86 78 82Type II 82 84 83 81 85 83Type III 74 75 75 61 59 60Type IV 67 18 29 00 00 00Weighted Avg. 84 84 84 67 68 68RQ4:Are DL-based models applicable for detecting clones among differ-ent projects? We identified several groups of duplicate code across different Apachecommons libraries in the dataset. The largest group of clones is between the librarieslang3-3.6 and text-1.1. The duplicated code involves classes operating on Strings, forexample: StringMatcher, StrBuilderWriter, StrTokenizer, StrSubstitutor etc., for a totalof 21 shared similar classes identified. We also identified another group of similar classesbetween text-1.1 (package diff) and collections4-4.1 (package sequence). In partic-ular, the class StringsComparator in text-1.1 appears to be a Type III clone of the classSequencesComparator in collections4-4.1. An example of Type II clone within thesetwo libraries is instead the pair EditScript and ReplacementsFinder classes. These clonesare classified as Type II, since only package information has been changed.121The libraries math3-3.6.1 and rng-1.0 contain two shared classes: ISAACRandomand Gamma/InternalGamma. ISACCRandom is a pseudo-random number generator. Theclasses share many similarities across the two libraries, however, they differ on a statementand structural level. As an example, both classes implement a method which sets a seed.In one class, the seed is set by an integer passed through an argument, while the otherclass sets the seed by using a combination of the current time and system hash codeof the instance. Gamma and InternalGamma, named respectively to the library theybelong to, are clones since parts of the Gamma class were used to develop InternalGamma.The developers only took the needed functionalities out of the Gamma class, strippedit of unnecessary code, and built InternalGamma. Therefore, many of the methods inGamma either do not appear or are significantly smaller in InternalGamma. DespiteInternalGamma being significantly smaller than Gamma, our tool was still able to detectsimilarity between the two classes.The libraries codec-1.9 and net-3.6 share the same implementation for the classBase64, providing encoding/decoding features.Note that the classes mentioned in this study do not refer to imported libraries but toactual Java duplicated code (i.e., the source code files are in both libraries).We also identified false positives in this study, mainly due to small inner classes andenumerators. Enumerators have a very similar bytecode structure even if containing dif-ferent constants. They are uninteresting with respect to the goal of this scenario.Imported and shaded classes. We identified a large list of shared classes between thelibrary j (weaver-1.3) and the following libraries in the dataset L: collections4-4.1(373 classes), lang3-3.6 (79), and io-2.5 (13). A closer inspection of the building filesof weaver-1.3 showed that the aforementioned libraries have been imported and shaded.That is, the dependencies have been included and relocated in a different package name inorder to create a private copy that weaver-1.3 bundles alongside its own code.RQ5:Can trained DL-based models be reused on different, previously unseenprojects? Table 4.8 shows the percentage of candidates in LR that are also in LO and vice122versa, both at method- and class-level. Generally, the list of candidates identified by thereused model and the original models tend to be similar. At method-level, we can see that97% of the candidates identified by the reused model were also identified by the originalmodel. Similarly, 93% of the candidates returned by the original model are identified bythe reused model. At class-level we notice smaller percentages. This is mostly due tothe fact that fewer clones are identified at the class-level. For example, for antlr-3.4the reused model identifies three candidates while the original model only identifies one.For maven-3.0.5, two candidates are identified by the reused model and only one by theoriginal model. Still, 90% of the class-level candidates identified by the original models aredetected by the reused model.Table 4.8: Model ReusabilityProject Methods % Classes %LR ∈ LO LO ∈ LR LR ∈ LO LO ∈ LRant-1.8.2 99 88 73 31antlr-3.4 100 100 33 100argouml-0.34 99 96 97 73hadoop-1.1.2 99 95 95 74hibernate-4.2.0 89 82 30 84jhotdraw-7.5.1 99 98 82 77maven-3.0.5 97 84 50 100pmd-4.2.5 97 99 99 99tomcat-7.0.2 98 97 87 69Overall 97 93 58 90We also show that combined models can be reused on different systems. The CloneDe-tector model has been trained only on the data available for one project (hibernate) andtested on all the instances of the remaining projects. It achieved 98% precision and 92%recall at method-level and 99% precision and 95% recall at class-level.1234.5 Threats To ValidityConstruct validity. The main threat is related to how we assess the complementarityof the code representations. We mitigate this claim performing different analyses: (i)complementarity metrics; and (ii) correlation tests.Internal validity. This is related to possible subjectiveness when evaluating similaritiesof code fragments. To mitigate such threat, we employed three evaluators who indepen-dently checked the candidates. Then, we computed two-judge agreement on the evaluatedcandidates. We also qualitatively discuss false positives and borderline cases. Also, all ofour evaluations are publicly available [7].External validity. The results obtained in our study using the selected datasets mightnot generalize to other projects. To mitigate this threat, we applied our approach in differ-ent contexts and used two different datasets; Projects and Libraries. For Projects, which isa subset of systems from the Qualitas.class corpus, we selected diverse systems in terms ofsize and domain, focusing on popular ones. All the Libraries studied in this work, whichare comprised of all the Apache commons libraries, are publicly available to ensure thereplicability of the study. We did not utilize other clone-focused datasets (e.g., such asBigCloneBench [254]) because compiled code is required in order to extract the represen-tations of CFGs and Bytecode. Another threat in this category is related to the fact thatwe apply our approach on Java code only. While the representation extraction steps areimplemented for Java, all the subsequent steps are completely language-agnostic becausethey rely on a corpus of sentences with arbitrary tokens. In fact, Recursive Autoencodershave been used in several contexts with different inputs such as natural language, sourcecode, images etc. We do not compare our approach against code clone detection techniquessince the focus of this paper is to show a general technique on how to learn similaritiesfrom multiple code representations, rather than building a code clone detector tool. Last,but not least, we focused on four code representations, but there may be others that areworthwhile to investigate (e.g., data-flow or program dependency graphs).1244.6 ConclusionIn this chapter, we show that code similarities can be learned from diverse representationsof the code, such as Identifiers, ASTs, CFGs and bytecode. We evaluated the performanceof each representation for detecting code clones and show that such representations areorthogonal and complement each other. We also show that our model is reusable, therefore,eliminating the need for retraining the DL approach so that it is project specific. Thiseliminates a large timesink, native to DL approaches, and broadens the applicability ofour approach.Moreover, combined models relying on multiple representations can be effective incode clone detection and classification. Additionally, we show that Bytecode and CFGrepresentations can be used for library provenance and code maintainability. These findingsspeak to the vast amount of SE tasks which benefit from analyzing multiple representationsof the code. We instantiated our approach in different use case scenarios and datasets.Our approach inherently highlights the benefits of not only single code representations,but the combinations of these representations. This work also begins to emphasize theattributes that different code representations can accentuate. This allows for a more tar-geted choice by SE researchers of a code representation when applying representation-DLalgorithms to a SE tasks. We believe that learning similarities from different represen-tations of the code, without manually specifying features, has broad implications in SEtasks, and is not limited solely to clone detection.125Chapter 5Deep Learning Assert Statements5.1 IntroductionWriting high-quality software tests is a difficult and time-consuming task. To help tamethe complexity of testing, ideally, development teams should follow the prescriptions of thetest automation pyramid [68], which suggests first writing unit tests that evaluate small,functionally discrete portions of code to spot specific implementation issues and quicklyidentify regressions during software evolution. Despite their usefulness, prior work has il-lustrated that once a project reaches a certain complexity, incorporating unit tests requiresa substantial effort in traceability, decreasing the likelihood of unit test additions [150].Further challenges exist for updating existing unit tests during software evolution andmaintenance [150].To help address these issues the software testing research community has respondedwith a wealth of research that aims to help developers by automatically generating tests [95,213]. However, recent work has pointed to several limitations of these automation toolsand questioned their ability to adequately meet the software testing needs of industrialdevelopers [26, 239]. For example, it has been found that the assert statements generatedby state-of-the-art approaches are often incomplete or lacking the necessary complexity tocapture a designated fault. The generation of meaningful assert statements is one126of the key challenges in automatic test case generation. Assert statements providecrucial logic checks in a test case to ensure that the program is functioning properly andproducing expected results. However, writing or generating effective assert statements isa complex problem that requires knowledge pertaining to the purpose of a particular unittest and the functionality of the related production code. Thus, an effective technique forthe generation of assert statements requires predicting both the type and logical nature ofthe required check, using source and test code as contextual clues for prediction.To help advance techniques that aid developers in writing or generating unit tests,we designed Atlas, an approach for automatically generating syntactically and semanti-cally correct unit test assert statements using Neural Machine Translation (NMT). Atlasgenerates models trained on large-scale datasets of source code to accurately predict as-sert statements within test methods. We take advantage of the deep learning strategy ofNMT, which has become an important tool for supporting software-related tasks such asbug-fixing [262, 63, 194, 121], code changes [260], code migration [208, 207], code sum-marization [163, 185, 291], pseudo-code generation [210], code deobfuscation [268, 133]and mutation analysis [265]. To the best of our knowledge, this is the first empirical steptoward evaluating an NMT-based approach for the automatic generation of assert state-ments. Specifically, we embed a test method along with the context of its focal method[222] (i.e., a declared method, within the production code, whose functionality is testedby a particular assert statement) and we "translate" this input into an appropriate assertstatement. Since our model only requires the test method and the focal method, we areable to aid developers in automatic assert generation even if the project suffers from a lackof initial testing infrastructure. Note that our approach is not an alternative to automatictest case generation techniques [95, 213], but rather, a complementary technique that canbe combined with them to improve their effectiveness. In other words, the automatic testcase generation tools can be used to create the test method and our approach can help indefining a meaningful assert statement for it.127To train Atlas, we mined GitHub for every Java project making use of the JUnit assertclass. In total we analyzed over 9k projects to extract 2,502,623 examples of developer-written assert statements being used within test methods. This data was used to give theNMT model the ability to generate assert statements that closely resemble those createdby developers. Therefore, not only do we enable efficiency within the software testingphase but we also facilitate accuracy and naturalness by learning from manually writtenassert statements. After we extracted the pertinent test methods containing assert state-ments from the Java projects, we automatically identified the focal method for each assertstatement based on the intuition from Qusef et al. [222]. We hypothesize that combiningthe test method and the focal method should provide the model with enough context toautomatically generate meaningful asserts.We then quantitatively and qualitatively evaluated our NMT model to validate its use-fulness for developers. For our quantitative analysis, we compared the models generatedassert statements with the oracle assert statements manually written by developers. Weconsidered the model successful if it was able to predict an assert statement which is identi-cal to the developer-written one. Our results indicate that Atlas is able to automaticallygenerate asserts that are identical to the ones manually written by developers in 31.42%of cases (4,968 perfectly predicted assert statements) when only considering the top-1 pre-dicted assert. When looking at the top-5 recommendations, this percentage rises to 49.69%(7,857).For our qualitative analysis we analyzed “imperfect predictions” (i.e., predictions whichcan differ semantically or syntactically as compared to the assert manually written bydevelopers) to understand whether they could be considered an acceptable alternative tothe original assert. We found this to be true in the 10% of cases we analyzed. Finally, wecomputed the edit distance between the imperfect predictions and original asserts in orderto assess the effort required for a developer to adapt a recommended assert statement intoone she would use. We show that slight changes to the “imperfect” asserts can easily convert128them into a useful recommendation. To summarize, this paper provides the followingcontributions:• We introduce Atlas, a NMT-based approach for automatically generating assertstatements. We provide details pertaining to the mining, synthesizing, and pre-processing techniques to extract test methods from the wild, and to train and testAtlas;• An empirical analysis of Atlas and its ability to use NMT to accurately generate asemantically and syntactically correct assert statement for a given test method;• A quantitative evaluation of the model, and a detailed comparison between modelingraw and abstracted test methods;• A publicly available replication package [9] containing the source code, model, toolsand datasets discussed in this paper.5.2 ApproachWe provide an overview of the Atlas workflow for learning assert statements via NMT inFig. 5.1. Our approach begins with the mining and extraction of test methods from Javaprojects. To do this, we mine GitHub projects that use the JUnit testing framework (Sec.5.2.1). From those projects, we mine all test methods denoted with the @Test annotationas well as every declared method within the project (Sec. 5.2.2). We then filter this data,identify the appropriate focal method context, and generate pairs containing the contextualtest method (i.e., the test method augmented with information about the focal method ittests) and the relevant assert statement (Sec. 5.2.3 & Sec. 5.2.4). We refer to these pairsas Test-Assert Pairs (TAPs). Next, we generate two datasets of TAPs: (i) Raw SourceCode, where TAPs are simply tokenized; (ii) Abstract Code, where we abstract the TAPsthrough our abstraction process (Sec. 5.2.5). Finally, we train two RNN encoder-decoder129GitHubTest MethodsDatasetsMine JUnit test methods21Spoon FrameworkFocal MethodsCreate TestMethod Context3Filtering4Abstraction5Raw SourceCodeAbstractCodeRNNs (Encoder-Decoder)CopyAhtxtAhtxtA!n.AhtxtAhtxt6 7Figure 5.1: Overview of ATLAS Workflowmodels; one using the copy mechanism trained on the Raw Source Code TAPs, and anotherusing only the attention mechanism trained on the Abstract Code TAPs (Sec. 5.2.6).5.2.1 GitHub MiningOur main motivation toward studying Java projects that use the JUnit framework is appli-cability. As of August 2019 the TIOBE Programming Community Index indicated Java asthe most popular programming language [3]. In addition, a study done by Oracle in 2018,found that JUnit was the most popular Java library [221]. Hence, curating a dataset ofprojects that use Java and JUnit lends to the potential for impact on real-world softwaredevelopment.We identified GitHub projects using the JUnit testing framework. Since client projectscan use JUnit by declaring a dependency through Apache Maven [4], we started by using130the GitHub search API to identify all Java projects in GitHub having at least one pom fileneeded to declare dependencies toward Maven libraries. This resulted in the identificationof 17,659 client projects, using 118,626 pom files and declaring ∼1.1M dependencies intotal. We downloaded all the identified pom files and mined them to identify all clientprojects declaring a dependency toward JUnit version 4 and all its minor releases. Thesedependencies can be easily identified by looking in the pom file for artifacts having thejunit groupId, junit artifactId, and a version starting with “4.”. Using this process, wecollected a total of 9,275 projects. Note that we decided to focus on JUnit v.4 since, in themined dataset of pom files, we found that the majority of them had a dependency towardsthis version.5.2.2 Method ExtractionAfter mining these projects from GitHub, we downloaded the source code and extracted therelevant test methods using Spoon [218]. This framework allows for source code analysisthrough the creation of a meta-model where the user can access program elements. Toaccess relevant methods, we extract methods beginning with the @Test annotation, whichis inherent to the JUnit framework. After extracting the test methods, we extract everymethod declared within the project, excluding methods from third party libraries. Theextracted methods comprise a pool, from which we can determine the focal method ofinterest for a particular test method. The reason we only consider methods declaredwithin the project is two-fold. First, most assert statements are evaluating the internalinformation of the project itself rather than information taken from third party librariesor external packages. Second, it would require a substantial effort to retrieve the methodbodies and signatures from all the third party libraries and external packages. Since ourgoal is to learn appropriate assert statements for a given test method and its context,any test method without an assert statement has been discarded. Also, since this is thefirst work in the literature applying NMT to automatically generate assert statements, wedecided to focus on test methods having a single assert statement and, thus, we exclude131those implementing multiple asserts. While we acknowledge that this is a simplification ofthe problem we tackle, we preferred to first investigate the “potential” usefulness of NMTin a well-defined scenario in which, for example, it is safe to assume that the whole testmethod provides contextual information for the unique assert statement it contains. Thisassumption is not valid in the case of multiple asserts, and instead requires the developmentof techniques which are able to link parts of the test method body to the different assertsin order to understand the relevant context for each of them. This is part of our futurework. Overall, we collected 188,154 test methods with a single assert statement.5.2.3 Identifying Focal MethodsOur next task is to identify the focal method that the assert statement, within the testmethod, is testing. To accomplish this we implement a heuristic inspired by Qusef et al.[222]. We begin by extracting every method called within the test method. The list ofinvoked methods is then queried against the previously extracted list of methods definedinside the project, considering the complete method signature. We then assume that thelast method call before the assert is the focal method of the assert statement [222]. In someinstances, the assert statement contains the method call within its parameters. In thesecases, we consider the method call within the assertion parameters as the focal method.It may appear problematic that we use the line we attempt to predict in order to extractthe focal method (since, in theory, the line to generate should not exist). However, in areal usage scenario, we assume that the developer can provide the focal method to ourmodel (i.e., she knows the method she wants to test). Since identifying the focal methodmanually for a large number of assert statements is unreasonable, we used the heuristicpreviously described in place of manual identification of the focal method. We note this asa limitation but find it reasonable that either a developer or an automated test generationstrategy would provide this information to our approach.1325.2.4 FilteringIn this work, we are attempting to generate semantically and syntactically correct assertstatements from the test method and focal method context. Thus, we are creating a modelwhich must learn relationships from source code. Modeling this type of data presentscertain challenges, such as the open vocabulary problem and the “length of the input”problem [151]. Usually, these problems are tackled by limiting the vocabulary and the inputlength so that the model can adequately learn [252]. We employ similar solutions whentraining our model. We filter the data in three distinct ways: i) excluding test methodslonger than 1,000 tokens; ii) filtering test methods that contain an assert statement whichrequires the synthesis of unknown tokens; and iii) removing duplicate examples within thedataset. Every filtering step helps to address NMT-related challenges and have been used inprevious approaches that take advantage of this deep learning based strategy [151, 63, 262].Our first filtering step is fairly straightforward: we remove all test methods that exceed1,000 tokens. The second filtering step removes test methods in which the appropriateassert statement requires the synthesis of one or more unknown tokens. This means thatthe syntactically and semantically correct assert statement requires a token that cannot befound in the vocabulary or in the contextual method (i.e., test method + focal method).Indeed, there is no way to synthesize these tokens when the model attempts to generatea prediction. We further explain this problem as well as our developed solution in section5.2.4.1. Lastly, our third filtering step aims at removing duplicated instances, ensuringthat every contextual method and assert statement pair in our dataset is unique.5.2.4.1 VocabularyWe have alluded to the open vocabulary problem which is an inherent limitation of NMT.This issue arises because developers are not limited in the number of unique tokens they canuse. They are not limited to, for example, English vocabulary, but also create “new” tokensby combining existing words (e.g., by using the CamelCase notation) or inventing new133words to comprise identifiers (e.g., V0_new). For this reason, the source code vocabularyfrequently needs to be artificially truncated. To deal with this problem, we studied thetokens distribution in our dataset, observing that it follows Zipf’s law, as also found byprevious work analyzing open source software lexicons [220]. This means that the dataset’stokens follow a power-law like distribution with a long tail, and that many of the TAPs inour dataset can be successfully represented by only considering a small subset of its 695,433unique tokens. Based on analyzing the data and on previous findings in the literature [62],we decided to limit our vocabulary to the 1,000 most frequent tokens. This allows us tosuccessfully represent all tokens for 41.5% of the TAPs in our dataset (i.e., 204,317 out of491,649).This filtering step aimed at removing instances for which the model would need togenerate an unknown token. This can be formally defined as follows: Given a contex-tual method (i.e., the test method tm and the corresponding focal method fm), weremove TAPs for which the anticipated assert to generate contains a token t such thatt /∈ { Vglobal∪Vtm∪Vfm}, where V represents the vocabulary. We refer to Vtm∪Vfm as thecontextual method. Each filtering step was due to concrete limitations of state-of-the-artNMT models. In numbers, starting from ∼ 750 thousand test methods having a singleassert, ∼ 2.5 thousand tests are removed due to their excessive length, and ∼ 280 thousanddue to unknown tokens. Thus, ∼ 37% of the single assert test methods are removed.5.2.5 Test-Assert Pairs and AbstractionThe next step of our approach consists of preparing the data in such a manner that it canbe provided as input to Atlas’s NMT model. This process involves: i) the concatenationof the focal method to the test method in order to create the Test-Assert Pair (TAP);ii) the tokenization of the TAP; and iii) the abstraction of the TAP. Starting from thefirst point, it is important to note that not every test method will inherently contain afocal method being tested. However, for the test methods that do posses a focal method,we append its signature and body to the end of the test method. While comments and134whitespaces are ignored, we do not remove any other token from either the test or focalmethod. We then proceed to remove the entire assert statement from the test method,replacing it with the unique token "AssertPlaceHolder". Therefore, the first part of aTAP consists of the concatenated test and focal method, and the second part consists ofthe assert statement to generate.We generate two separate datasets of TAPs. The first uses the raw source code torepresent the test and the focal method. The second dataset consists of abstracted TAPs,in which the source code tokens are abstracted to limit the vocabulary and to increasethe possibility of observing recurring patterns in the data. As previous studies have shown[262], this can lead to an increase in performance without losing the ability to automaticallymap back the abstracted tokens to the raw source code.Our abstraction process tokenizes the input, determines the type for each individualtoken, replaces the raw token with its abstraction and finally creates a map from theabstracted tokens back to the raw ones. Each TAP is abstracted in isolation and has noaffect on the way other TAPs are abstracted. We start by using the javalang tool totokenize the input, which allows us to analyze the type of token. This tool transforms theJava method into a stream of tokens, which is then parsed by the same tool to determinetheir type (e.g., whether a token represents an identifier, a method declaration, etc.) [257].We use these types in order to create an expressive representation that maintains thestructure of the code. In total, there are 13 different types of tokens that we use as part ofour abstraction process (complete list in our replication package [9]). When we encounterone of these types, we replace the raw source code token with an abstraction term thatindicates the type and number of occurrences of that type of token up to that moment.In other words, we use a numerical value to differentiate tokens of the same type. Forexample, suppose we encounter a token that is determined to be a method call. This tokenis replaced with the term METHOD_0, since this is the first type of that token we haveencountered. The next time we encounter a new method call in the same stream of tokens,it would be assigned the term METHOD_1. In the event where the same token appears135multiple times within the TAP, it is given the same abstraction term and numerical value.This means that if the same method is invoked twice in the test or in the focal method andit is represented with the term METHOD_0, the abstraction will contain “METHOD_0”twice. Since all TAPs are abstracted in isolation, we can reuse these terms when abstractinga new TAP, thus limiting the vocabulary size for our NMT model.Test method + focal methodAbstracted test method + focal method Abstracted assert statement Abstracted test method + focal method with idiomsAbstracted assert statement with idiomsshouldCreateRedMana () { mage.Mana mana = mage.Mana.RedMana (1); “” ;} getRed () { return red ; }org.junit.Assert.assertEquals(2, mana.getRed())METHOD_0 () { IDENT_0.IDENT_1 IDENT_2 OP_0 IDENT_0.IDENT_1.METHOD_1 (INT_0); STRING_0; } METHOD_3 () { return IDENT_3 ;}METHOD_0 () { IDENT_0.IDENT_1 IDENT_2 OP_0 IDENT_0.IDENT_1.METHOD_1 (1); STRING_0; } METHOD_3 () { return IDENT_3 ;}IDENT_4.IDENT_5.IDENT_6.METHOD_2(INT_1, IDENT_2.METHOD_3())org.junit.Assert.assertEquals(2, IDENT_2.METHOD_3())Assert statementLearning ProcessFigure 5.2: Overview of the abstraction processIn addition to the abstraction terms that replace the raw source code tokens, we takeadvantage of idioms in order to create an abstraction that captures more semantic infor-mation. The inclusion of idioms can also help contextualize surrounding tokens, since someabstracted tokens may be more likely to appear around a particular idiom. In addition,idioms help to prevent the exclusion of a TAP due to the synthesis of an unknown token.For instance, consider the example of abstraction shown in figure 5.2. In the middle ofthe figure it is possible to see that INT_1, IDENT_4, IDENT_5 and IDENT_6 onlyappear in the abstracted assert statement, but do not appear in the abstracted test andfocal method. If we only relied on the abstraction, we would be unable to resolve thesetokens that are unique to the predicted assert statement. Therefore, we keep the raw value136of common idioms (i.e., the top 1,000 tokens in our dataset in terms of frequency) in ourabstracted representation, as shown in the bottom part of figure 5.2.Overall, the vocabulary of the Abstract TAPs comprises 1,000 idioms plus∼100 typifiedIDs, while the vocabulary of the Raw Source Code TAPs contains 1,000 tokens.5.2.6 Sequence to Sequence LearningOur approach applies sequence-to-sequence learning through a recurrent neural network(RNN) encoder-decoder model to automatically learn assert statements within test meth-ods. This model is inspired by Chen et al. [63], which attempts to predict a single lineof code that has a predetermined place holder within the method. The goal of this deeplearning strategy is to learn a conditional distribution of a variable length sequence condi-tioned on a completely separate variable length sequence P (y1, y2, . . . , ym|x1, x2, . . . , xn).Where n and m may differ. During training, the model’s encoder is fed the tokenizedinput sequence of the test method plus the context of the focal method as a single streamof tokens (x1, . . . , xn). The assert statement, which is our target output sequence, hasbeen removed and replaced with a specialized token. The decoder attempts to accuratelypredict the assert (y1, . . . , ym) by minimizing the error between the decoder’s generatedassert and the oracle assert statement. This is accomplished by using the negative loglikelihood of the target tokens using stochastic gradient descent [144]. An overview of anRNN encoder-decoder can be seen in figure 5.1.5.2.7 EncoderThe encoder is a single layer bi-directional RNN, which is comprised of two distinct LSTMRNNs. This bidirectionality allows the encoder to consider both the tokens that comebefore and the tokens that come after as context for the token of interest. The encodertakes a variable length sequence of source code tokens X = (x1, x2, . . . , xn) as input. Fromthese tokens, the encoder produces a sequence of hidden states (h1, h2, . . . , hn) generatedfrom LSTM RNN cells. These cells perform a series of operations to propagate relevant137h1 h2 hnRNN Cell (LSTM)x1 x2 y1 ymcis1 s2 smRNN Cell (LSTM)[ci, si]So!maxtest/focal method assert statementEncoder RNN A"ention Decoder RNN............Copy MechanismFigure 5.3: The RNN bidirectional encoder decoder modelinformation, including previous hidden states, to the next cell. Due to the bidirectionalityof the encoder, there exists a sequence of hidden states when considering the token sequencefrom left to right−→hi = f(xi, hi−1) and right to left←−hi = f(xi0 , hi0+1). For our model,each hidden state can be formally described as the non-linear activation of the currentsequence token and the previously synthesized hidden state. Once the hidden state foreach directional pass is found, they are concatenated to derive the finalized hidden statehi. The sequence of resulting hidden states is propagated through the model as the contextvector. The encoder also applies the regularization technique of dropout at a rate of 0.2.5.2.8 Attention MechanismThe context vector C, commonly referred to as an attention mechanism, is computed as aweighted average of the hidden states from the encoder C =∑ni=1 αihi. In this equation αrepresents a vector of weights used to denote the influence of different parts of the inputsequence. In this manner, the model can pay greater attention to particular tokens ofthe input sequence when attempting to predict the output token yi. The weights which138influence the attention mechanism are trained over time to help identify the patterns ofcontribution from different input tokens.5.2.9 Decoder and Copy MechanismThe decoder is a double layer LSTM RNN that learns to take a fixed length contextvector and translates it into a variable length sequence of output tokens. Given a previoushidden state hî−1, the previous predicted token yî−1 and the context vector C the decodergenerates a new hidden state that can be used to predict the next output token. As donefor the encoder, we apply regularization to the decoder by using dropout at a rate of 0.20,and the Adam optimizer for learning with a starting learning rate of 0.0001:hî = f(hî−1, yî−1, C)The decoder generates a new hidden state each time it predicts the next token in thesequence until a special stop token is reached. The hidden states are generated using theequation above. However, these hidden states are also used by the copy mechanism tohelp predict the appropriate output token. In particular, the copy mechanism works tocalculate two separate probabilities. The first, is the probability that the next predictedtoken in the output sequence should be taken from the vocabulary. The second, is theprobability that the next predicted token in the output sequence should be copied fromthe input sequence. With the ability to consider copying tokens from the input sequenceto the output sequence, we can artificially extend the vocabulary of the encoder-decorderRNN model for the raw dataset. Each sequence inferred from the model now has the abilityto consider any predetermined vocabulary token, in addition to any token from the inputsequence. The downside is that the copy mechanism is a trainable extension of the model,that learns which input tokens should be copied over. The benefit is that the model canbetter deal with rare tokens that would otherwise not appear in the vocabulary.139Test method + focal methodshouldParseVersion() {subject.parse(“--version”); “”;} isPrintVersionMode(){return printVersionMode;}org.junit.Assert.assertTrue(subject.isPrintVersionMode())Assert statementVocabularyv1v3v5v11< subject, assertTrue, assert, org>< Assert, ), “--version”, junit >< parse, ), assertEquals, Assert >< Assert, junit, assert, UNK >...... ...... Copied from test methodCopied from vocabularyFigure 5.4: Copy Mechanism ExampleTo further demonstrate how the copy mechanism works, consider figure 5.4. In thisexample, we see the vocabulary tokens predicted for each time step vt. For the purpose ofthis example we do not consider the translation of the separator tokens. However, in a realscenario, our model would predict these tokens in identical fashion. The first predictedtoken for the output sequence is the org token, which is copied from the vocabulary. Thisprocess is repeated for the junit token, the Assert token and would continue for all furthertokens but the last one (i.e., isPrintVersionMode). The last token is not found anywherein the vocabulary. At time step v11 the model recommends the UNK token to indicate thatthe highest probability token does not exist within the defined vocabulary. In this case,the copy mechanism is used to determine which input token has the highest probabilityto be the predicted token. In the case shown in the example, the context appended fromthe focal method contains this token and it is copied to the output sequence. Without thecopy mechanism, there would have been no way to resolve this example and it would havebeen discarded. It is important to note that the copy mechanism is trained along with thenetwork and requires no effort on the end users.Also, note that the copy mechanism is only applied to the raw dataset of source codetokens. The reasoning is that after abstracting the source code, all tokens are availablewithin the vocabulary. Therefore, the probability that a predicted token would be outsideof the vocabulary, and within the input sequence, is 0%, rendering the copy mechanism140useless. Rather, our abstraction process serves as a pseudo copy mechanism of typifiedIDs. Consider the previous example in figure 5.2 where isPrintVersionMode was notfound anywhere in the vocabulary. In our abstraction process, this token is abstractedinto METHOD_1 and since METHOD_1 is a term contained within our vocabulary themodel has no problem in predicting this token in the output assert statement. Then, whenwe map the abstracted assert statement back to raw source code, we replace METHOD_1with the token isPrintVersionMode without relying on the copy mechanism.5.3 Experimental DesignThe goal of our study is to determine if Atlas can generate meaningful, syntactically andsemantically correct assert statements for a given test method and focal method context.Additionally, it is important for our approach to be lightweight in order to require as littleoverhead as possible if, for example, it is combined with approaches for the automaticgeneration of test cases.The context of our study is represented by a dataset of 158,096 TAPs for the abstracteddataset and 188,154 TAPs for the raw dataset. These datasets are further broken downinto 126,477 TAPs for training, 15,809 TAPs for validation, and 15,810 TAPs for testingin the abstract dataset. Likewise, we had 150,523 TAPs for training, 18,816 TAPs forvalidation and 18,815 TAPs for testing in our raw dataset. The differences in number ofexamples between the two datasets is due exclusively to the removal of duplicates. Sincethe abstracted model reuses typified IDs, there is a greater chance for the duplicationof TAPs within the abstracted dataset. Our evaluation aims at answering the researchquestions described in the following paragraphs.RQ1: Is ATLAS able to generate assert statements resembling those man-ually written by developers? We assess whether Atlas is a viable solution for gen-erating semantically and syntactically correct assert statements. Therefore, we performexperiments on real-world test methods and determine if our model can predict the appro-141priate assert statement. We use both datasets (i.e., raw source code and abstracted code)to train the encoder-decoder recurrent neural network model. During training, we use ourvalidation set to determine the optimal parameterization of the NMT model (complete listof used parameters available in [9]). We then evaluate the trained model on the test set,which contains examples previously unseen in both the training set and the validation set.We begin training our model on TAPs, feeding the model the test method and associ-ated focal method context. We train our model until the evaluation on the validation setshows that the models parameterization has reached a (near-)optimal state (i.e., the modelis no longer improving the calculated loss for data points outside the training set). This isa common practice to prevent the effects of overfitting to the training data. Our trainingphase results in two separate models, one for predicting raw source code assert statementsand the other for predicting abstracted asserts. Remember that when working with rawsource code, we also implement the copy mechanism, which is not used for abstractedcode. In total, the abstract model trained for 34 hours while the raw model trained for38 hours. The difference in training time can be attributed to the use and training of thecopy mechanism in conjunction with the lack of abstraction.Once the model is trained, inference is performed using beam search [226]. The mainintuition behind beam search decoding is that rather than predicting at each time step thetoken with the best probability, the decoding process keeps track of k hypotheses (withk being the beam size). Thus, for a given input (i.e., test method + focal method), themodel will produce k examples of assert statements. We experiment with beam sizes goingfrom k = 1 to k = 50 at steps of 5.Given the assert statements predicted by our approach, we consider a prediction ascorrect if it is identical to the one manually written by developers for the test methodprovided as input. We refer to these asserts as “perfect predictions”. When experimentingwith different beam sizes, we check whether a perfect prediction exists within the k gener-ated solutions. We report the raw counts and percentages associated with the number ofperfect predictions.142Note that, while the perfect predictions certainly represent cases of success for our ap-proach, this does not imply that the “imperfect predictions” all represent failure cases (i.e.,the generated asserts are not meaningful). Indeed, for the same test/focal method, differ-ent assert statements could represent a valid solution. Therefore, we sample 100 “imperfectpredictions” and manually analyze them to understand whether, while different from theoriginal assert statements written by the developer, they still represent a meaningful pre-diction for the given test method. In particular, we split the “imperfect predictions” intofour sets based on their BLEU-4 score [217] value. The BLEU score is a well-known metricfor assessing the quality of text automatically translated from one language to another[217]. In our case, the two “languages” are represented by i) the test method and the focalmethod, and ii) the generated assert statement. We use the BLEU-4 variant, meaningthat the BLEU score is computed by considering the 4-grams in the generated text, aspreviously done in other software-related tasks in the literature [109, 135]. The BLEUscore ranges between 0% and 100%, with 100% indicating, in our case, that the generatedassert is identical to the reference one (i.e., the one manually written by developers). Weuse the BLEU score ranges 0-24, 25-49, 50-74 and 75-99 to split the imperfect predictions.Then, we randomly selected 25 instances from each set and the first author manually eval-uated them to determine if the generated assert statement is meaningful in the contextof the related test/focal methods. To avoid subjectiveness issues, the 100 instances werealso randomly assigned to four other authors (25 each) who acted as second evaluator foreach instance. Conflicts (i.e., cases in which one of the two evaluators classified the assertstatement as meaningful while the other did not) arose in a single case that was solvedthrough an open discussion. We report the number of meaningful assert statements wefound in the manually analyzed sample as empirical evidence that imperfect assert state-ments could still be useful in certain cases. Note that, while there might be authors’ bias inassessing the meaningfulness of the “imperfect” assert statements (i.e., authors may tendto be too positive in evaluating the meaningfulness of the asserts), we make our evalua-143tion publicly available in the replication package [9], to allow the reader to analyze theperformed classification.RQ2: Which types of assert statements is ATLAS capable of generating?After obtaining the perfect predictions from RQ1, we analyze the types of assert statementsour model can generate. In particular, we analyze the taxonomy of assert statementsgenerated by our approach in the context of perfect predictions to determine the types ofassert statements the model is able to correctly predict. We then report the raw countsand the percentages for each type of assert statement the model can perfectly predict.Note for this evaluation, we only report results for k = 1, since we found that alreadywith this beam size the model was able to generate all types of assert statements in theused dataset.RQ3: Does the abstraction process aid in the prediction of meaningful assertstatements? Remember that while our abstraction model generates abstracted asserts,we can map them back to the raw source code at no cost to the developer. Thus, we cancheck whether the generated assert is a perfect prediction, as we do for the raw sourcecode. Besides comparing the performance of the two models, we also analyze whether theyproduce complementary results by computing the following overlap metrics:ppR∩A =|ppR ∩ ppA||ppR ∪ ppA|ppR\A =|ppR \ ppA||ppR ∪ ppA|ppA\R =|ppA \ ppR||ppR ∪ ppA|The formulas above use the following metrics: ppR (ppA) represents the set of perfectpredictions generated using the raw (abstracted) source code dataset; ppR∩A measures theoverlap between the set of perfect predictions generated by using the two datasets; ppR\Ameasures the perfect predictions generated on the raw source code but not when usingabstraction (vice versa for ppA\R).RQ4: What is the effect of using the copy mechanism in our model? With theaddition of the copy mechanism we can perfectly predict assert statements which contain144tokens only found in the input sequence and not in the vocabulary. Here we want toquantify the effect of the copy mechanism and see how many assert predictions we wouldbe capable of producing without its usage. Therefore, we analyze the perfect predictionset from the raw source code model. If the perfect prediction contains a token not found inthe vocabulary, then we know that its generation was possible due to the usage of the copymechanism. We report the raw counts and percentages of the number of assert statementsthat were resolved thanks to the use of the copy mechanism.RQ5: Does ATLAS outperform a baseline, frequency-based approach? Asoutput of RQ2 we defined a taxonomy of assert statements that our approach is able tocorrectly generate in the perfect predictions. We noted that there are eight types of assertstatements that the model is capable of generating. However, the model also predicts thevariables and method calls contained within the assert statement. Therefore, we want todetermine if the most frequently used assert statements found within our dataset (e.g.,assert(true)), could be used to create a frequency-based approach that outperforms ourlearning-based approach.We analyze the duplicated assert statements generated in our perfect prediction set.This means that the model generated the same (correct) assert statement for differenttest methods provided as input. It is important to highlight here that while the sameassert statement can be used in different test methods, this does not imply that we haveduplications in our datasets. Indeed, while the same assert statement can be used indifferent TAPs, the test/focal methods are guaranteed to be unique.The developed frequency-based approach takes the most commonly found assert state-ments and applies them as a solution for a given test method. In particular, we take thetop k most frequent assert statements and test if any of them represents a viable solutionfor each test method in the test set. In this evaluation we set the same k for both ap-proaches (i.e., the frequency-based and the learning-based ones). For example, assumingk = 5, this means that for the frequency-based approach we use the five most frequent145assert statements as predictions, while for the learning-based approach we set beam sizeto five. We report the raw counts of the frequency-based approach as compared to ourNMT-based approach. We compare the two approaches at k = 1, 5, 10.RQ6: What is the inference time of the model? Our last research question speaksto the applicability and ease of use of our model. When developing test cases within asoftware project, it is unreasonable to expect the developer to spend a considerable amountof time to set up and run an inference of the model. Therefore, we performed a timinganalysis to assess the time needed to generate assert statements for a variety of beam sizes.In particular, we used k = 1 to k = 50 with an increment of 5 to test the trade off betweenthe timing of the model’s inference and the increased prediction results of the model.Table 5.1: Prediction ClassificationBeam Size Raw Model Abstract ModelPeferct PredictionPercentagePerfect PredictionCountsPerfect PredictionPercentagePerfect PredictionCounts1 17.66% 3323 31.42% 49685 23.33% 4390 49.69% 785710 24.73% 4654 55.73% 881215 25.53% 4805 58.76% 929120 25.88% 4871 60.43% 955425 26.19% 4929 61.75% 976430 26.43% 4973 62.73% 991835 26.63% 5012 63.68% 1006840 26.81% 5045 64.38% 1017945 26.91% 5064 64.81% 1024750 27.01% 5083 65.31% 10327We record the results in number of seconds and map the increased performance againstthe increased time. We do not consider the time it may take for a developer to look atall resulting predictions of assert statements for different beam sizes. Note that we do notconsider the training time since this is a one time cost that does not affect the usability ofthe approach.1465.4 ResultsRQ1 & RQ3 & RQ4: Ability to generate meaningful assert statements, compar-ison between raw and abstracted dataset, and usefulness of copy mechanism.Table 5.1 shows the perfect prediction rate for both the “raw model” and the “abstractmodel” for beam sizes 1 and 5-50 at increments of 5 (RQ1 & RQ3). As expected, theperfect prediction rate increases using larger beam sizes, with a plateau reached at beamsize 20 (i.e., only minor increases in performance are observed for larger beam sizes).When using beam size equal to 1 for the model trained/tested with the raw dataset,our approach generates 17.66% perfect predictions, resulting in over 3.3k correctly gen-erated assert statements. The average BLEU score for the asserts predicted when onlyconsidering the top recommendation (i.e., beam size = 1) is 61.85. We also found that thecopy mechanism helps the raw model in generating perfect predictions (RQ4). Indeed, wedetermined how many perfect predictions require the use of the copy mechanism, findingthat, when using beam size equals 1, our approach is able to perfectly predict 3323 exam-ples by using the copy mechanism, and 2439 when only relying on the vocabulary. Thismeans that the copy mechanism is responsible for resolving 884 perfect predictions, whichconstitutes 4.69% of the perfect prediction rate.For the abstract model, the percentage of perfect predictions goes up to 31.42% (∼5kcorrectly generated asserts). Here the average BLEU score is 68.01. Note that we mentionthe BLEU scores for completeness, but do not perform a full evaluation using this metric.Of particular note is the substantial bump in perfect predictions that we obtain as resultof increasing the beam size to five for both models, especially for the abstracted one. Forthe latter, in 49.69% of test/focal methods provided as input, one of the generated assertsis identical to the one manually written by the developer (for a total of 7.9k perfectlypredicted asserts). This indicates the potential of our approach in an automatic “codecompletion” scenario, in which the developer could easily pick one of the five recommendedasserts.14702550Edit OperationsRaw Code02550Edit OperationsRaw Code Abstracted CodeFigure 5.5: Edit Distance between Imperfect Predictions and Ground Truth (Truncatedtail of higher edit distances)As mentioned in section 5.3, we also analyze the complementarity of the perfect pre-dictions generated by the raw and the abstracted models when using beam size equals1. In terms of overlap (i.e., ppR∩A) we found that only 117 examples were captured andperfectly predicted by both models. Also, 3206 (39.2%) of the perfect predictions are onlygenerated by the raw model (ppR\A), while 4851 (59.3%) can only be obtained by using theabstracted model (ppA\R). This shows a surprising complementarity of the two models,that are able to learn different patterns in the data. The combination of these two differentrepresentations, for example by using a model with two encoders (one per representation)and a single decoder, may be an interesting direction for future work.However, the overall higher performance in both BLEU score and perfect prediction rateof the abstract model shows that it is better able to translate a meaningful assert statementfrom the test and focal method context. This is likely due to the different way in which thetwo models see the out-of-vocabulary tokens. In the raw dataset they are all represented asunknown tokens, while in the abstracted dataset they are represented with the associatedtype (e.g., METHOD, VAR, INT, etc.). When dealing with out-of-vocabulary tokens, thishelps the abstracted model in learning patterns such as: a "METHOD" token is likely to148Perfect Prediction Generated by the Abstract Model 1. METHOD_0 ( ) { org . IDENT_0 . IDENT_1 . conf . IDENT_2 IDENT_3 = new org . IDENT_0 . IDENT_1 . conf . METHOD_1 ( ) ; IDENT_3 . setName ( org . IDENT_0 . IDENT_1 . conf . IDENT_4 . IDENT_5 ) ; "" ; } getName ( ) { return name ; } 2. org . junit . Assert . assertEquals ( org . IDENT_0 . IDENT_1 . conf . IDENT_4 . IDENT_5 , IDENT_3 . getName ( ) ) Perfect Prediction Unabstracted 3. testName ( ) { org . kefirsf . bb . conf . NamedValue namedValue = new org . kefirsf . bb . conf . NamedValue ( ) ; namedValue . setName ( org . kefirsf . bb . conf . NamedValueTest . NAME ) ; "" ; } getName ( ) { return name ; } 4. org . junit . Assert . assertEquals ( org . kefirsf . bb . conf . NamedValueTest . NAME , namedValue . getName ( ) ) Figure 5.6: Example of a Perfect Prediction Made by the Abstract Model, and its unab-stracted mapping to actual codefollow a "VAR" and a "." token. For the raw model, this only results in observing “UNK .UNK”, hindering its ability to learn meaningful patterns in these cases. Due to the betterperformance ensured by the abstract model, the subsequent analyses are performed usingthis model.Here, we discuss an interesting example generated by our abstracted model. Figure 5.6presents the abstracted contextual method in line 1 and the abstracted assert statementin line 2. Lines 3 and 4 are the result of the unabstraction process, where we map backthe abstracted tokens into raw source code. In this example, the test method is creating anew object namedValue and setting the NAME attribute of this object. The test methodcontains the method call getName, which our heuristic identifies as the focal method andis appended to the end of the test method. The model then generates an assert statementthat compares the NAME attribute of the new object with the results from the getNamemethod call. Indeed, the model learns the relationship between the test and focal methodin order to generate a meaningful assert statement which appropriate tests the method’slogic. This assert statement is a nontrivial example of the model determining what type ofassert statement to generate, as well as the appropriate parameters the assert statementshould contain.Concerning the manual inspection of the 100 “imperfect predictions”, we found that 10%of them can represent a valuable assert statement for the provided contextual method, de-spite them being different from the asserts manually written by developers. Note that,while a 10% might look like a “negative” result, it is important to understand that theseresults are in addition to the already perfectly predicted assert statements. The full eval-uation of the 100 cases is available in our replication package [9]. Here we discuss a149representative example, for a provided test/focal method, where our approach generatedthe assert statement assertSame(input,result) while the assert manually written by thedeveloper was assertEquals("Blablabla",result). The input object is present in thetest method, and contains indeed the value "Blablabla". Basically, our model generated an“easier to maintain” assert, since changes to the value assigned to the input objects do notrequire changes to the assert statement. Concerning the difference between assertSameand assertEquals, the former compares two objects using the == operator, while the latteruses the equals method that, if not overridden, does also perform the comparison usingthe == operator.Although we have shown that our model can produce meaningful results outside ofthe perfect predictions, we wanted to understand “how far” are the imperfect predictionsfrom the manually written assert statements. Therefore, for each imperfect predictiongenerated by the abstract model with k = 1, we computed the number of tokens one needsto change, add, or delete to convert it into the manually written assert. We found that byonly changing one token it is possible to convert 23.62% of the imperfect assert statements(i.e., 3660 instances) into perfect predictions. Also, the median of 3 indicates that in halfof the cases, changing only three tokens would be sufficient. Note that the average numberof tokens in the generated assert statements is 17.1. Figure 5.5 shows the distribution ofedit changes needed for the imperfect predictions to become perfect predictions.150Summary for RQ1, RQ3, & RQ4. Atlas’s abstracted model is able to perfectlypredict assert statements (according to a developer written ground truth) 31.42% and49.69% of the time for top-1 and top-5 predictions respectively. Conversely, the modeloperating on raw source code with the copy mechanism achieved a perfect predictionrate of 17.66% and 23.33% for top-1 and top-5 predictions respectively. This indicatesthe abstracted model performs better overall. However, we also found that models hada relatively high degree of orthogonality, with 39.2% of all perfect predictions generatedby the raw model, and 59.3% generated by the abstracted model, illustrating that thecopy-mechanism allowed for the prediction of a unique set of assert statements.RQ2: Which types of assert statements is ATLAS capable of generating?We also analyzed the types of JUnit4 assert statements that were perfectly predicted.Here we use the 4,968 perfect predictions generated using the abstract model and beamsize equals one. Table 5.2 shows that our approach was able to correctly predict eightdifferent types of assert statements, with assertEquals and assertTrue being the mostcommonly predicted type of assert statement. Note that some JUnit4 assert types (e.g.,assertNotSame and assertFail) are not generated by our model because they were notpresent in our dataset.Table 5.2: Types of Predicted Assert StatementsAssert Type Count Total in DatasetassertEquals 2518 7923assertTrue 973 2817assertNotNull 606 1175assertThat 250 1449assertNull 238 802assertFalse 232 1017assertArrayEquals 102 311assertSame 49 314As it can be seen in table 5.2, the distribution of assert statements we are able to predictis similar to the one of the assert statements in the entire dataset. This result mitigates thepossible threat that our approach is only successful in generating a specific type of assert151Table 5.3: Learning Based vs. Frequency BasedBeam Size Number of Perfect PredictionsFrequency Model Abstract Model1 455 49685 970 785710 1299 8812statement. Indeed, as shown in table 5.2, the lack of a uniform distribution of data pointsin the dataset from which Atlas is learning seems to be the main reason for the skeweddistribution of correctly predicted asserts. The main exception to this trend is representedby the assertThat statements. We hypothesize that assertThat statements are moredifficult to predict due to the nature of the assert itself. These types of asserts compare avalue with a matcher statement, which can be quite complex, since matcher statements canbe negated, combined, or customized. Despite the complexities of assertThat statements,the model is still able to perfectly predict 17.2% of the ones seen in the testing set.Summary for RQ2. We found that assertEquals was the most common type ofassert generated, matching the distribution we were learning from. Our analysis showedthat Atlas is capable of learning every type of assert statement found in developerwritten test cases.RQ5: Does ATLAS outperform a baseline, frequency-based approach? Inthis research question we explore whether our abstract model outperforms a baseline,frequency-based approach (see section 5.3). Table 5.3 shows the results of this comparison.We note that our DL-based approach outperforms the frequency based approach at eachexperimented beam size. The difference in terms of performance is substantial, resultingin 6.8 (beam size=10) to 10.9 (beam size=1) times more perfect predictions generated byour approach. For example, when only considering the top candidate assert statementfor both techniques, Atlas correctly predicts 4968 assert statements, as compared tothe 455 of the frequency-based model. The achieved results indicate that Atlas is infact learning relationships based on hierarchical features and not being overwhelmed byrepetitive assert statements. We also want to note that Atlas encompassed a majority of152the assert statements found by the frequency based baseline. Therefore, we do not combinea frequency based approach with Atlas and believe our implementation to be superior onits own.Summary for RQ5. Atlas is able to significantly outperform a frequency-basedbaseline prediction technique by a factor of 10.0.250.200.150.100Beam Size0 5 10 15 20 25 30 35 40 45 50Time in SecondsFigure 5.7: Seconds per Assert GenerationRQ6: What is the inference time of the model? Our last research question as-sesses the time required by our approach to generate meaningful assert statements. Giventhe previously discussed performance of the experimented models, we computed the gen-eration time on the abstract model. It is important to note the reported time does notinclude the code abstraction nor the mapping of the prediction back to source code, al-though these operations are typically more efficient than inference. Figure 5.7 shows theincrease in time based on the number of solutions the model generates. These timingsconcern the generation of assert statements for 15,810 test/focal methods provided as in-put. The reported time is in seconds per provided input, and includes the generation of all153assert statements in the given beam size. For example, assuming a beam size of 5, it wouldtake around 0.14 seconds to generate all 5 predictions for a particular test/focal method.These results were computed using a single consumer-grade NVIDIA RTX GPU.Summary for RQ6. We found that with a beam size of 5, we can generate all predictionsin 0.14 seconds per test method + focal method pairing. When increasing beam size, wefind that the time needed to generate assert statements appears to scale linearly.5.5 ThreatsConstruct validity threats concern the relationship between theory and observation, andare mainly related to sources of imprecision in our analyses. We automatically mined andparsed the TAPs used in our study. In this process, the main source of noise is representedby the heuristic we used to identify the focal method for a given assert statement. As said,in a real usage scenario, this information could be provided by the developer who wrotethe test or by the automatic test case generation tool. Despite the presence of introducednoise, our approach was still able to generate meaningful asserts, confirming the robustnessof our NMT model.Internal validity threats concern factors internal to our study that could influence ourresults. The performance of our approach depends on the hyperparameter configuration,that we report in our online replication package [9]. However, given how computationally-expensive the hyperparameter search was, we did not investigate the impact of the copymechanism across all configurations.External validity threats concern the generalizability of our findings. We did notcompare ATLAS with state-of-the-art test case generation techniques using the heuristicsdescribed in section 5.2 to define appropriate asserts. This comparison would requirea manual evaluation of the correctness of the asserts generated by ATLAS and by thecompetitive techniques for automatically generated tests, likely on software of which we154have little knowledge (assuming Open-Source projects). Furthermore, we would have tomake the subject systems executable (e.g., as required by EvoSuite) which is known to bedifficult. Hence, this would not allow us to scale such an experiment to the size of ourcurrent evaluation. Indeed, our main goal was to empirically investigate the feasibility ofa learning-based approach for assert statement generation. Comparing/combining the twofamilies of approaches is part of our future work. Finally, we only focused on Java programsand the JUnit framework. However, ATLAS’s learning process is language-independentand its NMT infrastructure can easily be ported to other programming languages.5.6 ConclusionIn this work, we mined over 9k GitHub projects to identify test methods with their relatedassert statements. We then used a heuristic to identify the focal method associated withan assert statement and generated a dataset of Test-Assert Pairs (TAPs), composed byi) an input sequence of the test and focal method, and ii) a target sequence reportingthe appropriate assert statement for the input sequence. We used these TAPs to create aNMT-based approach, called ATLAS, capable of learning semantically and syntacticallycorrect asserts given the context of the test and focal methods.We found that the model was capable of successfully predicting over 31% of the assertstatements developers wrote by only using the top-ranked prediction. When looking atthe top-5 predictions the percentage of correctly generated assert statements grew up to∼50%. We also showed that among the “imperfect predictions”, meaning the scenario inwhich ATLAS generates assert statements different from the ones manually written bydevelopers, there are assert statements that either represent a plausible alternative the theone written by the developer or can be converted into the latter by just modifying a fewtokens (≤ 3 in 50% of cases). Finally, some of the limitations of our approach, as well asthe extensive empirical study we conducted, provide us with a number of lessons learnedthat can drive future research in the field:155Raw code vs. Abstracted code : Our results show that through the abstraction mech-anism, applications of NMT on code can ensure better performance, as already observedin previous studies [260, 265, 262]. More interestingly, we found that the two code repre-sentations are quite complementary, and allow our approach to generate different sets of“perfect predictions”. This points to the possibility of combining the two representationsinto a single model that could benefit from an architecture having two encoders (one perrepresentation) and a single decoder.On the possibility of generating multiple assert statements: We investigated thisresearch direction while working on ATLAS. The main problem we faced was the automaticidentification of the part of test method body that it is relevant for a given assert. Indeed,while with a single assert we can assume that the whole method body is relevant, this isnot the case when dealing with multiple asserts. Here, a possible heuristic could be toconsider the statements preceding the assert statement a1, but coming after the previousassert a0, as relevant for a1. However, it is unlikely that the statements coming beforea0 are totally irrelevant for a1. Another possibility we considered was to apply backwardslicing on each assert statement but, unfortunately, this resulted in scalability issues and in(well-known) problems related to the automatic compilation of open source projects [259].Approaching this problem is a compelling direction for future work.Integrating a learning-based approach in tools for automatic test case genera-tion : As discussed earlier, we foresee two possible usages for ATLAS. First, it can be usedas a code completion mechanism when manually writing unit tests in the IDE. Second, itcould be combined with existing tools for automatic test case generation [94, 213, 5]. Wecurrently lack empirical evidence to substantiate any claim on the effectiveness of ATLASin improving automatically generated tests, which would be an important first step priorto integration. Additionally, before combining ATLAS with existing tools, it is necessaryto deeply understand the cases in which the two families of approaches (i.e., the ones inte-grated in the test case generation tool and the learning-based one) succeed and fail. In thisway, learning-based approaches could be used only when needed (i.e., when the standard156approach implemented in the test case generation tools is likely to fail), thus increasing theeffectiveness of the generated tests. Studying and comparing the strengths and weaknessof the two families of techniques is part of our future research agenda.The automatic generation of meaningful assert statements is a compelling problemwithin the software engineering community. We showed that a learning-based approachcould help aid in this problem, and opened a complementary research direction to the onealready adopted in automatic test case generation tools [94, 213, 5].157Chapter 6Conclusions and Future WorkIn this dissertation, we presented two distinct DL based approaches for the task of detectingsimilar pieces of source code and the automatic generation of assert statements for testmethods. We then presented a systematic literature review, which explores the vast arrayof DL solutions for a variety of SE tasks. The goal of this work is to show the practicalapplications of DL in SE, but also to consider the future of DL applications and thepotential of applying SE methodologies to these complex models.We begin this dissertation by focusing on the summarization of the use of DL withinSE. We follow the guidelines set for SLRs by Kitchenham et al. and systematically gatherresearch that implements a DL based approach for a SE task. We then build a taxonomyof the research gathered for the SE tasks addressed, the SE artifacts mined and processedfor DL, and the DL architectures used in the approaches. Throughout our review, wefind a number of papers related to the idea of DL but do not fall within the scope of ourstudy. Although our taxonomy focuses on papers that implement a DL based approach, wetake note of the interesting and emerging field of SE for DL. In many ways, these papersexamine the faulty nature of using DL based algorithms without fully understanding howthey work. As our survey showed, a number of important SE tasks have DL based solutionsapplied to them. As a community, we have subjected ourselves to results based analysisof these models and overlook their shortcomings when applying them to SE problems.158Therefore, we provide guidelines and instructions for important components of learning toinclude and report on. We also provide a clear road map for future directions in DL4SEresearch. In the future, we hope the SE community not only looks to evolve and optimizethe use of DL solutions to SE problems, but also reevaluate the way in which we apply thesepowerful models. SE research has created methods for developing, optimizing and testingtraditional software projects, however, DL development calls for a shift in the paradigmof the software development life cycle. This survey finds research papers that indicate thebeginnings of that paradigm shift.Our next chapter begins by using a recursive autoencoder to find similar pieces ofsource code through the use of multiple code representations. We mine 10 pertinent Javaprojects and extract their classes and methods. Our next step encodes these code frag-ments into four separate representations: identifiers, bytecode, AST and CFG. We useda recurrent neural network to create a continuous value vector for each word within thecode fragment. We then implemented a recursive autoencoder to encode the arbitrarilylong vector of encoded words. The resulting embedding from the recursive autoencodermaps to a multidimensional space and the similarity to other embedded code fragmentscan be calculated based on distance. After we measured this similarity, we combined thesefour models via ensemble learning where we demonstrated that considering multiple repre-sentations allowed us to detect similar code fragments that no single model could entirelycapture. To show the value of considering multiple representations of source code, we ap-plied our methodology to the process of detecting clones among 10 different Java projects.We found that we achieve a relatively high precision, recall and F-measure when detectingclones among these 10 Java projects. More importantly, through PCA, we found that ourability to detect clones relied on the orthogonal contributions of each representation.Our last chapter involves the DL strategy of NMT, which encoded a sequence of tokensinto a hidden state that is then translated into different tokens by a decoder. We appliedthis methodology to the task of assert statement generation for test methods. We minedthousands of Java projects from GitHub that used the Junit package. We used the tool159spoon to extract the test methods and all potential focal methods that the assert statementsevaluate. Next, we used a heuristic to identify the focal method pertaining to the assertstatement we wanted to predict. We then extracted the focal method signature and bodyas context, which helped the model to predict an accurate assert statement. We performeda series of filtering and data preparation steps. We then created two distinct models, onefor raw source code tokens and the other for abstracted source code tokens. These NMTmodels took the test method and focal method context as input, and output a syntacticallyand semantically correct assert statement. Our evaluation determined how many of thepredicted assert statements were identical to the assert statement written by developers.We also evaluated the quality of the potential assert statement predictions to determine ifour model could generate asserts tantamount to those written by developers.For future work, we consider the taxonomies highlighted in the SLR and the SE is-sues that are not addressed with DL based solutions. We also see a lot of growth anddevelopment in applying SE methodologies to DL software. There is a lot of potential forresearchers to examine Tensorflow, PyTorch, Theano and others for software bugs, imple-mentation bugs and overarching testing strategies. It is the opinion of this researcher thatDL based approaches will continue to be applied to issues within SE as well as other do-mains. Therefore, there will be a large opportunity for the development of a new softwaredevelopment life cycle pertaining to DL.160Bibliography[1] 2015 impact factor released for cochrane database of systematic reviews.[2] How many words are there in the english language.[3] Tiobe index for august 2019.[4] https://maven.apache.org/.[5] Utilizing fast testing to transform java development into an agile, quick release, lowrisk process.[6] Apache commons project distributions: https://archive.apache.org/dist/commons/, 2017.[7] Online appendix (anonymized): https://sites.google.com/view/learningcodesimilarities, 2017.[8] Soot: https://github.com/Sable/soot, 2017.[9] Atlas anonymous online appendix: https://sites.google.com/view/atlas-nmt/home, 2019.[10] Best data science machine learning platform, Feb 2020.[11] B A. Kitchenham. Kitchenham, B.: Guidelines for performing Systematic Liter-ature Reviews in software engineering. EBSE Technical Report EBSE-2007-01. 012007.161[12] Yaser S Abu-Mastafa. The learning problem.[13] Yaser S Abu-Mastafa. The learning problem.[14] Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin. Learn-ing From Data. AMLBook, 2012.[15] M. Allamanis, E. Barr, C. Bird, and C. Sutton. Suggesting accurate methodand class names. FSE’15.[16] Miltiadis Allamanis. The adverse effects of code duplication in machine learningmodels of code. CoRR, abs/1812.06469, 2018.[17] Miltiadis Allamanis, Earl T. Barr, Christian Bird, and Charles Sut-ton. Suggesting accurate method and class names. In Proceedings of the 201510th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, pages38–49, New York, NY, USA, 2015. ACM.[18] Miltiadis Allamanis, Earl T. Barr, Christian Bird, and Charles Sut-ton. Suggesting accurate method and class names. In Proceedings of the 201510th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, pages38–49, New York, NY, USA, 2015. ACM.[19] Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu, and CharlesSutton. A survey of machine learning for big code and naturalness. ACM Comput.Surv., 51(4):81:1–81:37, July 2018.[20] Miltiadis Allamanis, Earl T. Barr, Premkumar T. Devanbu, andCharles A. Sutton. A survey of machine learning for big code and naturalness.CoRR, abs/1709.06182, 2017.[21] Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi.Learning to represent programs with graphs. CoRR, abs/1711.00740, 2017.162[22] Miltiadis Allamanis, Hao Peng, and Charles A. Sutton. A convolutionalattention network for extreme summarization of source code. CoRR, abs/1602.03001,2016.[23] Miltiadis Allamanis, Hao Peng, and Charles A. Sutton. A convolutionalattention network for extreme summarization of source code. CoRR, abs/1602.03001,2016.[24] Miltiadis Allamanis, Hao Peng, and Charles A. Sutton. A convolutionalattention network for extreme summarization of source code. In Proceedings of the33nd International Conference on Machine Learning, ICML 2016, New York City,NY, USA, June 19-24, 2016, pages 2091–2100, 2016.[25] Miltiadis Allamanis and Charles A. Sutton. Mining idioms from source code.In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundationsof Software Engineering, (FSE-22), Hong Kong, China, November 16 - 22, 2014,pages 472–483, 2014.[26] M. Almasi et al. An industrial evaluation of unit test generation: Finding realfaults in a financial application. In ICSE-C’17.[27] Anh Tuan Nguyen and T. N. Nguyen. Automatic categorization with deepneural network for open-source java projects. In 2017 IEEE/ACM 39th Interna-tional Conference on Software Engineering Companion (ICSE-C), pages 164–166,May 2017.[28] Forough Arabshahi, Sameer Singh, and Animashree Anandkumar. Com-bining symbolic expressions and black-box function evaluations in neural programs.CoRR, abs/1801.04342, 2018.163[29] Andrea Arcuri and Gordon Fraser. Java enterprise edition support in search-based junit test generation. In International Symposium on Search Based SoftwareEngineering, pages 3–17. Springer, 2016.[30] M. I. Babar, M. Ghazali, and D. N. A. Jawawi. Systematic reviews in require-ments engineering: A systematic review. In 2014 8th. Malaysian Software Engineer-ing Conference (MySEC), pages 43–48, Sep. 2014.[31] Michael Backes, Sven Bugiel, and Erik Derr. Reliable third-party librarydetection in android and its security applications. In Proceedings of the 2016 ACMSIGSAC Conference on Computer and Communications Security, Vienna, Austria,October 24-28, 2016, pages 356–367, 2016.[32] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machinetranslation by jointly learning to align and translate. CoRR, abs/1409.0473, 2015.[33] B. Baker. On finding duplication and near-duplication in large software systems.WCRE’95.[34] B. Baker. A program for identifying duplicated code. In Computer Science andStatistics, 1992.[35] B. Baker. Parameterized pattern matching: Algorithms and applications. JCSS,52(1), 1996.[36] Matej Balog, Alexander L. Gaunt, Marc Brockschmidt, SebastianNowozin, and Daniel Tarlow. Deepcoder: Learning to write programs. CoRR,abs/1611.01989, 2016.[37] Earl T. Barr, Yuriy Brun, Premkumar Devanbu, Mark Harman, andFederica Sarro. The plastic surgery hypothesis. In Proceedings of the 22Nd ACMSIGSOFT International Symposium on Foundations of Software Engineering, FSE2014, pages 306–317, New York, NY, USA, 2014. ACM.164[38] Gabriele Bavota, Bogdan Dit, Rocco Oliveto, Massimiliano Di Penta,Denys Poshyvanyk, and Andrea De Lucia. An empirical study on the develop-ers' perception of software coupling. In Proceedings of the 2013 InternationalConference on Software Engineering, ICSE ’13, pages 692–701, Piscataway, NJ, USA,2013. IEEE Press.[39] Gabriele Bavota, Malcom Gethers, Rocco Oliveto, Denys Poshyvanyk,and Andrea de Lucia. Improving software modularization via automated analysisof latent topics and dependencies. ACM Trans. Softw. Eng. Methodol., 23(1):4:1–4:33, February 2014.[40] Gabriele Bavota, Rocco Oliveto, Malcom Gethers, Denys Poshyvanyk,and Andrea De Lucia. Methodbook: Recommending move method refactoringsvia relational topic models. IEEE Trans. Softw. Eng., 40(7):671–694, July 2014.[41] I. Baxter, A. Yahin, L. Moura, M. Sant’Anna, and L. Bier. Clone detectionusing abstract syntax trees. ICSM’98.[42] Raja Ben Abdessalem, Shiva Nejati, Lionel C. Briand, and ThomasStifter. Testing advanced driver assistance systems using multi-objective searchand neural networks. In Proceedings of the 31st IEEE/ACM International Confer-ence on Automated Software Engineering, ASE 2016, pages 63–74, New York, NY,USA, 2016. ACM.[43] Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. Neu-ral code comprehension: A learnable representation of code semantics. CoRR,abs/1806.07336, 2018.[44] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representationlearning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell.,35(8):1798–1828, August 2013.165[45] Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Unsuper-vised feature learning and deep learning: A review and new perspectives. CoRR,abs/1206.5538, 2012.[46] Sahil Bhatia, Pushmeet Kohli, and Rishabh Singh. Neuro-symbolic programcorrector for introductory programming assignments. In Proceedings of the 40thInternational Conference on Software Engineering, ICSE ’18, pages 60–70, New York,NY, USA, 2018. ACM.[47] Leo Breiman. Random forests. Machine Learning, 45(1):5–32, Oct 2001.[48] Amar Budhiraja, Kartik Dutta, Raghu Reddy, and Manish Shrivastava.Dwen: Deep word embedding network for duplicate bug report detection in softwarerepositories. In Proceedings of the 40th International Conference on Software Engi-neering: Companion Proceeedings, ICSE ’18, pages 193–194, New York, NY, USA,2018. ACM.[49] Rudy Bunel, Matthew J. Hausknecht, Jacob Devlin, Rishabh Singh, andPushmeet Kohli. Leveraging grammar and reinforcement learning for neural pro-gram synthesis. CoRR, abs/1805.04276, 2018.[50] Jonathon Cai, Richard Shin, and Dawn Song. Making neural programmingarchitectures generalize via recursion. CoRR, abs/1704.06611, 2017.[51] C. Calero, M. F. Bertoa, and M. Á. Moraga. A systematic literature reviewfor software sustainability measures. In 2013 2nd International Workshop on Greenand Sustainable Software (GREENS), pages 46–53, May 2013.[52] C. Chen, Z. Xing, Y. Liu, and K. L. X. Ong. Mining likely analogical apisacross third-party libraries via large-scale unsupervised api semantics embedding.IEEE Transactions on Software Engineering, pages 1–1, 2019.166[53] Chao Chen, Wenrui Diao, Yingpei Zeng, Shanqing Guo, and Chengyu Hu.DRLGENCERT: deep learning-based automated testing of certificate verification inSSL/TLS implementations. CoRR, abs/1808.05444, 2018.[54] Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and YangLiu. From ui design image to gui skeleton: A neural machine translator to bootstrapmobile gui implementation. In Proceedings of the 40th International Conference onSoftware Engineering, ICSE ’18, pages 665–676, New York, NY, USA, 2018. ACM.[55] Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and YangLiu. From ui design image to gui skeleton: A neural machine translator to bootstrapmobile gui implementation. In Proceedings of the 40th International Conference onSoftware Engineering, ICSE ’18, pages 665–676, New York, NY, USA, 2018. ACM.[56] G. Chen, C. Chen, Z. Xing, and B. Xu. Learning a dual-language vector spacefor domain-specific cross-lingual question retrieval. In 2016 31st IEEE/ACM Inter-national Conference on Automated Software Engineering (ASE), pages 744–755, Sep.2016.[57] K. Chen, P. Liu, and Y. Zhang. Achieving accuracy and scalability simultane-ously in detecting application clones on android markets. ICSE’14.[58] Qingying Chen and Minghui Zhou. A neural framework for retrieval and sum-marization of source code. In Proceedings of the 33rd ACM/IEEE International Con-ference on Automated Software Engineering, ASE 2018, pages 826–831, New York,NY, USA, 2018. ACM.[59] Qingying Chen and Minghui Zhou. A neural framework for retrieval and sum-marization of source code. In Proceedings of the 33rd ACM/IEEE International Con-ference on Automated Software Engineering, ASE 2018, pages 826–831, New York,NY, USA, 2018. ACM.167[60] Xinyun Chen, Chang Liu, and Dawn Song. Tree-to-tree neural networks forprogram translation. In Advances in Neural Information Processing Systems 31,S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett,editors, pages 2547–2557. Curran Associates, Inc., 2018.[61] Xinyun Chen, Chang Liu, and Dawn Xiaodong Song. Towards synthesizingcomplex programs from input-output examples. arXiv: Learning, 2018.[62] Zimin Chen et al. Sequencer: Sequence-to-sequence learning for end-to-end pro-gram repair. CoRR, abs/1901.01808, 2019.[63] Zimin Chen, Steve Kommrusch, Michele Tufano, Louis-Noël Pouchet,Denys Poshyvanyk, and Martin Monperrus. Sequencer: Sequence-to-sequence learning for end-to-end program repair. CoRR, abs/1901.01808, 2019.[64] Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, FethiBougares, Holger Schwenk, and Yoshua Bengio. Learning phrase repre-sentations using RNN encoder-decoder for statistical machine translation. CoRR,abs/1406.1078, 2014.[65] M. Choetkiertikul, H. K. Dam, T. Tran, A. Ghose, and J. Grundy. Pre-dicting delivery capability in iterative software development. IEEE Transactions onSoftware Engineering, 44(6):551–573, June 2018.[66] M. Choetkiertikul, H. K. Dam, T. Tran, T. Pham, A. Ghose, and T. Men-zies. A deep learning model for estimating story points. IEEE Transactions onSoftware Engineering, 45(7):637–656, July 2019.[67] Morakot Choetkiertikul, Hoa Khanh Dam, Truyen Tran, and AdityaGhose. Predicting the delay of issues with due dates in software projects. EmpiricalSoftware Engineering, 22(3):1223–1263, Jun 2017.168[68] Mike Cohn. Succeeding with Agile: Software Development Using Scrum. Addison-Wesley Professional, 1st edition, 2009.[69] C. S. Corley, K. Damevski, and N. A. Kraft. Exploring the use of deeplearning for feature location. In 2015 IEEE International Conference on SoftwareMaintenance and Evolution (ICSME), pages 556–560, Sep. 2015.[70] C. S. Corley, K. Damevski, and N. A. Kraft. Exploring the use of deeplearning for feature location. In 2015 IEEE International Conference on SoftwareMaintenance and Evolution (ICSME), pages 556–560, Sep. 2015.[71] Chris Cummins, Pavlos Petoumenos, Alastair Murray, and HughLeather. Compiler fuzzing through deep learning. In Proceedings of the 27th ACMSIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018,pages 95–105, New York, NY, USA, 2018. ACM.[72] H. K. Dam, T. Tran, T. T. M. Pham, S. W. Ng, J. Grundy, and A. Ghose.Automatic feature learning for predicting vulnerable software components. IEEETransactions on Software Engineering, pages 1–1, 2018.[73] Julius Davies, Daniel M. Germán, Michael W. Godfrey, and Abram Hin-dle. Software bertillonage - determining the provenance of software developmentartifacts. Empirical Software Engineering, 18(6):1195–1237, 2013.[74] Rina Dechter. Learning while searching in constraint-satisfaction-problems.In Proceedings of the Fifth AAAI National Conference on Artificial Intelligence,AAAI’86, pages 178–183. AAAI Press, 1986.[75] Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, and Alexander M. Rush.Image-to-markup generation with coarse-to-fine attention. In Proceedings of the 34thInternational Conference on Machine Learning - Volume 70, ICML17, page 980989.JMLR.org, 2017.169[76] J. Deshmukh, A. K. M, S. Podder, S. Sengupta, and N. Dubash. Towards ac-curate duplicate bug retrieval using deep learning techniques. In 2017 IEEE Interna-tional Conference on Software Maintenance and Evolution (ICSME), pages 115–124,Sep. 2017.[77] P. Devanbu. New initiative: The naturalness of software. In 2015 IEEE/ACM 37thIEEE International Conference on Software Engineering, volume 2, pages 543–546,May 2015.[78] Jacob Devlin, Rudy Bunel, Rishabh Singh, Matthew J. Hausknecht, andPushmeet Kohli. Neural program meta-induction. CoRR, abs/1710.04157, 2017.[79] Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh,Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural programlearning under noisy I/O. CoRR, abs/1703.07469, 2017.[80] Arilo C. Dias Neto, Rajesh Subramanyan, Marlon Vieira, and Guil-herme H. Travassos. A survey on model-based testing approaches: A systematicreview. In Proceedings of the 1st ACM International Workshop on Empirical Assess-ment of Software Engineering Languages and Technologies: Held in Conjunction withthe 22Nd IEEE/ACM International Conference on Automated Software Engineering(ASE) 2007, WEASELTech ’07, pages 31–36, New York, NY, USA, 2007. ACM.[81] Bogdan Dit, Latifa Guerrouj, Denys Poshyvanyk, and Giuliano Anto-niol. Can better identifier splitting techniques help feature location? In Proceedingsof the 2011 IEEE 19th International Conference on Program Comprehension, ICPC’11, pages 11–20, Washington, DC, USA, 2011. IEEE Computer Society.[82] Bogdan Dit, Evan Moritz, Mario Linares-Vásquez, Denys Poshyvanyk,and Jane Cleland-Huang. Supporting and accelerating reproducible empiricalresearch in software evolution and maintenance using tracelab component library.Empirical Softw. Engg., 20(5):1198–1236, October 2015.170[83] Bogdan Dit, Meghan Revelle, Malcom Gethers, and Denys Poshy-vanyk. Feature location in source code: a taxonomy and survey. Journal of Software:Evolution and Process, 25(1):53–95, 2013.[84] Bogdan Dit, Meghan Revelle, and Denys Poshyvanyk. Integrating infor-mation retrieval, execution and link analysis algorithms to improve feature locationin software. Empirical Softw. Engg., 18(2):277–309, April 2013.[85] Pedro Domingos. Occams two razors: The sharp and the blunt. In Proceedingsof the Fourth International Conference on Knowledge Discovery and Data Mining,KDD98, page 3743. AAAI Press, 1998.[86] S. Ducasse, M. Rieger, and S. Demeyer. A language independent approachfor detecting duplicated code. ICSM’99.[87] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methodsfor online learning and stochastic optimization. J. Mach. Learn. Res., 12:2121–2159,July 2011.[88] Kevin Ellis, Lucas Morales, Mathias Sablé-Meyer, Armando Solar-Lezama, and Josh Tenenbaum. Learning libraries of subroutines for neu-rally–guided bayesian program induction. In Advances in Neural Information Pro-cessing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, pages 7805–7815. Curran Associates, Inc., 2018.[89] Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Josh Tenen-baum. Learning to infer graphics programs from hand-drawn images. In Advancesin Neural Information Processing Systems 31, S. Bengio, H. Wallach, H. Larochelle,K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, pages 6059–6068. CurranAssociates, Inc., 2018.171[90] F. Falcini, G. Lami, and A. Mitidieri. Yet another challenge for the automotivesoftware: Deep learning. IEEE Software, pages 1–1, 2017.[91] Fabio Falcini, Giuseppe Lami, and Alessandra Costanza. Deep learning inautomotive software. IEEE Software, 34:56–63, 05 2017.[92] Usama Fayyad, Gregory Piatetsky-Shapiro, and Padhraic Smyth. Fromdata mining to knowledge discovery in databases. AI Magazine, 17(3):37, Mar. 1996.[93] J.L. Fleiss et al. Measuring nominal scale agreement among many raters. Psy-chological Bulletin, 76(5):378–382, 1971.[94] Gordon Fraser and Andrea Arcuri. EvoSuite: Automatic Test Suite Genera-tion for Object-oriented Software. In Proceedings of the 19th ACM SIGSOFT Sym-posium and the 13th European Conference on Foundations of Software Engineering,ESEC/FSE ’11, pages 416–419, Szeged, Hungary, 2011. ACM.[95] Gordon Fraser et al. Evosuite: automatic test suite generation for object-oriented software. In ESEC/FSE’11, pages 416–419.[96] Wei Fu and Tim Menzies. Easy over hard: a case study on deep learning. InProceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering,ESEC/FSE 2017, Paderborn, Germany, September 4-8, 2017, pages 49–60, 2017.[97] M. Gabel, L. Jiang, and Z. Su. Scalable detection of semantic clones. ICSE’08.[98] M. Galster, D. Weyns, D. Tofan, B. Michalik, and P. Avgeriou. Vari-ability in software systems—a systematic literature review. IEEE Transactions onSoftware Engineering, 40(3):282–306, March 2014.[99] Jian Gao, Xin Yang, Ying Fu, Yu Jiang, and Jiaguang Sun. Vulseeker:A semantic learning based vulnerability seeker for cross-platform binary. In Pro-ceedings of the 33rd ACM/IEEE International Conference on Automated SoftwareEngineering, ASE 2018, pages 896–899, New York, NY, USA, 2018. ACM.172[100] Alexander L. Gaunt, Marc Brockschmidt, Nate Kushman, and DanielTarlow. Differentiable programs with neural libraries. In Proceedings of the 34thInternational Conference on Machine Learning, Doina Precup and Yee Whye Teh,editors, volume 70 of Proceedings of Machine Learning Research, pages 1213–1222,International Convention Centre, Sydney, Australia, 06–11 Aug 2017. PMLR.[101] R. L. Glass. The naturalness of object orientation: beating a dead horse? IEEESoftware, 19(3):104–103, May 2002.[102] Patrice Godefroid, Hila Peleg, and Rishabh Singh. Learn&fuzz: Ma-chine learning for input fuzzing. In Proceedings of the 32Nd IEEE/ACM InternationalConference on Automated Software Engineering, ASE 2017, pages 50–59, Piscataway,NJ, USA, 2017. IEEE Press.[103] C. Goller and A. Küchler. Learning task-dependent distributed representationsby backpropagation through structure. ICNN’96.[104] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning.The MIT Press, 2016.[105] Palash Goyal and Emilio Ferrara. Graph embedding techniques, applications,and performance: A survey. CoRR, abs/1705.02801, 2017.[106] X. Gu, H. Zhang, D. Zhang, and S. Kim. Deep api learning. FSE’16.[107] Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. Deep code search. InProceedings of the 40th International Conference on Software Engineering, ICSE ’18,pages 933–944, New York, NY, USA, 2018. ACM.[108] Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. Deep code search. InProceedings of the 40th International Conference on Software Engineering, ICSE ’18,pages 933–944, New York, NY, USA, 2018. ACM.173[109] Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. Deepapi learning. In Proceedings of the 2016 24th ACM SIGSOFT International Sympo-sium on Foundations of Software Engineering, FSE 2016, pages 631–642, New York,NY, USA, 2016. ACM.[110] Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. Deepapi learning. In Proceedings of the 2016 24th ACM SIGSOFT International Sympo-sium on Foundations of Software Engineering, FSE 2016, pages 631–642, New York,NY, USA, 2016. ACM.[111] J. Guo, J. Cheng, and J. Cleland-Huang. Semantically enhanced softwaretraceability using deep learning techniques. In 2017 IEEE/ACM 39th InternationalConference on Software Engineering (ICSE), pages 3–14, May 2017.[112] J. Guo, J. Cheng, and J. Cleland-Huang. Semantically enhanced softwaretraceability using deep learning techniques. In 2017 IEEE/ACM 39th InternationalConference on Software Engineering (ICSE), pages 3–14, May 2017.[113] Jianmin Guo, Yu Jiang, Yue Zhao, Quan Chen, and Jiaguang Sun. Dlfuzz:Differential fuzzing testing of deep learning systems. CoRR, abs/1808.09413, 2018.[114] Jin Guo, Jinghui Cheng, and Jane Cleland-Huang. Semantically enhancedsoftware traceability using deep learning techniques. In Proceedings of the 39th In-ternational Conference on Software Engineering, ICSE ’17, pages 3–14, Piscataway,NJ, USA, 2017. IEEE Press.[115] Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. Deepfix:Fixing common c language errors by deep learning. In Proceedings of the Thirty-FirstAAAI Conference on Artificial Intelligence, AAAI’17, pages 1345–1351. AAAI Press,2017.174[116] Kamill Gusmanov. On the adoption of neural networks in modeling softwarereliability. In Proceedings of the 2018 26th ACM Joint Meeting on European SoftwareEngineering Conference and Symposium on the Foundations of Software Engineering,ESEC/FSE 2018, pages 962–964, New York, NY, USA, 2018. ACM.[117] T. Hall, S. Beecham, D. Bowes, D. Gray, and S. Counsell. A systematicliterature review on fault prediction performance in software engineering. IEEETransactions on Software Engineering, 38(6):1276–1304, Nov 2012.[118] Jiawei Han, Jian Pei, and Yiwen Yin. Mining frequent patterns without can-didate generation. pages 1–12. ACM Press, 2000.[119] Z. Han, X. Li, Z. Xing, H. Liu, and Z. Feng. Learning to predict severity ofsoftware vulnerability using only vulnerability description. In 2017 IEEE Interna-tional Conference on Software Maintenance and Evolution (ICSME), pages 125–136,Sep. 2017.[120] Jacob Harer, Onur Ozdemir, Tomo Lazovich, Christopher Reale, Re-becca Russell, Louis Kim, and peter chin. Learning to repair software vul-nerabilities with generative adversarial networks. In Advances in Neural InformationProcessing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, pages 7933–7943. Curran Associates, Inc., 2018.[121] Hideaki Hata, Emad Shihab, and Graham Neubig. Learning to generatecorrective patches using neural machine translation. CoRR, abs/1812.07170, 2018.[122] Vincent J. Hellendoorn, Christian Bird, Earl T. Barr, and MiltiadisAllamanis. Deep learning type inference. In Proceedings of the 2018 26th ACMJoint Meeting on European Software Engineering Conference and Symposium on theFoundations of Software Engineering, ESEC/FSE 2018, pages 152–162, New York,NY, USA, 2018. ACM.175[123] Vincent J. Hellendoorn and Premkumar Devanbu. Are deep neural net-works the best choice for modeling source code? In Proceedings of the 2017 11thJoint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, pages 763–773, New York, NY, USA, 2017. ACM.[124] Vincent J. Hellendoorn and Premkumar T. Devanbu. Are deep neuralnetworks the best choice for modeling source code? In Proceedings of the 2017 11thJoint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn,Germany, September 4-8, 2017, pages 763–773, 2017.[125] Vincent J. Hellendoorn, Premkumar T. Devanbu, and Mohammad AminAlipour. On the naturalness of proofs. In Proceedings of the 2018 26th ACMJoint Meeting on European Software Engineering Conference and Symposium on theFoundations of Software Engineering, ESEC/FSE 2018, pages 724–728, New York,NY, USA, 2018. ACM.[126] A. Hindle, E. T. Barr, Z. Su, M. Gabel, and P. Devanbu. On the naturalnessof software. In 2012 34th International Conference on Software Engineering (ICSE),pages 837–847, June 2012.[127] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. NeuralComput., 9(8):1735–1780, November 1997.[128] S. Hosseini, B. Turhan, and D. Gunarathna. A systematic literature reviewand meta-analysis on cross project defect prediction. IEEE Transactions on SoftwareEngineering, 45(2):111–147, Feb 2019.[129] Q. Huang, X. Xia, D. Lo, and G. C. Murphy. Automating intention mining.IEEE Transactions on Software Engineering, pages 1–1, 2018.176[130] Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Ric-cio, Andrea Stocco, and Paolo Tonella. Taxonomy of real faults in deeplearning systems, 2019.[131] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deepnetwork training by reducing internal covariate shift. CoRR, abs/1502.03167, 2015.[132] A.G. Ivakhnenko and V.G. Lapa. Cybernetic Predicting Devices. JPRS 37, 803.Purdue University School of Electrical Engineering, 1966.[133] Alan Jaffe, Jeremy Lacomis, Edward J. Schwartz, Claire Le Goues, andBogdan Vasilescu. Meaningful variable names for decompiled code: A machinetranslation approach. In Proceedings of the 26th Conference on Program Compre-hension, ICPC ’18, pages 20–30, New York, NY, USA, 2018. ACM.[134] L. Jiang, G. Misherghi, Z. Su, and S. Glondu. Deckard: Scalable and accuratetree-based detection of code clones. ICSE’07.[135] S. Jiang, A. Armaly, and C. McMillan. Automatically generating commitmessages from diffs using neural machine translation. In 2017 32nd IEEE/ACMInternational Conference on Automated Software Engineering (ASE), ASE’17, pages135–146, October 2017. ISSN:.[136] Siyuan Jiang, Ameer Armaly, and Collin McMillan. Automatically gener-ating commit messages from diffs using neural machine translation. In Proceedings ofthe 32Nd IEEE/ACM International Conference on Automated Software Engineering,ASE 2017, pages 135–146, Piscataway, NJ, USA, 2017. IEEE Press.[137] J. Johnson. Identifying redundancy in source code using fingerprints. CASCON’93.[138] J. Johnson. Substring matching for clone detection and change tracking. ICSM’94.[139] J. Johnson. Visualizing textual redundancy in legacy source. CASCON’94.177[140] Nal Kalchbrenner and Phil Blunsom. Recurrent continuous models. InEMNLP, 2013.[141] T. Kamiya, S. Kusumoto, and K. Inoue. Ccfinder: A multilinguistic token-basedcode clone detection system for large scale source code. TSE, 28(7), 2002.[142] Rafael-Michael Karampatsis and Charles A. Sutton. Maybe deep neuralnetworks are the best choice for modeling source code. CoRR, abs/1903.05734, 2019.[143] Andrej Karpathy and Andrej Karpathy. Software 2.0, Nov 2017.[144] Jeannette Kiefer and Jacob Wolfowitz. Stochastic estimation of the maxi-mum of a regression function. 1952.[145] Jinhan Kim, Robert Feldt, and Shin Yoo. Guiding deep learning systemtesting using surprise adequacy. In Proceedings of the 41st International Conferenceon Software Engineering, ICSE ’19, page to appear, New York, NY, USA, 2010.ACM.[146] Junhwi Kim, Minhyuk Kwon, and Shin Yoo. Generating test input with deepreinforcement learning. In Proceedings of the 11th International Workshop on Search-Based Software Testing, SBST ’18, pages 51–58, New York, NY, USA, 2018. ACM.[147] Pieter-Jan Kindermans, Kristof T. Schtt, Maximilian Alber, Klaus-Robert Mller, Dumitru Erhan, Been Kim, and Sven De. Learning how toexplain neural networks: Patternnet and patternattribution, 2017.[148] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimiza-tion, 2014.[149] B. Kitchenham and S Charters. Guidelines for performing systematic literaturereviews in software engineering, 2007.178[150] C. Klammer et al. Writing unit tests: It’s now or never! In ICSTW’15, pages1–4, April 2015.[151] Philipp Koehn and Rebecca Knowles. Six challenges for neural machine trans-lation. CoRR, abs/1706.03872, 2017.[152] R. Komondoor and S. Horwitz. Using slicing to identify duplication in sourcecode. SAS’01.[153] B. Korel and A. M. Al-Yami. Assertion-oriented automated test data genera-tion. In Proceedings of IEEE 18th International Conference on Software Engineering,pages 71–80, March 1996.[154] R. Koschke, R. Falke, and P. Frenzel. Clone detection using abstract syntaxsuffix trees. WCRE’06.[155] J. Krinke. Identifying similar code with program dependence graphs. WCRE’01.[156] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenetclassification with deep convolutional neural networks. In Advances in Neural In-formation Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q.Weinberger, editors, pages 1097–1105. Curran Associates, Inc., 2012.[157] A. Lam, A. Nguyen, H. Nguyen, and T. Nguyen. Combining deep learningwith information retrieval to localize buggy files for bug reports. ASE’15.[158] A. N. Lam, A. T. Nguyen, H. A. Nguyen, and T. N. Nguyen. Combiningdeep learning with information retrieval to localize buggy files for bug reports (n). In2015 30th IEEE/ACM International Conference on Automated Software Engineering(ASE), pages 476–481, Nov 2015.[159] A. N. Lam, A. T. Nguyen, H. A. Nguyen, and T. N. Nguyen. Combiningdeep learning with information retrieval to localize buggy files for bug reports (n). In1792015 30th IEEE/ACM International Conference on Automated Software Engineering(ASE), pages 476–481, Nov 2015.[160] An Ngoc Lam, Anh Tuan Nguyen, Hoan Anh Nguyen, and Tien N.Nguyen. Bug localization with combination of deep learning and information re-trieval. In Proceedings of the 25th International Conference on Program Compre-hension, ICPC 2017, Buenos Aires, Argentina, May 22-23, 2017, pages 218–229,2017.[161] Tien-Duy B. Le, Lingfeng Bao, and David Lo. Dsm: A specification miningtool using recurrent neural network based language model. In Proceedings of the2018 26th ACM Joint Meeting on European Software Engineering Conference andSymposium on the Foundations of Software Engineering, ESEC/FSE 2018, pages896–899, New York, NY, USA, 2018. ACM.[162] Tien-Duy B. Le and David Lo. Deep specification mining. In Proceedings of the27th ACM SIGSOFT International Symposium on Software Testing and Analysis,ISSTA 2018, pages 106–117, New York, NY, USA, 2018. ACM.[163] Alexander LeClair, Siyuan Jiang, and Collin McMillan. A neural modelfor generating natural language summaries of program subroutines. In Proceedings ofthe 41st International Conference on Software Engineering, ICSE ’19, pages 795–806,Piscataway, NJ, USA, 2019. IEEE Press.[164] Yann LeCun, Yoshua Bengio, and Geoffrey E. Hinton. Deep learning.Nature, 521(7553):436–444, 2015.[165] Yann LeCun, Koray Kavukcuoglu, and Clément Farabet. Convolutionalnetworks and applications in vision. In International Symposium on Circuits andSystems (ISCAS 2010), May 30 - June 2, 2010, Paris, France, pages 253–256, 2010.180[166] Sun-Ro Lee, Min-Jae Heo, Chan-Gun Lee, Milhan Kim, and Gaeul Jeong.Applying deep learning based automatic bug triager to industrial projects. In Pro-ceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering,ESEC/FSE 2017, pages 926–931, New York, NY, USA, 2017. ACM.[167] Dor Levy and Lior Wolf. Learning to align the source code to the compiledobject code. In Proceedings of the 34th International Conference on Machine Learn-ing, Doina Precup and Yee Whye Teh, editors, volume 70 of Proceedings of MachineLearning Research, pages 2043–2051, International Convention Centre, Sydney, Aus-tralia, 06–11 Aug 2017. PMLR.[168] D. Li, Z. Wang, and Y. Xue. Fine-grained android malware detection based ondeep learning. In 2018 IEEE Conference on Communications and Network Security(CNS), pages 1–2, May 2018.[169] D. Li, Z. Wang, and Y. Xue. Fine-grained android malware detection based ondeep learning. In 2018 IEEE Conference on Communications and Network Security(CNS), pages 1–2, May 2018.[170] Liuqing Li, He Feng, Wenjie Zhuang, Na Meng, and Barbara G. Ryder.Cclearner: A deep learning-based clone detection approach. 2017 IEEE InternationalConference on Software Maintenance and Evolution (ICSME), pages 249–260, 2017.[171] Xiangang Li and Xihong Wu. Constructing long short-term memory baseddeep recurrent neural networks for large vocabulary speech recognition. CoRR,abs/1410.4281, 2014.[172] Z. Li, S. Lu, S. Myagmar, and Y. Zhou. Cp-miner: Finding copy-paste andrelated bugs in large-scale software code. TSE, 32(3), 2006.[173] Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc V Le, andNi Lao. Memory augmented policy optimization for program synthesis and seman-181tic parsing. In Advances in Neural Information Processing Systems 31, S. Bengio,H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors,pages 9994–10006. Curran Associates, Inc., 2018.[174] B. Lin, C. Nagy, G. Bavota, and M. Lanza. On the impact of refactoringoperations on code naturalness. In 2019 IEEE 26th International Conference onSoftware Analysis, Evolution and Reengineering (SANER), pages 594–598, Feb 2019.[175] B. Lin, F. Zampetti, G. Bavota, M. Di Penta, M. Lanza, and R. Oliveto.Sentiment analysis for software engineering: How far can we go? In 2018 IEEE/ACM40th International Conference on Software Engineering (ICSE), pages 94–104, May2018.[176] Bingchang Liu, Wei Huo, Chao Zhang, Wenchao Li, Feng Li, Aihua Piao,and Wei Zou. αdiff: Cross-version binary code similarity detection with dnn. InProceedings of the 33rd ACM/IEEE International Conference on Automated SoftwareEngineering, ASE 2018, pages 667–678, New York, NY, USA, 2018. ACM.[177] C. Liu, C. Chen, J. Han, and P. Yu. Gplag: Detection of software plagiarismby program dependence graph analysis. KDD’06.[178] Chang Liu, Xinyun Chen, Richard Shin, Mingcheng Chen, and DawnSong. Latent attention for if-then program synthesis. In Advances in Neural Infor-mation Processing Systems 29, D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon,and R. Garnett, editors, pages 4574–4582. Curran Associates, Inc., 2016.[179] Hui Liu, Zhifeng Xu, and Yanzhen Zou. Deep learning based feature envydetection. In Proceedings of the 33rd ACM/IEEE International Conference on Auto-mated Software Engineering, ASE 2018, pages 385–396, New York, NY, USA, 2018.ACM.182[180] Hui Liu, Zhifeng Xu, and Yanzhen Zou. Deep learning based feature envydetection. In Proceedings of the 33rd ACM/IEEE International Conference on Auto-mated Software Engineering, ASE 2018, pages 385–396, New York, NY, USA, 2018.ACM.[181] K. Liu, D. Kim, T. F. Bissyande, S. Yoo, and Y. Le Traon. Mining fixpatterns for findbugs violations. IEEE Transactions on Software Engineering, pages1–1, 2018.[182] P. Liu, X. Zhang, M. Pistoia, Y. Zheng, M. Marques, and L. Zeng. Au-tomatic text input generation for mobile testing. In 2017 IEEE/ACM 39th Interna-tional Conference on Software Engineering (ICSE), pages 643–653, May 2017.[183] Peng Liu, Xiangyu Zhang, Marco Pistoia, Yunhui Zheng, Manoel Mar-ques, and Lingfei Zeng. Automatic text input generation for mobile testing. InProceedings of the 39th International Conference on Software Engineering, ICSE ’17,pages 643–653, Piscataway, NJ, USA, 2017. IEEE Press.[184] Zhongxin Liu, Xin Xia, Ahmed E. Hassan, David Lo, Zhenchang Xing,and Xinyu Wang. Neural-machine-translation-based commit message generation:How far are we? In Proceedings of the 33rd ACM/IEEE International Conference onAutomated Software Engineering, ASE 2018, pages 373–384, New York, NY, USA,2018. ACM.[185] Zhongxin Liu, Xin Xia, Ahmed E. Hassan, David Lo, Zhenchang Xing,and Xinyu Wang. Neural-machine-translation-based commit message generation:How far are we? In Proceedings of the 33rd ACM/IEEE International Conference onAutomated Software Engineering, ASE 2018, pages 373–384, New York, NY, USA,2018. ACM.[186] Lei Ma, Felix Juefei-Xu, Jiyuan Sun, Chunyang Chen, Ting Su, FuyuanZhang, Minhui Xue, Bo Li, Li Li, Yang Liu, Jianjun Zhao, and Yadong183Wang. Deepgauge: Comprehensive and multi-granularity testing criteria for gaugingthe robustness of deep learning systems. CoRR, abs/1803.07519, 2018.[187] Lei Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li,Chunyang Chen, Ting Su, Li Li, Yang Liu, Jianjun Zhao, and YadongWang. Deepgauge: Multi-granularity testing criteria for deep learning systems. InProceedings of the 33rd ACM/IEEE International Conference on Automated SoftwareEngineering, ASE 2018, pages 120–131, New York, NY, USA, 2018. ACM.[188] Shiqing Ma, Yingqi Liu, Wen-Chuan Lee, Xiangyu Zhang, and AnanthGrama. Mode: Automated neural network model debugging via state differentialanalysis and input selection. In Proceedings of the 2018 26th ACM Joint Meeting onEuropean Software Engineering Conference and Symposium on the Foundations ofSoftware Engineering, ESEC/FSE 2018, pages 175–186, New York, NY, USA, 2018.ACM.[189] Ziang Ma, Haoyu Wang, Yao Guo, and Xiangqun Chen. Libradar: fast andaccurate detection of third-party libraries in android apps. In Proceedings of the 38thInternational Conference on Software Engineering, ICSE 2016, Austin, TX, USA,May 14-22, 2016 - Companion Volume, pages 653–656, 2016.[190] David J. C. MacKay. Information Theory, Inference Learning Algorithms. Cam-bridge University Press, USA, 2002.[191] Suvodeep Majumder, Nikhila Balaji, Katie Brey, Wei Fu, and Tim Men-zies. 500+ times faster than deep learning: A case study exploring faster methodsfor text mining stackoverflow. In Proceedings of the 15th International Conferenceon Mining Software Repositories, MSR ’18, pages 554–563, New York, NY, USA,2018. ACM.[192] Andrian Marcus and Jonathan I. Maletic. Identification of high-level conceptclones in source code. In 16th IEEE International Conference on Automated Software184Engineering (ASE 2001), 26-29 November 2001, Coronado Island, San Diego, CA,USA, pages 107–114, 2001.[193] Collin McMillan, Mark Grechanik, and Denys Poshyvanyk. Detectingsimilar software applications. In Proceedings of the 34th International Conference onSoftware Engineering, ICSE ’12, pages 364–374, Piscataway, NJ, USA, 2012. IEEEPress.[194] Ali Mesbah, Andrew Rice, Emily Johnston, Nick Glorioso, and EdwardAftandilian. Deepdelta: Learning to repair compilation errors. In Proceedingsof the 2019 27th ACM Joint Meeting on European Software Engineering Conferenceand Symposium on the Foundations of Software Engineering, ESEC/FSE 2019, pages925–936, New York, NY, USA, 2019. ACM.[195] T. Mikolov. Statistical Language Models Based on Neural Networks. PhD thesis,2012.[196] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean. Distributedrepresentations of words and phrases and their compositionality. NIPS’13.[197] Narcisa Andreea Milea, Lingxiao Jiang, and Siau-Cheng Khoo. Vectorabstraction and concretization for scalable detection of refactorings. In Proceedingsof the 22Nd ACM SIGSOFT International Symposium on Foundations of SoftwareEngineering, FSE 2014, pages 86–97, New York, NY, USA, 2014. ACM.[198] Matthew Mirman, Timon Gehr, and Martin Vechev. Differentiable ab-stract interpretation for provably robust neural networks. In Proceedings of the 35thInternational Conference on Machine Learning, Jennifer Dy and Andreas Krause,editors, volume 80 of Proceedings of Machine Learning Research, pages 3578–3586,Stockholmsman, Stockholm Sweden, 10–15 Jul 2018. PMLR.185[199] K. P. Moran, C. Bernal-Cenas, M. Curcio, R. Bonett, and D. Poshy-vanyk. Machine learning-based prototyping of graphical user interfaces for mobileapps. IEEE Transactions on Software Engineering, pages 1–1, 2018.[200] Kevin Moran, Carlos Bernal-Cárdenas, Michael Curcio, RichardBonett, and Denys Poshyvanyk. Machine learning-based prototyping of graph-ical user interfaces for mobile apps. CoRR, abs/1802.02312, 2018.[201] Kevin Moran, Cody Watson, John Hoskins, George Purnell, and DenysPoshyvanyk. Detecting and summarizing gui changes in evolving mobile apps. InProceedings of the 33rd ACM/IEEE International Conference on Automated SoftwareEngineering, ASE 2018, pages 543–553, New York, NY, USA, 2018. ACM.[202] Vijayaraghavan Murali, Swarat Chaudhuri, and Chris Jermaine.Bayesian specification learning for finding api usage errors. In Proceedings of the2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017,pages 151–162, New York, NY, USA, 2017. ACM.[203] Vijayaraghavan Murali, Letao Qi, Swarat Chaudhuri, and Chris Jer-maine. Neural sketch learning for conditional program generation. In InternationalConference on Learning Representations, 2018.[204] Graham Neubig. Neural machine translation and sequence-to-sequence models: Atutorial. CoRR, abs/1703.01619, 2017.[205] Anh Tuan Nguyen and Tien N. Nguyen. Graph-based statistical languagemodel for code. In Proceedings of the 37th International Conference on SoftwareEngineering - Volume 1, ICSE ’15, pages 858–868, Piscataway, NJ, USA, 2015. IEEEPress.[206] Anh Tuan Nguyen and Tien N. Nguyen. Automatic categorization with deepneural network for open-source java projects. In Proceedings of the 39th International186Conference on Software Engineering, ICSE 2017, Buenos Aires, Argentina, May 20-28, 2017 - Companion Volume, pages 164–166, 2017.[207] Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N. Nguyen. Lexicalstatistical machine translation for language migration. In Proceedings of the 20139th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2013, pages651–654, New York, NY, USA, 2013. ACM.[208] Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N. Nguyen. Migratingcode with statistical machine translation. In Companion Proceedings of the 36thInternational Conference on Software Engineering, ICSE Companion 2014, pages544–547, New York, NY, USA, 2014. ACM.[209] Tung Thanh Nguyen, Hoan Anh Nguyen, Nam H. Pham, Jafar M. Al-Kofahi, and Tien N. Nguyen. Graph-based mining of multiple object usagepatterns. In Proceedings of the the 7th Joint Meeting of the European SoftwareEngineering Conference and the ACM SIGSOFT Symposium on The Foundationsof Software Engineering, ESEC/FSE ’09, pages 383–392, New York, NY, USA, 2009.ACM.[210] Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, SakrianiSakti, Tomoki Toda, and Satoshi Nakamura. Learning to generate pseudo-code from source code using statistical machine translation. In Proceedings of the 30thIEEE/ACM International Conference on Automated Software Engineering, ASE ’15,pages 574–584, Piscataway, NJ, USA, 2015. IEEE Press.[211] J. Ott, A. Atchison, P. Harnack, A. Bergh, and E. Linstead. A deep learn-ing approach to identifying source code in images and video. In 2018 IEEE/ACM15th International Conference on Mining Software Repositories (MSR), pages 376–386, May 2018.187[212] Mingdong Ou, Peng Cui, Jian Pei, Ziwei Zhang, and Wenwu Zhu. Asym-metric transitivity preserving graph embedding. In Proceedings of the 22Nd ACMSIGKDD International Conference on Knowledge Discovery and Data Mining, KDD’16, pages 1105–1114, New York, NY, USA, 2016. ACM.[213] Carlos Pacheco and Michael D. Ernst. Randoop: Feedback-directed randomtesting for java. In OOPSLA’07, pages 815–816, 01 2007.[214] Annibale Panichella, Bogdan Dit, Rocco Oliveto, Massimiliano DiPenta, Denys Poshyvanyk, and Andrea De Lucia. How to effectively usetopic models for software engineering tasks? an approach based on genetic algo-rithms. In 35th International Conference on Software Engineering, ICSE ’13, SanFrancisco, CA, USA, May 18-26, 2013, pages 522–531, 2013.[215] Annibale Panichella, Bogdan Dit, Rocco Oliveto, Massimiliano DiPenta, Denys Poshyvanyk, and Andrea De Lucia. Parameterizing and as-sembling ir-based solutions for SE tasks using genetic algorithms. In IEEE 23rd In-ternational Conference on Software Analysis, Evolution, and Reengineering, SANER2016, Suita, Osaka, Japan, March 14-18, 2016 - Volume 1, pages 314–325, 2016.[216] Stefano Panzeri, Cesare Magri, and Ludovico Carraro. Sampling bias.[217] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu:A method for automatic evaluation of machine translation. In Proceedings of the40th Annual Meeting on Association for Computational Linguistics, ACL ’02, pages311–318, 2002.[218] Renaud Pawlak, Martin Monperrus, Nicolas Petitprez, CarlosNoguera, and Lionel Seinturier. Spoon: A library for implementing anal-yses and transformations of java source code. Softw. Pract. Exper., 46(9):1155–1179,September 2016.188[219] Chris Piech, Jonathan Huang, Andy Nguyen, Mike Phulsuksombati,Mehran Sahami, and Leonidas Guibas. Learning program embeddings to prop-agate feedback on student code. In Proceedings of the 32nd International Conferenceon Machine Learning, Francis Bach and David Blei, editors, volume 37 of Proceed-ings of Machine Learning Research, pages 1093–1102, Lille, France, 07–09 Jul 2015.PMLR.[220] D. Pierret and D. Poshyvanyk. An empirical exploration of regularities in open-source software lexicons. In 2009 IEEE 17th International Conference on ProgramComprehension, pages 228–232, May 2009.[221] Yolande Poirier. What are the most popular libraries java developers use? basedon github’s top projects.[222] Abdallah Qusef, Rocco Oliveto, and Andrea De Lucia. Recovering trace-ability links between unit tests and classes under test: An improved method. In 26thIEEE International Conference on Software Maintenance (ICSM 2010), September12-18, 2010, Timisoara, Romania, pages 1–10, 2010.[223] Danijel Radjenović, Marjan Heričko, Richard Torkar, and AlešŽivkovič. Software fault prediction metrics. Inf. Softw. Technol., 55(8):1397–1418,August 2013.[224] Carl Edward Rasmussen and Zoubin Ghahramani. Occam's razor. In Ad-vances in Neural Information Processing Systems 13, T. K. Leen, T. G. Dietterich,and V. Tresp, editors, pages 294–300. MIT Press, 2001.[225] B. Ray, V. Hellendoorn, S. Godhane, Z. Tu, A. Bacchelli, and P. De-vanbu. On the "naturalness" of buggy code. In 2016 IEEE/ACM 38th InternationalConference on Software Engineering (ICSE), pages 428–439, May 2016.189[226] Veselin Raychev, Martin Vechev, and Eran Yahav. Code completion withstatistical language models. In Proceedings of the 35th ACM SIGPLAN Conferenceon Programming Language Design and Implementation, PLDI ’14, pages 419–428,New York, NY, USA, 2014. ACM.[227] Scott Reed and Nando Freitas. Neural programmer-interpreters. 11 2015.[228] Meghan Revelle, Bogdan Dit, and Denys Poshyvanyk. Using data fusionand web mining to support feature location in software. In Proceedings of the 2010IEEE 18th International Conference on Program Comprehension, ICPC ’10, pages14–23, Washington, DC, USA, 2010. IEEE Computer Society.[229] Daniel Rodriguez, Javier Dolado, and Javier Tuya. Bayesian concepts insoftware testing: An initial review. In Proceedings of the 6th International Workshopon Automating Test Case Design, Selection and Evaluation, A-TEST 2015, pages41–46, New York, NY, USA, 2015. ACM.[230] S. Romansky, N. C. Borle, S. Chowdhury, A. Hindle, and R. Greiner.Deep green: Modelling time-series of software energy consumption. In 2017 IEEEInternational Conference on Software Maintenance and Evolution (ICSME), pages273–283, Sep. 2017.[231] Chanchal Kumar Roy and James R. Cordy. NICAD: accurate detection ofnear-miss intentional clones using flexible pretty-printing and code normalization. InThe 16th IEEE International Conference on Program Comprehension, ICPC 2008,Amsterdam, The Netherlands, June 10-13, 2008, pages 172–181, 2008.[232] Vaibhav Saini, Farima Farmahinifarahani, Yadong Lu, Pierre Baldi, andCristina V. Lopes. Oreo: Detection of clones in the twilight zone. CoRR,abs/1806.05837, 2018.190[233] Hitesh Sajnani and Cristina Lopes. A parallel and efficient approach to largescale clone detection. In Proceedings of the 7th International Workshop on SoftwareClones, IWSC ’13, pages 46–52, Piscataway, NJ, USA, 2013. IEEE Press.[234] Hitesh Sajnani, Vaibhav Saini, Jeffrey Svajlenko, Chanchal K. Roy,and Cristina V. Lopes. Sourcerercc: Scaling code clone detection to big-code.In Proceedings of the 38th International Conference on Software Engineering, ICSE’16, pages 1157–1168, New York, NY, USA, 2016. ACM.[235] Jonathan Schaffer. What not to multiply without necessity. Australasian Jour-nal of Philosophy, 93(4):644–664, 2015.[236] Jürgen Schmidhuber. Deep learning in neural networks: An overview. CoRR,abs/1404.7828, 2014.[237] Jan Schroeder, Christian Berger, Alessia Knauss, Harri Preenja, Mo-hammad Ali, Miroslaw Staron, and Thomas Herpel. Predicting and eval-uating software model growth in the automotive industry. CoRR, abs/1708.02884,2017.[238] D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, ToddPhillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, and Dan Dennison. Hidden technical debt in machine learn-ing systems. In Advances in Neural Information Processing Systems 28, C. Cortes,N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, pages 2503–2511.Curran Associates, Inc., 2015.[239] Sina Shamshiri. Automated Unit Test Generation for Evolving Software. In Pro-ceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering,FSE’15, pages 1038–1041, Bergamo, Italy, 2015. ACM.191[240] P. Sharma and J. Singh. Systematic literature review on software effort estima-tion using machine learning approaches. In 2017 International Conference on NextGeneration Computing and Information Systems (ICNGCIS), pages 43–47, Dec 2017.[241] Richard Shin, Illia Polosukhin, and Dawn Song. Improving neural programsynthesis with inferred execution traces. In Advances in Neural Information Process-ing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi,and R. Garnett, editors, pages 8917–8926. Curran Associates, Inc., 2018.[242] Xujie Si, Hanjun Dai, Mukund Raghothaman, Mayur Naik, and Le Song.Learning loop invariants for program verification. In Advances in Neural InformationProcessing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, pages 7751–7762. Curran Associates, Inc., 2018.[243] R. Socher, C. Lin, A. Ng, and C. Manning. Parsing natural scenes and naturallanguage with recursive neural networks. ICML’11.[244] R. Socher, J. Pennington, E. Huang, A. Ng, and C. Manning. Semi-supervised recursive autoencoders for predicting sentiment distributions. EMNLP’11.[245] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. Manning, A. Ng, andC. Potts. Recursive deep models for semantic compositionality over a sentimenttreebank. EMNLP’13.[246] Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng,and Christopher D. Manning. Semi-supervised recursive autoencoders for pre-dicting sentiment distributions. In Proceedings of the Conference on Empirical Meth-ods in Natural Language Processing, EMNLP ’11, pages 151–161, Stroudsburg, PA,USA, 2011. Association for Computational Linguistics.[247] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christo-pher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep192models for semantic compositionality over a sentiment treebank. In Proceedings ofthe 2013 Conference on Empirical Methods in Natural Language Processing, pages1631–1642, Seattle, Washington, USA, October 2013. Association for ComputationalLinguistics.[248] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever,and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networksfrom overfitting. Journal of Machine Learning Research, 15(56):1929–1958, 2014.[249] Student. An experimental determination of the probable error of dr spearman’scorrelation coefficients. Biometrika, 13(2/3):263–282, 1921.[250] Shao-Hua Sun, Hyeonwoo Noh, Sriram Somasundaram, and Joseph Lim.Neural program synthesis from diverse demonstration videos. In Proceedings ofthe 35th International Conference on Machine Learning, Jennifer Dy and AndreasKrause, editors, volume 80 of Proceedings of Machine Learning Research, pages 4790–4799, Stockholmsman, Stockholm Sweden, 10–15 Jul 2018. PMLR.[251] Youcheng Sun, Min Wu, Wenjie Ruan, Xiaowei Huang, MartaKwiatkowska, and Daniel Kroening. Concolic testing for deep neural net-works. In Proceedings of the 33rd ACM/IEEE International Conference on Auto-mated Software Engineering, ASE 2018, pages 109–119, New York, NY, USA, 2018.ACM.[252] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learn-ing with neural networks. CoRR, abs/1409.3215, 2014.[253] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learn-ing with neural networks. In Advances in Neural Information Processing Systems27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger,editors, pages 3104–3112. Curran Associates, Inc., 2014.193[254] J. Svajlenko and C. Roy. Evaluating clone detection tools with bigclonebench.ICSME’15, 2015.[255] Jeffrey Svajlenko, Chanchal K. Roy, and James R. Cordy. A mutationanalysis based benchmarking framework for clone detectors. In Proceeding of the 7thInternational Workshop on Software Clones, IWSC 2013, San Francisco, CA, USA,May 19, 2013, pages 8–9, 2013.[256] Ricardo Terra, Luis Fernando Miranda, Marco Tulio Valente, andRoberto S. Bigonha. Qualitas.class Corpus: A compiled version of the QualitasCorpus. Software Engineering Notes, 38(5):1–4, 2013.[257] Chris Thunes. c2nes/javalang, Feb 2019.[258] Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. Deeptest: Auto-mated testing of deep-neural-network-driven autonomous cars. In Proceedings of the40th International Conference on Software Engineering, ICSE ’18, pages 303–314,New York, NY, USA, 2018. ACM.[259] Michele Tufano, Fabio Palomba, Gabriele Bavota, Massimiliano DiPenta, Rocco Oliveto, Andrea De Lucia, and Denys Poshyvanyk. Thereand back again: Can you compile that snapshot? Journal of Software: Evolutionand Process, 29(4):e1838.[260] Michele Tufano, Jevgenija Pantiuchina, Cody Watson, GabrieleBavota, and Denys Poshyvanyk. On learning meaningful code changes vianeural machine translation. CoRR, abs/1901.09102, 2019.[261] Michele Tufano, Cody Watson, Gabriele Bavota, MassimilianoDi Penta, Martin White, and Denys Poshyvanyk. Deep learning similaritiesfrom different representations of source code. In Proceedings of the 15th International194Conference on Mining Software Repositories, MSR ’18, pages 542–553, New York,NY, USA, 2018. ACM.[262] Michele Tufano, Cody Watson, Gabriele Bavota, MassimilianoDi Penta, Martin White, and Denys Poshyvanyk. An empirical investigationinto learning bug-fixing patches in the wild via neural machine translation. In Pro-ceedings of the 33rd ACM/IEEE International Conference on Automated SoftwareEngineering, ASE 2018, pages 832–837, New York, NY, USA, 2018. ACM.[263] Michele Tufano, Cody Watson, Gabriele Bavota, MassimilianoDi Penta, Martin White, and Denys Poshyvanyk. An empirical investigationinto learning bug-fixing patches in the wild via neural machine translation. In Pro-ceedings of the 33rd ACM/IEEE International Conference on Automated SoftwareEngineering, ASE 2018, pages 832–837, New York, NY, USA, 2018. ACM.[264] Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano DiPenta, Martin White, and Denys Poshyvanyk. An empirical study onlearning bug-fixing patches in the wild via neural machine translation. CoRR,abs/1812.08693, 2018.[265] Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano DiPenta, Martin White, and Denys Poshyvanyk. Learning how to mutatesource code from bug-fixes. CoRR, abs/1812.10772, 2018.[266] Anne-Wil Harzing Sat 6 Feb 2016 16:10 (updated Thu 5 Sep 2019 10:36).Publish or perish.[267] Muhammad Usman, Emilia Mendes, Francila Weidt, and RicardoBritto. Effort estimation in agile software development: A systematic literaturereview. In Proceedings of the 10th International Conference on Predictive Modelsin Software Engineering, PROMISE ’14, pages 82–91, New York, NY, USA, 2014.ACM.195[268] Bogdan Vasilescu, Casey Casalnuovo, and Premkumar Devanbu. Recov-ering clear, natural identifiers from obfuscated js names. In Proceedings of the 201711th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, pages683–693, New York, NY, USA, 2017. ACM.[269] Mario Linares Vásquez, Andrew Holtzhauer, and Denys Poshyvanyk.On automatically detecting similar android apps. In 24th IEEE International Confer-ence on Program Comprehension, ICPC 2016, Austin, TX, USA, May 16-17, 2016,pages 1–10, 2016.[270] Ashwin J. Vijayakumar, Abhishek Mohta, Oleksandr Polozov, DhruvBatra, Prateek Jain, and Sumit Gulwani. Neural-guided deductive search forreal-time program synthesis from examples. CoRR, abs/1804.01186, 2018.[271] Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, JianWu, and Philip S. Yu. Improving automatic source code summarization viadeep reinforcement learning. In Proceedings of the 33rd ACM/IEEE InternationalConference on Automated Software Engineering, ASE 2018, pages 397–407, NewYork, NY, USA, 2018. ACM.[272] Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, JianWu, and Philip S. Yu. Improving automatic source code summarization viadeep reinforcement learning. In Proceedings of the 33rd ACM/IEEE InternationalConference on Automated Software Engineering, ASE 2018, pages 397–407, NewYork, NY, USA, 2018. ACM.[273] Jingyi Wang, Guoliang Dong, Jun Sun, Xinyu Wang Wang, and PeixinZhang. Adversarial sample detection for deep neural network through model muta-tion testing. In Proceedings of the 41st International Conference on Software Engi-neering, ICSE ’19, page to appear, New York, NY, USA, 2019. ACM.196[274] Ke Wang, Rishabh Singh, and Zhendong Su. Dynamic neural program em-bedding for program repair. CoRR, abs/1711.07163, 2017.[275] S. Wang, T. Liu, and L. Tan. Automatically learning semantic features for defectprediction. ICSE’16.[276] S. Wang, T. Liu, and L. Tan. Automatically learning semantic features fordefect prediction. In 2016 IEEE/ACM 38th International Conference on SoftwareEngineering (ICSE), pages 297–308, May 2016.[277] Song Wang, Taiyue Liu, and Lin Tan. Automatically learning semantic featuresfor defect prediction. In Proceedings of the 38th International Conference on SoftwareEngineering, ICSE ’16, pages 297–308, New York, NY, USA, 2016. ACM.[278] Tiantian Wang, Mark Harman, Yue Jia, and Jens Krinke. Searching forbetter configurations: a rigorous approach to clone evaluation. In Joint Meeting of theEuropean Software Engineering Conference and the ACM SIGSOFT Symposium onthe Foundations of Software Engineering, ESEC/FSE’13, Saint Petersburg, RussianFederation, August 18-26, 2013, pages 455–465, 2013.[279] Tiantian Wang, Mark Harman, Yue Jia, and Jens Krinke. Searching forbetter configurations: A rigorous approach to clone evaluation. In Proceedings of the2013 9th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2013,pages 455–465, New York, NY, USA, 2013. ACM.[280] Cody Watson, David Palacio, Nathan Cooper, Kevin Moran, and DenysPoshyvanyk. Data analysis for the systematic literature review of dl4se.[281] M. Wen, R. Wu, and S. C. Cheung. How well do change sequences predictdefects? sequence learning from software changes. IEEE Transactions on SoftwareEngineering, pages 1–1, 2018.197[282] M. White. Deep representations for software engineering. In 2015 IEEE/ACM 37thIEEE International Conference on Software Engineering, volume 2, pages 781–783,May 2015.[283] M. White, M. Tufano, C. Vendome, and D. Poshyvanyk. Deep learning codefragments for code clone detection. ASE’16.[284] M. White, M. Tufano, C. Vendome, and D. Poshyvanyk. Deep learningcode fragments for code clone detection. In 2016 31st IEEE/ACM InternationalConference on Automated Software Engineering (ASE), pages 87–98, Sep. 2016.[285] M. White, M. Tufano, C. Vendome, and D. Poshyvanyk. Deep learningcode fragments for code clone detection. In 2016 31st IEEE/ACM InternationalConference on Automated Software Engineering (ASE), pages 87–98, Sep. 2016.[286] Martin White, Christopher Vendome, Mario Linares-Vásquez, andDenys Poshyvanyk. Toward deep learning software repositories. In Proceedingsof the 12th Working Conference on Mining Software Repositories, MSR ’15, pages334–345, Piscataway, NJ, USA, 2015. IEEE Press.[287] Martin White, Christopher Vendome, Mario Linares-Vásquez, andDenys Poshyvanyk. Toward deep learning software repositories. In Proceedingsof the 12th Working Conference on Mining Software Repositories, MSR ’15, pages334–345, Piscataway, NJ, USA, 2015. IEEE Press.[288] H. Wu, L. Shi, C. Chen, Q. Wang, and B. Boehm. Maintenance effort es-timation for open source software: A systematic literature review. In 2016 IEEEInternational Conference on Software Maintenance and Evolution (ICSME), pages32–43, Oct 2016.[289] B. Xu, D. Ye, Z. Xing, X. Xia, G. Chen, and S. Li. Predicting semanticallylinkable knowledge in developer online forums via convolutional neural network. In1982016 31st IEEE/ACM International Conference on Automated Software Engineering(ASE), pages 51–62, Sep. 2016.[290] W. Yang. Identifying syntactic differences between two programs. SPE, 21(7), 1991.[291] Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Gra-ham Neubig. Learning to mine parallel natural language/source code corpora fromstack overflow. pages 388–389, 05 2018.[292] Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. Graying the black box:Understanding dqns. CoRR, abs/1602.02658, 2016.[293] Matthew D. Zeiler. Adadelta: An adaptive learning rate method, 2012.[294] Lisa Zhang, Gregory Rosenblatt, Ethan Fetaya, Renjie Liao, WilliamByrd, Matthew Might, Raquel Urtasun, and Richard Zemel. Neuralguided constraint logic programming for program synthesis. In Advances in NeuralInformation Processing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grau-man, N. Cesa-Bianchi, and R. Garnett, editors, pages 1737–1746. Curran Associates,Inc., 2018.[295] Mengshi Zhang, Yuqun Zhang, Lingming Zhang, Cong Liu, and SarfrazKhurshid. Deeproad: Gan-based metamorphic testing and input validation frame-work for autonomous driving systems. In Proceedings of the 33rd ACM/IEEE Inter-national Conference on Automated Software Engineering, ASE 2018, pages 132–142,New York, NY, USA, 2018. ACM.[296] Yuhao Zhang, Yifan Chen, Shing-Chi Cheung, Yingfei Xiong, andLu Zhang. An empirical study on tensorflow program bugs. In Proceedings of the27th ACM SIGSOFT International Symposium on Software Testing and Analysis,ISSTA 2018, pages 129–140, New York, NY, USA, 2018. ACM.199[297] Jinman Zhao, Aws Albarghouthi, Vaibhav Rastogi, Somesh Jha, andDamien Octeau. Neural-augmented static analysis of android communication.CoRR, abs/1809.04059, 2018.[298] Zhi-Hua Zhou. Ensemble Learning, pages 270–273. Springer US, Boston, MA,2009.[299] Luisa M. Zintgraf, Taco S. Cohen, Tameem Adel, and Max Welling.Visualizing deep neural network decisions: Prediction difference analysis. CoRR,abs/1702.04595, 2017.[300] Amit Zohar and Lior Wolf. Automatic program synthesis of long programswith a learned garbage collector. In Advances in Neural Information ProcessingSystems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi,and R. Garnett, editors, pages 2094–2103. Curran Associates, Inc., 2018.200

Software Engineering Meets Deep Learning: A Mapping Study

Abstract

Extracted data

Software Engineering Meets Deep Learning: A Mapping Study

Abstract

Extracted data

Related items

Related items