Understanding Machine Learning Performance with Experiment Databases (Het verwerven van inzichten in leerperformantie met experiment databanken)

Research in machine learning and data mining can be speeded up tremendously by moving empirical research results out of people s heads and labs, onto the network and into tools that help us structure and filter the information. The massive streams of experiments that are being executed to benchmark new algorithms, test hypotheses or model new datasets have many more uses beyond their original intent, but are often discarded or their details are lost over time.In this thesis, we developed a framework to automatically export experiments to experiment databases, databases specifically designed to collect all the details on large numbers of past experiments, performed by many different researchers, and to compose queries about almost any aspect of the behavior of learning algorithms. They can be set up for personal use, to share results within a lab, or to build community-wide repositories.Following similar developments in several other sciences, we first define a formal domain model, an ontology, for experimentation in machine learning, after which we use this ontology to define an XML-based languageto exchange experiments, as well as a database model to organize all submitted results.Finally, we demonstrate how such databases can be queried to meta-learn: to gain new insight into learning algorithm behavior. Using often no more than a single database query, we obtained surprising new results. This includes detailed rankings of learning algorithms, insight into the behavior of ensemble methods, suggestions for improvement of certain algorithms, learning curve analyses and insight into the bias-variance behavior of algorithms. We also built meta-models for predicting and explaining the suitability of learning algorithms and parameter settings.This illustrates that much can be learned by collecting and reusing past machine learning experiments, and that building experiment databases to query for them provides an effective way of tapping into this information, often yielding surprising new insight or generating interesting newresearch questions.1 Introduction 1.1 Machine Learning 1.2 Knowledge Discovery: An illustration 1.3 Meta-learning 1.4 Experiment Databases 1.5 Goal of this thesis 1.6 Roadmap and Contributions Prelude 2 Meta-Learning 2.1 Introduction 2.2 Flavors ofmeta-learning 2.3 An algorithm selection framework 2.4 The data meta-feature space F 2.5 The algorithm meta-feature space G 2.6 The problem spaces P and P' 2.7 The algorithm spaces A and A' 2.8 The performance space Y 2.9 The meta-learner S 2.10 Summary and conclusions 3 Intelligent Knowledge Discovery Support 3.1 Introduction 3.2 Expert systems 3.3 Meta-models 3.4 Planning 3.5 Case-based reasoning 3.6 Querying 3.7 A new platform for intelligent KD support 3.8 Conclusions Prelude Conclusions Part I Organizing Machine Learning Information Outline Part I 4 Experiment Databases 4.1 Motivation 4.2 Experiment databases 4.3 Experiment Repositories in e-Sciences 4.4 Designing Experiment Databases 4.5 Using Experiment Databases 4.6 Conclusion 5 The Exposé Ontology 5.1 Introduction 5.2 Previous work 5.3 The Exposé Ontology 5.4 Conclusions 6 The ExpML Markup Language 6.1 Minimal Information about an ML Experiment 6.2 From Ontology to Markup Language 6.3 Conclusions 7 Anatomy of an Experiment Database 7.1 From Ontology to Database Model 7.2 Populating the Database 7.3 Conclusions Conclusions Part I Part II Learning From the Past Outline Part II 8 Interfaces: Hiding the Complexity 8.1 Software Components 8.2 Interfaces 8.3 The graphical query too 8.4 Conclusions 9 Exploring learning behavior 9.1 Model-level analysis 9.2 Data-Level analysis 9.3 Method-Level Analysis 9.4 Conclusions Conclusions Part II Finale 10 Summary and Future Work 10.1 Summary 10.2 Future Work Appendix A Simple, Statistical and Information-Theoretic Data Properties A.1 Some notes on notation A.2 Simple features A.3 Normality-related features A.4 Redundancy-related features A.5 Attribute-target associations A.6 Algorithm-specific properties A.7 Propositional versus relational features Bibliography Publication Listnrpages: 258status: publishe