Generic simulation metamodeling: Towards an experimental design environment

Simulation models are used in many fields to experiment with real-world systems to gain insight into their behaviour. Experimenting with simulation models can be time consuming and costly. In order to save costs, experimenters reduce their scope of research or otherwise conduct less thorough investigations into the behaviour of a simulation model, increasing the risk of overlooking valuable information. A possible solution to this particular problem is metamodeling. Metamodeling is mentioned in literature as a way to reduce the number of required experiments. Only a reduced sample of experiments is run, after which the other results are estimated by certain interpolation techniques. While metamodeling is not new, it is mostly used in academic settings where metamodels are specifically tailored and designed for a specific simulation model or set of experiments. In commercial projects metamodeling has not been used often because of the lack of expertise that is required. A generic, automatic metamodeling tool or environment that allows simulation users to utilise the power of metamodeling can decrease the experimentation time for commercial projects as well as increase the quality of recommendations or decisions based on the results. The aim of this thesis is to find a metamodeling technique that allows this kind of generic use as well as investigating how this technique should work in practice. To achieve the latter, an experimental design environment, or tool, has been designed. By designing an experimental design environment with metamodeling capabilities, the setting in which metamodeling can benefit experimenters can be understood and more insight is given into the challenges surrounding the search for a generic metamodeling technique. The experimental design environment is designed focusing on the user’s pursuit for answers to questions about the system’s behaviour. This is done by focusing on three steps; selecting the input factors the user wants to vary with a certain range, setting up the simulation run (which behind the scenes uses metamodeling), and lastly viewing and comparing results. The way the experimental design environment is designed, allows the user to quickly understand the relationships between the input factors of a system and the results. The challenges that arise when choosing a metamodeling technique that can be used generically and automatically with any simulation model are finding the right metamodeling technique, using the right sampling technique that reduces the number of experiments, and using the right method to assess the quality of the final metamodel. When looking for a metamodeling technique that can be use generically and automatically with any simulation model, the techniques polynomial regression, spline, kriging and artificial neural networks were chosen based on literature. Based on a multi-criteria analysis two techniques were selected to experiment with: Polynomial regression metamodeling and Kriging. These two metamodeling techniques, both having different characteristics as well as using different sampling techniques for reducing the number of experiments, were tested for their accuracy in various situations. Both techniques were applied on eight sets of result data, using a reduced set of this data (using sampling) to create a metamodel in order to estimate the remainder of the data. By comparing the estimated data with the original data, the performance of both techniques was measured. Based qualitative and quantitative analyses it can be concluded that Kriging metamodeling is a suitable technique for generic use in a commercial simulation project. The strongest advantages of Kriging over other metamodeling techniques are the fact that it can be used without any prior knowledge of the behaviour of a simulation model, its overall accuracy and its ability to handle inherent erratic behaviour of discrete event simulation models. These characteristics make it possible to use Kriging as the single metamodeling technique to handle all kinds of simulation models, regardless of the expected behaviour of the responses, without the need for specific metamodeling or simulation expertise. Polynomial regression metamodeling has the drawback of requiring knowledge about the behaviour of the system it tries to estimate, in order to choose the right order for the polynomial it uses. Furthermore it was determined that it was less sufficient in handling inherent erratic behaviour of discrete event simulation models. While metamodeling does not always provide highly accurate results, it can be significantly valuable to experimenters and simulation users. Providing fast results, metamodeling can be used to initially scan a certain area of the design space before focusing on an area of interest. Requiring no metamodeling expertise, this can lead to a significantly reduced experimentation time for commercial simulation projects as well an increased quality of overall project results.Systems EngineeringTechnology, Policy and Managemen