Increased Transparency and Effective Communication through the Design of Simulation Experiments

Iris Lorscheid and Matthias Meyer

Simulation has become more and more established as a research method in the social sciences over the past years. Nevertheless, the extent of acceptance varies by field and some disciplines still hesitate to use simulation extensively. At least two possible reasons explain this situation. First, many may still perceive simulation models as a black box because simulation models and the analyses of their behavior are often not described in an exhaustive way. Second, the reluctance to accept simulation might also stem from the difficulty to communicate simulation results. To overcome these problems, this tutorial proposes a systematic and ideally standardized procedure for simulation research to address these two important challenges. Systematic Design of Simulation Experiments can increase (1) the transparency of simulation model behavior and (2) the effectiveness of reporting simulation results. Within this tutorial, a systematic procedure for simulation data analysis based on Design of Experiments (DOE) principles will be presented and exemplarly demonstrated by an example. This process supports the modeler by guiding the analysis of simulation models to produce simulation data in a systematic way. Above, concrete output templates for sharing and communicating simulation results in an efficient way are introduced. Within the tutorial, some challenges along the analysis process are addressed, such as the question of how to define parameter settings for a simulation experiment, the definition of a stopping criterion for simulation runs as well as the needed number of runs per simulation setting in stochastic simulations, and how to reveal interaction-effects.