CONCRETE: A benchmarking framework to CONtrol and Classify REpeatable Testbed Experiments

Abstract. The global research community has been recently directed towards implementation approaches and evaluation through experimentation in network-ing testbeds. A unique problem that arises during experimentation in such envi-ronments is that several different factors may impact the monitored performance of networks under consideration. More specifically, the major factor that signif-icantly impacts performance is the effect of interference that is generated either from testbed nodes that simultaneously transmit or from collocated commercial devices belonging to external networks. Moreover, as the level of interference significantly varies over time, the monitored performance may present high vari-ation among several executions of the same experiment. In order to ensure stable experimental conditions, specific tools have to be developed that would be able to detect whether the gathered results have been affected by external factors. In this work, we present a novel benchmarking framework, developed within the collaboration of CREW and OpenLab EU FP7 projects. The developed frame-work provides for proper evaluation of experimentally gathered results and thus enables the experimenter to arrive at solid conclusions regarding the validity of the conducted experiments.