A Roadmap for Benchmarking in Wireless Networks

Experimentation is evolving as a viable and realistic performance analysis approach in wireless networking research. Realism is provisioned by deploying real software (network stack, drivers, OS), and hardware (wireless cards, network equipment, etc.) in the actual physical environment. However, the experimenter is more likely to be dogged by tricky issues because of calibration problems and bugs in the software/hardware tools. This, coupled with difficulty of dealing with multitude of hardware/software parameters and unpredictable characteristics of the wireless channel in the wild, poses significant challenges in the way of experiment repeatability and reproducibility. Furthermore, experimentation has been impeded by the lack of standard definitions, measurement methodologies and full disclosure reports that are particularly important to understand the suitability of protocols and services to emerging wireless application scenarios. Lack of tools to manage large number experiment runs, deal with huge amount of measurement data and facilitate peer-verifiable analysis further complicates the process. In this paper, we present a holistic view of benchmarking in wireless networks and formulate a procedure complemented by step-by-step case study to help drive future efforts on benchmarking in wireless network applications and protocols