Validating wireless network simulations using direct execution

textSimulation is a powerful and efficient tool for studying wireless networks. Despite the widespread use of simulation, particularly in the study of IEEE 802.11-style networks (e.g., WLAN, mesh, and ad hoc networks), doubts about the credibility of simulation results still persist in the research community. These concerns stem, in part, from a lack of trust in some of the models used in simulation as they do not always accurately reflect reality. Models of the physical layer (PHY), in particular, are a key source of concern. The behavior of the physical layer varies greatly depending on the specifics of the wireless environment, making it difficult to characterize. Validation is the primary means of establishing trust in such models. We present an approach to validating physical layer models using the direct execution of a real PHY implementation inside the wireless simulation environment. This approach leverages the credibility inherent to testbeds, while maintaining the scalability and repeatability associated with simulation. Specifically, we use the PHY implementation from Hydra, a software-defined radio testbed, to validate the sophisticated physical layer model of a new wireless network simulator, called WiNS. This PHY model is also employed in other state-of-the-art network simulators, including ns-3. As such, this validation study also provides insight into the fidelity of other wireless network simulators using this model. This physical layer model is especially important because it is used to represent the physical layer for systems in 802.11-style networks. Network simulation is a particularly popular method for studying these kinds of wireless networks. We use direct-execution to evaluate the accuracy of our PHY model from the perspectives of different protocol layers. First, we characterize the link-level behavior of the physical layer under different wireless channels and impairments. We identify operating regimes where the model is accurate and show accountable difference where it is not. We then use direct-execution to evaluate the accuracy of the PHY model in the presence of interference. We develop "error-maps" that provide guidance to model users in evaluating the potential impact of model inaccuracy in terms of the interference in their own simulation scenarios. This part of our study helps to develop a better understanding of the fidelity of our model from a physical layer perspective. We also demonstrate the efficacy of direct-execution in evaluating the accuracy of our PHY model from the perspectives of the MAC and network layers. Specifically, we use direct-execution to investigate a rate-adaptive MAC protocol and an ad hoc routing protocol. This part of our study demonstrates how the semantics and policies of such protocols can influence the impact that a PHY model has on network simulations. We also show that direct-execution helps us to identify when a model that is inaccurate from the perspective of the PHY can still be used to generate trustworthy simulation results. The results of this study show that the leading physical layer model employed by WiNS and other state-of-the-art network simulators, including ns-3, is accurate under a limited set of wireless conditions. Moreover, our validation study demonstrates that direct-execution is an effective means of evaluating the accuracy of a PHY model and allows us to identify the operating conditions and protocol configurations where the model can be used to generate trustworthy simulation results.Electrical and Computer Engineerin