Easy fault injection and stress testing with FAIL-FCI

In a network consisting of several thousands computers, the occurrence of faults is unavoid- able. Being able to test the behavior of a distributed program in an environment where we can control the faults (such as the crash of a process) is an important feature that matters in the deployment of reliable programs. In this paper, we extend FAIL-FCI (for Fault Injection Language, and FAIL Cluster Im- plementation, respectively), a software tool that permits to elaborate complex fault scenarios in a simple way, while relieving the user from writing low level code. In particular, we show that not only we are able to fault-load existing distributed applications (as used in most cur- rent papers that address fault-tolerance issues), we are also able to inject qualitative faults, i.e. inject speci¯c faults at very speci¯c moments in the program code of the application under test. Finally, and although this was not the primary purpose of the tool, we are also able to inject speci¯c patterns of workload, in order to stress test the application under test. Interestingly enough, the whole process is driven by a simple uni¯ed description language, that is totally independent from the language of the application, so that no code changes or recompilation are needed on the application side