Software, the Urn Model, and Failure

Abstract: IEC 61508-7:2010 Annex D explicates some features of the statistical evaluation of software through its operational history. But it raises many questions. One is: how does the traditional urn model apply, if at all, to software? A second is: to what kinds of software does the reasoning in Annex D apply? A third is: do the confidence-interval numbers apply to all distributions of inputs? Recently, I have experienced reliability-engineering experts giving wrong answers to the second and third questions. It seemed worthwhile to explain correct answers in terms understandable also by non-professionals. This paper attempts to do so. IEC 61508-7:2010 Annex D includes a brief (and, some would say, obscure) introduction to the statistics of evaluating the reliability of software. Annex D relies on software execution being construed as a Bernoulli process, which is a sequence of Bernoulli trials (a “random ” operation resulting in one of two outcomes, traditionally denoted“success ” or “failure”, with a certain probability of “success”) with fixed probabilities of success for each trial [BedCoo01 Chapter 12]. This conception of software reliability has been developed in the software engineering literature for over forty years, and has had success in assessing the reliability of software for UK nuclear power plant protection systems using statistical methods [Lit10]. It is the basis for some of the most well-known literature in the power and pitfalls of assessing software statistically for ultra-high reliability [LitStr93,LitStr11]. The figures in Annex D Table D.1 are based upon calculations considering software failure as a Bernoulli process, defined to be a sequence of independent Bernoulli trials with a constant probability of success per trial [Lit10]. The salient information from Table D.1 is reproduced below in Tables 1 and 2. They are discussed further in [LadLit15]. Acceptable probability of failure to perform design function on demand Number of observed demands without failure for a confidence level of 99% Number of observed demands without failure for a confidence level of 95