Security Metrics

F or years, we’ve been trying to measure se-curity so that we can have objective ways to determine, for example, whether confiden-tial information is really kept confidential, whether data integrity is preserved, whether services will remain available only for authorized users, and whether systems meet stringent reliability require-ments.1 After all, we have many security-related deci-sions to make and often ask a variety of questions that security measurement might answer—from “How much more secure is this application, system, or net-work after we make these changes? ” to “What’s the right mix of controls that will get the most security for a given investment?” Shari Lawrence Pfleeger showed how some exist-ing metrics can help depict our systems ’ immunity and resilience,2 but more research is needed. For example, current measures depend greatly on threat models and attack types; they offer little information when the threat environment changes. Our rich and growing literature in security measurement (see the sidebar “A Brief History of Security Measurement”) is still in-adequate for answering important security questions. Why is security measurement so hard? Security mea-surement is different from—and harder than—some other kinds of measurement for at least nine reasons: • We can’t test all security requirements. • Environment, abstraction, and context affect security. • Measurement and security interact. • No system stands alone. • Security is multidimensional, emergent, and irreducible. • The adversary changes the en-vironment. • Measurement is both an expectation and an organi-zational objective. • We’re overoptimistic. • We perceive gain differently from loss. Let’s examine each one in turn