On security singularities

In future studies involving artificial intelligence, the so-called technological singularity is a key theme. It refers to a hypothetical point in the future where technological progress becomes automated through the creation of a new form of intelligence. Under the assumption of adversarial behaviour, this could pose an existential threat to humanity. More modestly, singularities and tipping points refer to thresholds beyond which the behaviour of a system changes in a qualitative way. The nonlinearity of the behaviour causes existing control mechanisms to become obsolete, guiding the system towards a new balance, if this exists. In this paper, we ask the question to what extent the notions of singularity and tipping point can contribute to an analysis of security in 2038. Can we expect to have seen such phenomena in twenty years time, and will they have changed our perception of what security entails? Or are they useless forms of speculation diverting our attention away from the day-to-day best practices that are needed to keep our basic security up-to-date? We discuss examples of singularity-style developments, characterise them in terms of acceleration mechanisms and discontinuities, and discuss whether and how these characteristics should be used to prepare ourselves. We conclude that a broad discussion on potential security singularities and associated general adaptation strategies is more useful than focusing on one big singularity.© Wolter Pieters 2018. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive version was published by ACM in NSPW '18 Proceedings of the New Security Paradigms Workshop, https://doi.org/10.1145/3285002.3285008