Toward Idealized Decision Theory

This paper motivates the study of decision theory as necessary for aligning smarter-than-human artificial systems with human interests. We discuss the shortcomings of two standard formulations of decision theory, and demon-strate that they cannot be used to describe an idealized decision procedure suitable for ap-proximation by artificial systems. We then ex-plore the notions of strategy selection and logical counterfactuals, two recent insights into deci-sion theory that point the way toward promis-ing paths for future research.