The Big Picture: Loss Functions at the Dataset Level

Loss functions play a key role in machine learning optimization problems. Even with their widespread use throughout the field, selecting a loss function tailored to a specific problem is more art than science. Literature on the properties of loss functions that might help a practitioner make an informed choice about these loss functions is sparse. In this thesis, we motivate research on the behavior of loss functions at the level of the dataset as a whole. We begin with a simple experiment that illustrates the differences in these loss functions. We then move on to a well-known attribute of perhaps the most ubiquitous loss function, the squared error. We will then characterize all loss functions that exhibit this property. Finally we end with extensions and possible directions of research in this field