Stochastic prediction in sequential high-dimensional observation space

Predicting the future in real-world settings, particularly from raw sensory observations such as images, is exceptionally challenging. Real-world events can be stochastic and unpredictable, and the high dimensionality and complexity of natural images require the predictive model to build an intricate understanding of the natural world. Many existing predictive methods tackle this problem by making simplifying assumptions about the environment. One common assumption is that the outcome is deterministic and there is only one plausible future. This can lead to low-quality predictions in real-world settings with stochastic dynamics. In this thesis, we study the importance of stochasticity in predicting high-quality predictions of the raw sequential observations. We develop a stochastic variational video prediction method that predicts a different possible future for each sample of its latent variables. We also provide an alternative method based on normalizing flows. To the best of our knowledge, these models are the first to provide an effective stochastic multi-frame prediction for real-world videos. We demonstrate the capability of these methods in predicting detailed future frames of videos on multiple real-world datasets, both action-free and action-conditioned. We also illustrate how such methods can improve the performance of autonomous agents where future prediction is a core required capability. We illustrate how these predictive models can be used for planning in real and simulated robotic tasks as well as improving the sample efficiency in model based reinforcement learning. We also show how similar stochastic techniques can be applied in other areas where stochasticity can be useful such as real-time style transfer