Augmenting Visual Memories

Thesis (Ph.D.)--University of Washington, 2023We often rely on photographs and other forms of physical and digital media for capturing, preserving, and sharing the visual experiences that we encounter throughout our lives. While many common forms of media (like photographs) are able to effectively capture and reproduce detailed representations of our visual experiences, much of what we remember from these experiences is usually not captured in a photograph. When compared to our (incredibly rich) visual memories, photographs often lack a lot of crucial contextual information that is necessary to our understanding a particular event, e.g., details that help us answer questions like "where did this take place?", "what led to this moment?", and "what happened next?". Unlike human memories, which include information about our unbounded, three-dimensional, and ever-changing surroundings, photographs only capture a fixed field-of-view and a fixed point in time. Although recent technological developments have brought about a number of ways for capturing more detailed, immersive, and realistic depictions of our visual memories (such as 3D cameras and 360-degree videos), the vast majority of our past memories were captured by older technologies, such as the still photograph. As such, these memories remain permanently preserved in outdated forms of media, limiting in the realism, immersion, and fidelity with which they are able to portray our past experiences. In this thesis, I investigate methods for elevating this type of legacy imagery to more modern and immersive visual experiences by recovering or synthesizing the context that was lost during the capture process. First, I provide a survey of modern forms of media, specifically focusing on those which provide more immersion or visual fidelity than the traditional still photograph. I also review methods for augmentation of legacy captures, i.e., methods that turn older, less immersive content into modern forms of media. Finally, I present three novel techniques for enabling automatic augmentation of captured memories: (1) a method for automatically turning captured photos into high-quality video sequences; (2) a method for 3D reconstruction of previously captured smartphone camera videos, enabling different forms of 3D interaction, like virtual object insertion and light manipulation; and (3) a method for large-scale 3D structure from motion targeted towards handheld smartphone footage, which enables 3D reconstruction and visualization of large scenes, like buildings and city blocks that would otherwise be difficult to capture