Computational Lighting Design And Image Filtering For Material Enhancement

Photography provides a powerful tool for depicting the world around us by capturing the intricate relationship between light and materials. Indeed, much of the art and craft of photography is based on understanding how light interacts with surface properties, and how this interaction gets captured by the camera. The recent digitization of photography, and in particular the point and shoot paradigm, has made photography accessible to millions of people by automating a large part of the decision making process. For example, modern cameras provide algorithms to automatically set many of the tunable parameters, such as white balance, exposure time and aperture size that are appropriate to a given scene. However, there is little or no support for the end-user when it comes to helping them to light or manipulate surface appearance in a scene. A good photographer is never passive towards the lighting in the scene and often manipulates the subject of interest in order to emphasize certain properties. For example, in portrait photography lights can be positioned to control the appearance of wrinkles, or makeup can be used to hide variation in skin color. One key challenge is that lighting design and manipulation of surface properties are considered highly labor intensive, manual tasks that require specialized equipment and lots of preparation, which is beyond the skills of casual photographers. In this thesis, we develop a family of new computational tools to make such advanced photographic tasks more accessible to novice users. We call our broad approach computational lighting and material design. Our approach is based on entirely image-based, post-process techniques combined with carefully designed user interactions that allow novice users to explore a non-trivial space of solutions, based on common goals in photography. As we show in this thesis, the results are often much more effective and easy to produce than what could be achieved using traditional techniques. We provide practical methods for three photographic tasks in particular. 1. Lighting design. For lighting design, we propose a computational approach that partially automates the process, by allowing novices to uniformly walk around a static scene with a single light source. Then, we describe a set of optimizations to assemble the input lights to create a few basis lights that correspond to common goals pursued by photographers, e.g., accentuating edges and curved regions. We also introduce modifiers that capture standard photographic tasks, e.g., to alter the lights to soften highlights and shadows, akin to umbrellas and soft boxes. This approach to lighting allows the photographer to achieve sophisticated lighting without a complicated manual setup. 2. Multi-lights white balance. When a scene has a mixture of lights with different colors, the common white balance problem becomes much more challenging. In this thesis, we propose a solution to the ill-posed mixed light white balance problem, based on user guidance. We allow users to scribble on a few regions that should have the same color, indicate one or more regions of neutral color, and select regions where the current color looks correct. Then, we reformulate the spatially varying white balance problem as a sparse data interpolation problem in which the user scribbles form constraints. We demonstrate that our approach can produce satisfying results on a variety of scenes with intuitive scribbles and without any knowledge about the lights. 3. Material editing. Our third work addresses the problem of surface manipulation, which is a common pre-processing step in the professional photography pipeline, that often requires a non-trivial physical interaction with the object. For example, in food photography glycerine may be used to give the food a more fresh and appealing look. In this thesis, we reformulate the problem as a post-process step, where the goal is to manipulate material properties after the image has been taken. For example, to increase shininess or to decrease aging cues, such as wrinkles and blemishes. We design and study a set of 2D image operations, based on multi-scale image analysis, that are easy and straightforward, and that can consistently modify perceived material properties. Through user studies, we identify a set of operators that yield consistent subjective effects for a variety of materials and scenes. The computational methods presented in this dissertation have made a step towards automating advanced photographic tasks by simplifying the required user preparation and interaction. Remaining challenges include developing more general basis lights to support a wider range of photographic objectives (such as portrait photography) as well as finding more powerful methods for acquisition of both static and dynamic scenes. We believe that our work in image-based material editing would spur others to explore this important area. A datadriven approach for both lighting design and material editing is a promising future direction of leveraging state-of-the art machine learning algorithms for developing fully automatic solutions. We also believe the techniques introduced in this dissertation can provide valuable insights for developing computational methods for lighting design and material alternation for both real and CG scenes