Reflectance and angular luminance for material recognition and segmentation

Real world scenes consist of surfaces made of numerous materials, such as wood, marble, dirt, metal, ceramic and fabric, which contribute to the rich visual variation we find in images. Materials play a fundamental role in numerous applications including asphalt for automated driving, tree-cover in fire risk assessment, path material (grass vs concrete) for robot navigation, and landcover albedo analysis for climate studies. This thesis is dedicated to developing compact and robust material and texture representations for material recognition and segmentation. Material properties affect the spatial variation of surface appearance and the angular variation of reflectance with respect to both view and illumination. Modeling the apparent or latent characteristic appearance of different materials is essential to robustly recognize them in images. We build representations that capture the intrinsic invariant properties of the surface, which enables fine-grained material recognition and segmentation. In particular, this thesis develop the following methods: 1. Differential Angular Imaging: We present a new measurement method called differential angular imaging where a surface is imaged from a particular viewing angle $v$ and then from an additional viewpoint $v+delta$. The motivation for this differential change in viewpoint is improved computation of the angular gradient of intensity $partial{I_v}/partial{v}$. We develop a framework for differential angular imaging, where small angular variations in image capture provide an enhanced appearance representation and significant recognition improvement. We build a large-scale material database, Ground Terrain in Outdoor Scenes (GTOS) database, geared towards real use for autonomous agents. The database consists of over 30,000 images covering 40 classes of outdoor ground terrain under varying whether and lighting conditions. 2. Deep Texture Manifold: For ground terrain recognition, many class boundaries are ambiguous. Therefore, it is of interest to find not only the class label but also the closest classes, or equivalently, the position in the manifold. We present a texture network called Deep Encoding Pooling Network (DEP) for the task of ground terrain recognition. Recognition of ground terrain is an important task in establishing robot or vehicular control parameters, as well as for localization within an outdoor environment. The architecture of DEP integrates orderless texture details and local spatial information. The resultant network shows excellent performance not only for GTOS-mobile, but also for more general databases (MINC and DTD). Based on DEP, we introduce a new texture manifold method, DEP-manifold, to find the relationship between newly captured images and images in dataset. 3. Texture Encoded Angular Network: We develop a novel approach for material recognition called texture-encoded angular network (TEAN) that combines deep encoding pooling of RGB information and differential angular images for angular-gradient features for the task of ground terrain recognition. With this novel network architecture, we extract characteristics of materials encoded in the angular and spatial gradients of their appearance. Our results show that TEAN achieves recognition performance that surpasses single view performance and standard (non-differential/large-angle sampling) multiview performance. 4. Angular Luminance: We utilize per-pixel angular luminance distributions as a key feature in discriminating the material of the surface. The angle-space sampling in a multiview image sequence is an unstructured sampling of the underlying reflectance function of the material. For real-world materials there is significant intra-class variation that can be managed by building a Angular Luminance Network (AngLNet). This network combines new angular reflectance cues from multiple images with more traditional spatial cues as in fully convolutional networks for semantic segmentation. We demonstrate the increased performance of AngLNet over prior state-of-the-art in material segmentation from drone video sequences and satellite imagery.Ph.D.Includes bibliographical reference