Distinguishing refracted features using light field cameras with application to structure from motion

To be effective, robots will need to reliably operate in scenes with refractive objects in a variety of applications; however, refractive objects can cause many robotic vision algorithms, such as structure from motion, to become unreliable or even fail. We propose a novel method to distinguish between refracted and Lambertian image features using a light field camera. While previous refracted feature detection methods are limited to light field cameras with large baselines relative to the refractive object, our method achieves comparable performance, and we extend these capabilities to light field cameras with much smaller baselines than previously considered, where we achieve up to 50% higher refracted feature detection rates. Specifically, we propose to use textural cross correlation to characterize apparent feature motion in a single light field, and compare this motion to its Lambertian equivalent based on 4-D light field geometry. For structure from motion, we demonstrate that rejecting refracted features using our distinguisher yields lower reprojection error, lower failure rates, and more accurate pose estimates when the robot is approaching refractive objects. Our method is a critical step toward allowing robots to operate in the presence of refractive objects.