The Geometry of Projective Reconstruction I: Matching Constraints and the Joint Image

Circulated in 1995. Accepted subject to revision to IJCV in 1995, but never completedThis paper studies the geometry of perspective projection into multiple images and the matching constraints that this induces between the images. The combined projections produce a 3D subspace of the space of combined image coordinates called the joint image. This is a complete projective replica of the 3D world defined entirely in terms of image coordinates, up to an arbitrary choice of certain scale factors. Projective reconstruction is a canonical process in the joint image requiring only the rescaling of image coordinates. The matching constraints tell whether a set of image points is the projection of a single world point. In 3D there are only three types of matching constraint: the fundamental matrix, Shashua's trilinear tensor, and a new quadrilinear 4 image tensor. All of these fit into a single geometric object, the joint image Grassmannian tensor. This encodes exactly the information needed for reconstruction: the location of the joint image in the space of combined image coordinates