Joint Information Augmentation of Road Maps, Aerial Images and Ground Images

Extracting information about roads is important for many applications, such as infrastructure monitoring, traffic management, urban planning, vehicle navigation, realistic driving simulations, and it will be essential in the future for autonomous driving cars. The most straightforward way to express the road information is through a detailed map. Collecting road information on the spot (the ground) for a larger area is labor and time intensive as the surveyor has to visit the whole area of interest. Aerial images provide a rich Information source to survey and map a larger area remotely, but if the images are interpreted manually, this process typically needs long, tedious work. Analyzing the aerial images automatically can make the analysis of remote sensing Images much more efficient. In the ideal case a complete map could be created from an aerial Image without any human intervention. However, this is a very difficult task, for many scenes the aerial view does not provide enough information and even humans can only hardly Interpret the image. The majority of available maps were created by also incorporating local Surveys or other information sources to resolve uncertainties present in the remote sensing images. This makes the maps a great tool to include ground measurements and prior knowledge into the analysis of aerial images. The goal of this thesis is to apply the already existing road maps jointly with aerial and ground images in fully automatic workflows. The information missing in the map is augmented with information present in the aerial and ground images and vice versa. Three problems are investigated for the information exchange between aerial imagery and road maps and one problem where ground images are also included. This is a cummulative dissertation, the four problems are addressed by four peer-reviewed papers: Fast multiclass vehicle detection on aerial images: Conventional maps describe the static characteristic of the roads, however, the dynamic traffic conditions on the roads are an important input for navigation, as well as planning and managing the infrastructure. A single aerial image with appropriate resolution enables to count the vehicles and extract their Location and orientation. This already allows to estimate the utilization of the roads and parking spaces. By using multiple image frames, the speed of the vehicles can also be measured to estimate the traffic flow over the area covered by the aerial imagery. The key problem to solve is the reliable and quick detection of the vehicles in the images. Toward this goal, a fast and high-performing vehicle detector is proposed for aerial images. In contrast to previous methods, this estimates also the direction and the category class of the vehicle. The method does not need an orthorectified image, it can work on an original image frame without a prior of the region of interest, e.g. a road mask. The performance of the method is examined on a new dataset containing several thousand vehicles. The fast speed of this detector makes it suitable for real-time airborne road traffic extraction. Large scale aerial image sequence geolocalization with road traffic as invariant feature: Aerial images are the most practicable if their geolocation is known and a pixel coordinate in the image can be transformed to a world coordinate. A novel approach is proposed to extract the geolocation of aerial images by using only a road map and the traffic visible in the images. In contrast to the three other tasks, here the aerial image is augmented with extra information from the map, i.e. the geolocation. The road network pattern over a larger area tends to be so unique that it can be used to recognize the geolocation. It is not even needed to extract the complete road network in the images, already a fraction is enough. Instead of detecting the roads directly, the traffic in the scene is extracted in form of tracks of moving vehicles. Using these tracks the images can be successfully geolocalized in a search area of 22500 km2 containing 32000 km of streets in the Munich metropolitan area. This method could replace the expensive and heavy Global Positioning System (GPS) + Inertial Measurement Unit (IMU) systems used for creating geolocalized aerial images. Such systems will ii be particularly important for Unmanned Aerial Vehicles (UAVs), where the weight and cost of the system is more critical. This method localizes the images similar as humans localize themselves, based only on visual information and a road map. Enhancing road maps with the street width by parsing aerial images: Currently used road maps are intended for the navigation of humans. The roads as stored as centerlines with Connections to other roads without providing detailed information about the physical dimensions of the road. Having access to the width attributes of the roads would be important for infrastructure planning, creating realistic simulations over roads and they can improve the automatic scene understanding of autonomous vehicles. To address these demands, a method is presented to automatically extract the road width while also considering misalignments between the road network and the aerial image. Instead of formulating the problem as a pixelwise semantic segmentation, the problem is defined as one of inference in a Markov Random Field (MRF) reasoning about the road parameters directly. This makes it very fast, more robust and the topology of the road network is preserved. Experiments are conducted on three datasets, over Bavaria, Germany, over the city of Karlsruhe, Germany and Google Earth images over various locations around the world. The proposed method outperforms the state of the art in both speed and accuracy. The ability of the detailed maps to improve the scene understanding of ground images is demonstrated on the KITTI autonomous driving dataset. Fine-grained road segmentation by parsing ground and aerial images: The estimation of the road width is extended to extract the fine-grained road layout by estimating the number and width of lanes plus the presence and width of parking spots and sidewalks. Importantly, the proposed approach applies existing road maps, aerial images and ground Images jointly. The problem is formulated as one of inference in an MRF reasoning about the road layout as well as the alignment between the aerial image, the map and the ground Image sequence in a joint energy function. The MRF takes features extracted from the images by deep learning as data terms and formulates the constraints on the lane sizes and the road layout as pairwise potentials. This allows robust estimation also in case when the image evidence (e.g. lane markings) is not visible or is missing. The alignment of ground and Aerial images is necessary as even when applying sophisticated GPS-IMU systems, registration Errors can still occur. The registration of the ground images to the map can also function as precise self-localization of the vehicle within the road, an important task for path planning and safe driving. Experiments are performed on a dataset including annotated aerial and ground images over the same area