An essential component of localization and mapping is visual matching which associates individual 2D and 3D observations of the same local structures in a real-world 3D space. As the precondition of geometry computation, the quality of visual matching is critical to the robustness and accuracy of localization and mapping. According to the dimension of data that matching algorithms manipulate, we classify the visual matching methodology into three categories: 2D-2D matching, 3D-3D matching and 2D-3D matching, which are addressed in the thesis within the context of image matching, point cloud registration and camera relocalization, respectively. While the three visual matching problems have been widely researched, factors such as large view scale changes, overwhelming outliers, la...[ Read more ]

An essential component of localization and mapping is visual matching which associates individual 2D and 3D observations of the same local structures in a real-world 3D space. As the precondition of geometry computation, the quality of visual matching is critical to the robustness and accuracy of localization and mapping. According to the dimension of data that matching algorithms manipulate, we classify the visual matching methodology into three categories: 2D-2D matching, 3D-3D matching and 2D-3D matching, which are addressed in the thesis within the context of image matching, point cloud registration and camera relocalization, respectively. While the three visual matching problems have been widely researched, factors such as large view scale changes, overwhelming outliers, lack of salient features and so on remain great challenges. In this thesis, we propose several approaches to advance the capacity of visual matching algorithms despite the effect of these challenging factors. Specifically, we first improve the scale invariance of 2D image matching by leveraging the principles of the scale space theory, which measures image similarities and identifies local feature correspondences while being invariant to significant view scale changes. Second, an outlier rejection strategy is developed to elevate the robustness of point cloud registration by inferring the existence of outliers from the spatial organizations of 3D matches through belief propagation. Third, we propose to learn temporal camera relocalization with recursive 2D-3D matching based on Kalman filters to overcome the potential difficulties of single-view matching. Finally, to reach a localization or mapping result with better accuracy, we present a stochastic bundle adjustment algorithm which refines the geometry globally at scale from all the visual associations. Extensive experimental evaluations on standardized benchmarks demonstrate the superior performance of the proposed visual matching and optimization algorithms compared to the state of the art. The integration of our visual matching methods has been proved to enhance the robustness and accuracy of the developed localization and mapping systems.