The thriving large-scale 3D reconstruction techniques make city-scale models available and find numerous applications in 3D mapping and reverse engineering. In this thesis, the focus is on urban scene segmentation, recognition and remodeling for the sake of understanding a modeled cityscape and in turn refining the urban models.

We first generate joint semantic segmentation for urban images and 3D data by surmounting the barrier of time-consuming manual annotation of training data. With the reconstructed 3D geometry, the training data are initialized resorting to urban priors. We segment the input images and 3D data into semantic categories simultaneously by employing a novel joint object correspondence graph which purifies the automatically generated training samples.

...[ Read more ]

The thriving large-scale 3D reconstruction techniques make city-scale models available and find numerous applications in 3D mapping and reverse engineering. In this thesis, the focus is on urban scene segmentation, recognition and remodeling for the sake of understanding a modeled cityscape and in turn refining the urban models.

We first generate joint semantic segmentation for urban images and 3D data by surmounting the barrier of time-consuming manual annotation of training data. With the reconstructed 3D geometry, the training data are initialized resorting to urban priors. We segment the input images and 3D data into semantic categories simultaneously by employing a novel joint object correspondence graph which purifies the automatically generated training samples.

Our second task is to recognize and segment individual building objects. We extract building objects from the orthographic view and then fuse the decomposed roof segmentation to 3D models through a structure-aware flooding algorithm. Each building footprint is abstracted by a set of structural primitives that best fit to the model geometry and also conform to the discovered global regularities. We extend our patchwork assembly to recognize more detailed but structured objects among the architectures, such as windows, and then reassemble them based on recognized grammars. A facade structure is recognized at the object level with a structure-driven Monte Carlo Markov Chain sampler. The solution space is explored with high efficiency because the structure-driven sampler accelerates convergence by utilizing the repetitiveness priors.

Semantic information helps to improve the robustness and accuracy of the initially reconstructed models by enhancing building model regularities. The building regularization is achieved by leveraging a set of structural linear features. We propose reliable linear features from images, triangulate them in space, and infer their spatial relations with a non-linear programing method. The poses of the linear features are adjusted to satisfy the inferred relations in a least-square manner, followed by a smooth deformation of the entire mesh geometry.

In this thesis, we transfer the 3D reconstruction from a pure geometry-based representation to a semantic representation at the object level on which high-level applications can be built, and remodel the object with specific semantic meanings to be visually pleasing and computationally compact. We demonstrate our methods on a few large and challenging datasets.