Growing 3D map services drives tremendous demand for photo-realistic modeling of cities from images captured at ground level. This modeling of cities reduces de facto to that of building façades. The accurate extraction and partition of individual façades from urban scenes and the semantic analysis of each individual façade are two main challenges.

The key to solve these problems is using special features existing in urban environment: rectilinearity and symmetry. First, a joint 2D-3D segmentation methods assuming rectilinear boundary of façade parses the environment into buildings, the ground, and the sky; for the first time, buildings are further partitioned into individual facades using the proposed dynamic programming optimization. The next step detects and segments st...[ Read more ]

Growing 3D map services drives tremendous demand for photo-realistic modeling of cities from images captured at ground level. This modeling of cities reduces de facto to that of building façades. The accurate extraction and partition of individual façades from urban scenes and the semantic analysis of each individual façade are two main challenges.

The key to solve these problems is using special features existing in urban environment: rectilinearity and symmetry. First, a joint 2D-3D segmentation methods assuming rectilinear boundary of façade parses the environment into buildings, the ground, and the sky; for the first time, buildings are further partitioned into individual facades using the proposed dynamic programming optimization. The next step detects and segments structural elements within individual façade by exploiting the information redundancy of repetition. We propose a dual image- and transform-space optimization method based on the formulation of Markov random field (MRF), capable of simultaneously discovering multiple interfering repetitions. After that, we extend the MRF formulation to the detection of per-pixel symmetry, and then develop a learning-based segmentation method that can extract symmetry objects, which are recognized as architecture elements, from background walls. Extensive evaluation on large-scale data sets of cities demonstrates both quantitative and qualitative improvements of our detection and segmentation methods over the state-of-the-art, especially dealing with multiple interfering symmetries, low-count symmetries, and architecture element extraction, etc.

The extracted architecture elements are re-assembled into a set of newly invented computer-generated architecture (CGA) grammar rules with contain rules. Given the façade analysis results and the learnt grammar rules, we develop a 3D city modeling method which is capable of generating detailed geometry models with refined textures. Besides, instead of creating 3D models from scratch, we make use of the existing approximate models of buildings, together with the analysis results of both 3D geometry and 2D textures, to generate finer models of the same buildings. The performance of our modeling and remodeling methods is demonstrated on several challenging data sets and both analytical and perceptual improvements are achieved.