Gigapixel videography, beyond the resolution of single camera and human visual perception, works to capture large-scale dynamic scene with extremely high resolution. Restricted by the spatial-temporal bandwidth product of optical system, gigapixel videography is faced with the challenges of expensive and complicated optical, electronic and mechanical design, laborious calibration, massive data processing etc. Aiming for the flexible, efficient and economized system, we propose a content adaptive gigapixel videography through the novel unstructured dual-scale camera array scheme, i.e., the global reference camera with wide-angle lens works to capture the global scene, the local gimbal camera with telephoto lens works to obtain the local high-resolution details. The adaptation of local gi...[ Read more ]

Gigapixel videography, beyond the resolution of single camera and human visual perception, works to capture large-scale dynamic scene with extremely high resolution. Restricted by the spatial-temporal bandwidth product of optical system, gigapixel videography is faced with the challenges of expensive and complicated optical, electronic and mechanical design, laborious calibration, massive data processing etc. Aiming for the flexible, efficient and economized system, we propose a content adaptive gigapixel videography through the novel unstructured dual-scale camera array scheme, i.e., the global reference camera with wide-angle lens works to capture the global scene, the local gimbal camera with telephoto lens works to obtain the local high-resolution details. The adaptation of local gimbal camera is enabled by the regions-of-interest (ROI) detection in global reference camera. The insight behind lies in the high sparsity of natural scene in both spatial and temporal domains.

More specifically, to retain high-resolution dynamic contents, a large-scale, multi-person and long-term tracking framework based on hierarchical image pyramid is introduced to sequentially select ROI. Meanwhile, a B-splines based motion planning algorithm for local gimbal camera is adopted to refine the motion trajectory. For synthesis of high-resolution video, we adopt multi-scale iterative feature matching and video warping with the huge resolution gap. The remaining static contents without local gimbal camera capturing are generated by Generative Adversarial Networks (GAN) based super resolution scheme. Experimental results conducted on the wild scenes demonstrate the flexibility and effectiveness of the content adaptive gigapixel videography scheme, enabling potential applications in various areas such as surveillance, aerial robots, autonomous driving etc.