This thesis presents new approaches in partitioning, characterizing, segmenting, and representing the content of video data. These approaches are developed based upon the analysis of spatio-temporal slices. While traditional approaches to motion sequence analysis tend to formulate computational methodologies on two or three adjacent frames, spatio-temporal slices provide rich visual patterns along a larger temporal scale. In our approach, the spatio-temporal slices are explored for video content analysis and representation....[ Read more ]

This thesis presents new approaches in partitioning, characterizing, segmenting, and representing the content of video data. These approaches are developed based upon the analysis of spatio-temporal slices. While traditional approaches to motion sequence analysis tend to formulate computational methodologies on two or three adjacent frames, spatio-temporal slices provide rich visual patterns along a larger temporal scale. In our approach, the spatio-temporal slices are explored for video content analysis and representation.

In our approach to video partitioning, the color, texture and statistical coherency in slices are exploited to detect three essential types of camera breaks, namely cuts, wipes and dissolves. Under this formulation, video partitioning is reduced to an image segmentation problem. In addition, using the oriented patterns depicting motion in slices, a new motion analysis method based on the structure tensor formulation is proposed. This method encodes the visual patterns of spatio-temporal slices in a tensor histogram, which on one hand, characterizes the changes of motion over time, on the other hand, describes the motion trajectories of different moving objects. Thus, for video characterization, an algorithm is developed, by analyzing tensor histograms, to temporally segment a video sequence into finer units, and to simultaneously characterize each unit with a coherent camera motion type. For video segmentation, the trajectories in tensor histograms are back-projected to the slices to form spatially separated motion layers. A novel integrated approach in reconstructing background panoramic image and detecting foreground objects is also proposed.

For video representation, issues on clustering, retrieval and summarizing the content of video are addressed by appropriately integrating the proposed techniques in video partitioning, characterization and segmentation. These issues include extracting foreground objects and motion features directly from tensor histograms for clustering and retrieval, representing shots adaptively and compactly through motion characterization, and reconstructing background images for scene change detection. The validity of the proposed approaches have been confirmed by extensive and rigorous experimentations.