Adaptive weighting between warped and learned foregrounds for view synthesize

In a free viewpoint video (FVV) system, a user has a freedom to choose a view angle arbitrarily. To facilitate view synthesis, we need to interpolate a number of adjacent views by exploiting spatial and temporal redundancy among them. Existing synthesizing techniques may concern poor rendering quality due to the occluded background areas and rounding integer error through warping. Currently, inpainting and background update techniques are used to address these issues. Inpainting techniques normally suffer quality degradation due to the low spatial correlation in the foreground/background boundary areas. Background update technique improves the quality, however, still suffers quality degradation due to the dependency on background warping and spatial correlation. To address the problem, in this paper we learn the foreground and background in each pixel using Gaussian mixture model (GMM) on the previous images in the interpolated view and then use the number of Gaussian models to identify the foreground and background pixels. The background pixel of the synthesized view is determined using the background model and the foreground pixel is determined using the weighted contribution of the foreground model and the warping image between two adjacent views. An adaptive weighting is proposed in this paper based on the amount of multiple models. The experimental results reveal that the proposed approach provides 0.60∼3.15dB PSNR improvement of the synthesized view compared with the three standard state-of-the-art methods.