Virtual view quality enhancement using side view temporal modelling information for free viewpoint video

Research output: Book chapter/Published conference paperConference paper

Abstract

Virtual viewpoint video needs to be synthesised from adjacent reference viewpoints to provide immersive perceptual 3D viewing experience of a scene. View synthesised techniques suffer poor rendering quality due to holes created by occlusion in the warping process. Currently, spatial and temporal correlation of texture images and depth maps are exploited to improve the quality of the final synthesised view. Due to the low spatial correlation at the edge between foreground and background pixels, spatial correlation e.g. inpainting and inverse mapping (IM) techniques cannot fill holes effectively. Conversely, a temporal correlation among already synthesised frames through learning by Gaussian mixture modelling (GMM) fill missing pixels in occluded areas efficiently. In this process, there are no frames for GMM learning when the user switches view instantly. To address the above issues, in the proposed view synthesis technique, we apply GMM on the adjacent reference viewpoint texture images and depth maps to generate a most common frame in a scene (McFIS). Then, texture McFIS is warped into the target viewpoint by using depth McFIS and both warped McFISes are merged. Then, we utilize the number of GMM models to refine pixel intensities of the synthesised view by using a weighting factor between the pixel intensities of the merged McFIS and the warped images. This technique provides a better pixel correspondence and improves 0.58∼0.70dB PSNR compared to the IM technique.

Original languageEnglish
Title of host publication2018 International Conference on Digital Image Computing
Subtitle of host publicationTechniques and Applications, DICTA 2018
EditorsMark Pickering, Lihong Zheng, Shaodi You, Ashfaqur Rahman, Manzur Murshed, Md Asikuzzaman, Ambarish Natu, Antonio Robles-Kelly, Manoranjan Paul
PublisherIEEE, Institute of Electrical and Electronics Engineers
ISBN (Electronic)9781538666029
DOIs
Publication statusPublished - 16 Jan 2019
Event2018 International Conference on Digital Image Computing: Techniques and Applications: DICTA 2018 - Canberra Rex Hotel, Canberra, Australia
Duration: 10 Dec 201813 Dec 2018
https://dicta2018.org/ (Conference website)

Publication series

Name2018 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2018

Conference

Conference2018 International Conference on Digital Image Computing: Techniques and Applications
CountryAustralia
CityCanberra
Period10/12/1813/12/18
Internet address

Fingerprint

Pixels
Learning
Textures
Switches

Cite this

Rahaman, D. M. M., Paul, M., & Shoumy, N. J. (2019). Virtual view quality enhancement using side view temporal modelling information for free viewpoint video. In M. Pickering, L. Zheng, S. You, A. Rahman, M. Murshed, M. Asikuzzaman, A. Natu, A. Robles-Kelly, ... M. Paul (Eds.), 2018 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2018 [8615827] (2018 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2018). IEEE, Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/DICTA.2018.8615827
Rahaman, D. M.Motiur ; Paul, Manoranjan ; Shoumy, Nusrat Jahan. / Virtual view quality enhancement using side view temporal modelling information for free viewpoint video. 2018 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2018. editor / Mark Pickering ; Lihong Zheng ; Shaodi You ; Ashfaqur Rahman ; Manzur Murshed ; Md Asikuzzaman ; Ambarish Natu ; Antonio Robles-Kelly ; Manoranjan Paul. IEEE, Institute of Electrical and Electronics Engineers, 2019. (2018 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2018).
@inproceedings{25bb6c320a7a4238aa1f4129d837295f,
title = "Virtual view quality enhancement using side view temporal modelling information for free viewpoint video",
abstract = "Virtual viewpoint video needs to be synthesised from adjacent reference viewpoints to provide immersive perceptual 3D viewing experience of a scene. View synthesised techniques suffer poor rendering quality due to holes created by occlusion in the warping process. Currently, spatial and temporal correlation of texture images and depth maps are exploited to improve the quality of the final synthesised view. Due to the low spatial correlation at the edge between foreground and background pixels, spatial correlation e.g. inpainting and inverse mapping (IM) techniques cannot fill holes effectively. Conversely, a temporal correlation among already synthesised frames through learning by Gaussian mixture modelling (GMM) fill missing pixels in occluded areas efficiently. In this process, there are no frames for GMM learning when the user switches view instantly. To address the above issues, in the proposed view synthesis technique, we apply GMM on the adjacent reference viewpoint texture images and depth maps to generate a most common frame in a scene (McFIS). Then, texture McFIS is warped into the target viewpoint by using depth McFIS and both warped McFISes are merged. Then, we utilize the number of GMM models to refine pixel intensities of the synthesised view by using a weighting factor between the pixel intensities of the merged McFIS and the warped images. This technique provides a better pixel correspondence and improves 0.58∼0.70dB PSNR compared to the IM technique.",
keywords = "depth image-based rendering, Gaussian mixture modelling, inverse mapping, occlusion, spatial, temporal correlation, View synthesis",
author = "Rahaman, {D. M.Motiur} and Manoranjan Paul and Shoumy, {Nusrat Jahan}",
year = "2019",
month = "1",
day = "16",
doi = "10.1109/DICTA.2018.8615827",
language = "English",
series = "2018 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2018",
publisher = "IEEE, Institute of Electrical and Electronics Engineers",
editor = "Mark Pickering and Lihong Zheng and Shaodi You and Ashfaqur Rahman and Manzur Murshed and Md Asikuzzaman and Ambarish Natu and Antonio Robles-Kelly and Manoranjan Paul",
booktitle = "2018 International Conference on Digital Image Computing",
address = "United States",

}

Rahaman, DMM, Paul, M & Shoumy, NJ 2019, Virtual view quality enhancement using side view temporal modelling information for free viewpoint video. in M Pickering, L Zheng, S You, A Rahman, M Murshed, M Asikuzzaman, A Natu, A Robles-Kelly & M Paul (eds), 2018 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2018., 8615827, 2018 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2018, IEEE, Institute of Electrical and Electronics Engineers, 2018 International Conference on Digital Image Computing: Techniques and Applications, Canberra, Australia, 10/12/18. https://doi.org/10.1109/DICTA.2018.8615827

Virtual view quality enhancement using side view temporal modelling information for free viewpoint video. / Rahaman, D. M.Motiur; Paul, Manoranjan; Shoumy, Nusrat Jahan.

2018 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2018. ed. / Mark Pickering; Lihong Zheng; Shaodi You; Ashfaqur Rahman; Manzur Murshed; Md Asikuzzaman; Ambarish Natu; Antonio Robles-Kelly; Manoranjan Paul. IEEE, Institute of Electrical and Electronics Engineers, 2019. 8615827 (2018 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2018).

Research output: Book chapter/Published conference paperConference paper

TY - GEN

T1 - Virtual view quality enhancement using side view temporal modelling information for free viewpoint video

AU - Rahaman, D. M.Motiur

AU - Paul, Manoranjan

AU - Shoumy, Nusrat Jahan

PY - 2019/1/16

Y1 - 2019/1/16

N2 - Virtual viewpoint video needs to be synthesised from adjacent reference viewpoints to provide immersive perceptual 3D viewing experience of a scene. View synthesised techniques suffer poor rendering quality due to holes created by occlusion in the warping process. Currently, spatial and temporal correlation of texture images and depth maps are exploited to improve the quality of the final synthesised view. Due to the low spatial correlation at the edge between foreground and background pixels, spatial correlation e.g. inpainting and inverse mapping (IM) techniques cannot fill holes effectively. Conversely, a temporal correlation among already synthesised frames through learning by Gaussian mixture modelling (GMM) fill missing pixels in occluded areas efficiently. In this process, there are no frames for GMM learning when the user switches view instantly. To address the above issues, in the proposed view synthesis technique, we apply GMM on the adjacent reference viewpoint texture images and depth maps to generate a most common frame in a scene (McFIS). Then, texture McFIS is warped into the target viewpoint by using depth McFIS and both warped McFISes are merged. Then, we utilize the number of GMM models to refine pixel intensities of the synthesised view by using a weighting factor between the pixel intensities of the merged McFIS and the warped images. This technique provides a better pixel correspondence and improves 0.58∼0.70dB PSNR compared to the IM technique.

AB - Virtual viewpoint video needs to be synthesised from adjacent reference viewpoints to provide immersive perceptual 3D viewing experience of a scene. View synthesised techniques suffer poor rendering quality due to holes created by occlusion in the warping process. Currently, spatial and temporal correlation of texture images and depth maps are exploited to improve the quality of the final synthesised view. Due to the low spatial correlation at the edge between foreground and background pixels, spatial correlation e.g. inpainting and inverse mapping (IM) techniques cannot fill holes effectively. Conversely, a temporal correlation among already synthesised frames through learning by Gaussian mixture modelling (GMM) fill missing pixels in occluded areas efficiently. In this process, there are no frames for GMM learning when the user switches view instantly. To address the above issues, in the proposed view synthesis technique, we apply GMM on the adjacent reference viewpoint texture images and depth maps to generate a most common frame in a scene (McFIS). Then, texture McFIS is warped into the target viewpoint by using depth McFIS and both warped McFISes are merged. Then, we utilize the number of GMM models to refine pixel intensities of the synthesised view by using a weighting factor between the pixel intensities of the merged McFIS and the warped images. This technique provides a better pixel correspondence and improves 0.58∼0.70dB PSNR compared to the IM technique.

KW - depth image-based rendering

KW - Gaussian mixture modelling

KW - inverse mapping

KW - occlusion

KW - spatial

KW - temporal correlation

KW - View synthesis

UR - http://www.scopus.com/inward/record.url?scp=85062232702&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85062232702&partnerID=8YFLogxK

U2 - 10.1109/DICTA.2018.8615827

DO - 10.1109/DICTA.2018.8615827

M3 - Conference paper

AN - SCOPUS:85062232702

T3 - 2018 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2018

BT - 2018 International Conference on Digital Image Computing

A2 - Pickering, Mark

A2 - Zheng, Lihong

A2 - You, Shaodi

A2 - Rahman, Ashfaqur

A2 - Murshed, Manzur

A2 - Asikuzzaman, Md

A2 - Natu, Ambarish

A2 - Robles-Kelly, Antonio

A2 - Paul, Manoranjan

PB - IEEE, Institute of Electrical and Electronics Engineers

ER -

Rahaman DMM, Paul M, Shoumy NJ. Virtual view quality enhancement using side view temporal modelling information for free viewpoint video. In Pickering M, Zheng L, You S, Rahman A, Murshed M, Asikuzzaman M, Natu A, Robles-Kelly A, Paul M, editors, 2018 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2018. IEEE, Institute of Electrical and Electronics Engineers. 2019. 8615827. (2018 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2018). https://doi.org/10.1109/DICTA.2018.8615827