Description
AbstractThe growing volume of multimedia content available in cyberspace has increased research on the development of effective video analysis and content management techniques for the applications of video summarisation, driverless cars, automatic surveillance, intelligent traffic surveillance etc. Video Content Analytics (VCA) provides the ability to automatically detect and extract meaningful information and events from video content. A significant research improvement has been made in this space. However, VCA in a challenging environment is still a daunting task. Capturing images in challenging conditions degrades the quality of the images, for example, during rain. These external phenomena create low contrast and blur, reducing the images' visibility. The reduced visibility affects many computer vision applications like visual traffic surveillance, intelligent vehicles, and entertainment. To improve the outcome of video analytics, it is necessary to acquire a clear vision of the images and other modalities such as audio signals. In sports videos, the audio signal contains important information. Initially, an algorithm was developed to extract audio features and summarise the video, including important information. Here innovative audio features were introduced in video summarisation as they characterised exciting events during the video. In many computer vision applications, audio features are not able to extract meaningful information from a video sequence.
The videos from a surveillance camera and driverless car do not contain any audio, or the audio does not have important information. Thus, improving the video’s visibility is a prime requirement in these applications. Rain streak removal is an essential issue in outdoor vision systems and has recently been investigated extensively as rain can significantly reduce visibility. Recently, many approaches have been proposed to remove rain streaks from video sequences. Some approaches are based on physical features (i.e., geometric, photometric, temporal etc.), mathematical models (i.e., low rank, sparsity), and some are data-driven (i.e.,deep-learning) models. Although the physical features-based methods have better rain interpretability, the challenges are extracting the appropriate features and fusing them for meaningful rain removal. Rain streaks and moving objects have dynamic physical characteristics and are sometimes difficult to distinguish. In this work, a few novel physical features of rain streaks were extracted, for example, time duration, width/height, relative location and photometric correlation of rain streaks against the static background. The method also adaptively scales features based on a video's different resolutions and frame rates. In some scenarios, these simple but impactful features are not sufficient to distinguish rain streaks and moving objects. As a result, a small portion of moving objects is missed in the rain-free videos.
Additionally, the outcome of the data-driven models mostly depends on variations relating to the training dataset. It is challenging to include datasets with all possible variations in model training. To address issues raised by both feature-based and data-driven methods, a novel hybrid method to improve the visibility of video sequences by removing rain streaks was proposed. Novel physical and data-driven features were extracted and combined to create an effective rain-streak removal strategy. Previously extracted physical features and the Mask-RCNN deep learning algorithm to distinguish objects from rain streaks were applied. The proposed method fuses the outcomes from physical characteristics and the Mask-RCNN algorithm in two steps. Moreover, it progressively processes features from upcoming frames to remove rain in real-time. The performance of the proposed algorithm has been tested in comparison to several relevant and contemporary methods using benchmark datasets. The experimental result shows that the proposed method outperforms the existing methods in terms of subjective, objective, and object detection comparisons for synthetic and real rain scenarios by removing rain streaks and retaining the moving objects more effectively.
| Period | Feb 2023 |
|---|---|
| Examinee | |
| Degree of Recognition | International |