Sequential deep learning for action recognition with synthetic multi-view data from depth maps

Bin Liang, Lihong Zheng, Xinying Li

Research output: Book chapter/Published conference paperConference paperpeer-review

1 Citation (Scopus)


Recurrent neural network (RNN) has proven successful recently in action recognition. However, depth sequences are of high dimensionality and contain rich human dynamics, which makes traditional RNNs difficult to capture complex action information. This paper addresses the problem of human action recognition from sequences of depth maps using sequential deep learning. The proposed method first synthesizes multi-view depth sequences by rotating 3D point clouds from depth maps. Each depth sequence is then split into short-term temporal segments. For each segment, a multi-view depth motion template (MVDMT), which compresses the segment to a motion template, is constructed for short-term multi-view action representation. The MVDMT effectively characterizes the multi-view appearance and motion patterns within a short-term duration. Convolutional Neural Network (CNN) models are leveraged to extract features from MVDMT, and a CNN-RNN network is subsequently employed to learn an effective representation for sequential patterns of the multi-view depth sequence. The proposed multi-view sequential deep learning framework can simultaneously capture spatial-temporal appearance and motion features in the depth sequence. The proposed method has been evaluated on the MSR Action3D and MSR Action Pairs datasets, achieving promising results compared with the state-of-the-art methods based on depth data.
Original languageEnglish
Title of host publicationData Mining - 16th Australasian Conference, AusDM 2018, Revised Selected Papers
EditorsYanchang Zhao, Graco Warwick, David Stirling, Chang-Tsun Li, Yun Sing Koh, Rafiqul Islam, Zahidul Islam
PublisherSpringer-Verlag London Ltd.
Number of pages12
ISBN (Electronic)9789811366611
ISBN (Print)9789811366604
Publication statusPublished - 2019
Event16th Australasian Conference on Data Mining, AusDM 2018 - Charles Sturt University , Bathurst, Australia
Duration: 28 Nov 201830 Nov 2018

Publication series

NameCommunications in Computer and Information Science
ISSN (Print)1865-0929


Conference16th Australasian Conference on Data Mining, AusDM 2018
Internet address


Dive into the research topics of 'Sequential deep learning for action recognition with synthetic multi-view data from depth maps'. Together they form a unique fingerprint.

Cite this