Sequential deep learning for action recognition with synthetic multi-view data from depth maps

Bin Liang, Lihong Zheng, Xinying Li

Research output: Book chapter/Published conference paperConference paperpeer-review

1 Citation (Scopus)


Recurrent neural network (RNN) has proven successful recently in action recognition. However, depth sequences are of high dimensionality and contain rich human dynamics, which makes traditional RNNs difficult to capture complex action information. This paper addresses the problem of human action recognition from sequences of depth maps using sequential deep learning. The proposed method first synthesizes multi-view depth sequences by rotating 3D point clouds from depth maps. Each depth sequence is then split into short-term temporal segments. For each segment, a multi-view depth motion template (MVDMT), which compresses the segment to a motion template, is constructed for short-term multi-view action representation. The MVDMT effectively characterizes the multi-view appearance and motion patterns within a short-term duration. Convolutional Neural Network (CNN) models are leveraged to extract features from MVDMT, and a CNN-RNN network is subsequently employed to learn an effective representation for sequential patterns of the multi-view depth sequence. The proposed multi-view sequential deep learning framework can simultaneously capture spatial-temporal appearance and motion features in the depth sequence. The proposed method has been evaluated on the MSR Action3D and MSR Action Pairs datasets, achieving promising results compared with the state-of-the-art methods based on depth data.
Original languageEnglish
Title of host publicationData Mining
Subtitle of host publication16th Australasian Conference, AusDM 2018, Bahrurst, NSW, Australia, November 28–30, 2018, Revised selected papers
EditorsYanchang Zhao, Graco Warwick, David Stirling, Chang-Tsun Li, Yun Sing Koh, Rafiqul Islam, Zahidul Islam
PublisherSpringer-Verlag London Ltd.
Number of pages12
ISBN (Electronic)9789811366611
ISBN (Print)9789811366604
Publication statusPublished - 2019
EventThe 16th Australasian Data Mining Conference - Charles Sturt University , Bathurst, Australia
Duration: 28 Nov 201830 Nov 2018 (Conference website) (Call for papers) (Conference program)

Publication series

NameCommunications in Computer and Information Science
ISSN (Print)1865-0929


ConferenceThe 16th Australasian Data Mining Conference
OtherThe Australasian Data Mining Conference (AusDM) has established itself as the premier Australasian meeting for both practitioners and researchers in data mining. It is devoted to the art and science of intelligent analysis of (usually big) data sets for meaningful (and previously unknown) insights. This conference will enable the sharing and learning of research and progress in the local context and new breakthroughs in data mining algorithms and their applications across all industries.
Internet address


Dive into the research topics of 'Sequential deep learning for action recognition with synthetic multi-view data from depth maps'. Together they form a unique fingerprint.

Cite this