This paper presents a vision based sign language gesture recognition framework that can assist people with impaired hearing and speech with their social interaction and interactive communications. Utilizing a low-cost sensor, such as Microsoft Kinect combined with advanced machine learning analysis, it aims to ease the challenging issue of increasing demand for professional sign language interpreting services. Specifically, this paper discusses a powerful discriminating descriptor called 3D motion map based pyramid histograms of oriented gradient (M-PHOG) which is proposed for depth-based human gesture recognition. The 3D motion map is generated through the entire depth video sequence to encode additional motion information from three projected orthogonal planes. By adding pyramid representation, HOG descriptor is extended to M-PHOG which can characterize local shapes at different spatial grid sizes for gesture recognition. The proposed approach is evaluated on MSR Gesture3D and DEVISIGN two data sets captured by depth cameras. Experimental results show that the proposed approach outperforms the current state-of-the-art methods and demonstrates the effectiveness and robustness of the proposed 3D M-PHOG descriptor. The proposed approach can translate the meaning of captured gestures as professional interpreters currently do. The novelty framework has the potential to improve the quality of life for the deaf community and reduce the communication barriers they currently experience.
Original languageEnglish
Title of host publicationProceedings of the 14th international conference on control, automation, robotics and vision
Subtitle of host publicationICARCV 2016
Place of PublicationUnited States
PublisherIEEE, Institute of Electrical and Electronics Engineers
Number of pages6
ISBN (Electronic)9781509035496
ISBN (Print)9781509035502 (Print on demand)
Publication statusPublished - 31 Jan 2017
Event14th International Conference on Control, Automation, Robotics and Vision: ICARCV 2016 - Duangjitt Resort & Spa, Phuket, Thailand
Duration: 13 Nov 201615 Nov 2016
http://icarcv.org/2016/home.asp (Conference website)


Conference14th International Conference on Control, Automation, Robotics and Vision
OtherInaugurated in 1990, ICARCV is a biennial event and has proven to be a premium forum where researchers, professionals and engineers in the area of Control, Automation, Robotics and Vision meet to interact and exchange the latest theoretical and experimental results in these four areas. The future ICARCV will strive to serve the same purpose. To help delegates to be in touch with us, a permanent email address (secretariat@icarcv.org) and website (http://www.icarcv.org) have been set up.
Internet address


Dive into the research topics of 'Sign language recognition using depth images'. Together they form a unique fingerprint.

Cite this