Sign language recognition using depth images

This paper presents a vision based sign language gesture recognition framework that can assist people with impaired hearing and speech with their social interaction and interactive communications. Utilizing a low-cost sensor, such as Microsoft Kinect combined with advanced machine learning analysis, it aims to ease the challenging issue of increasing demand for professional sign language interpreting services. Specifically, this paper discusses a powerful discriminating descriptor called 3D motion map based pyramid histograms of oriented gradient (M-PHOG) which is proposed for depth-based human gesture recognition. The 3D motion map is generated through the entire depth video sequence to encode additional motion information from three projected orthogonal planes. By adding pyramid representation, HOG descriptor is extended to M-PHOG which can characterize local shapes at different spatial grid sizes for gesture recognition. The proposed approach is evaluated on MSR Gesture3D and DEVISIGN two data sets captured by depth cameras. Experimental results show that the proposed approach outperforms the current state-of-the-art methods and demonstrates the effectiveness and robustness of the proposed 3D M-PHOG descriptor. The proposed approach can translate the meaning of captured gestures as professional interpreters currently do. The novelty framework has the potential to improve the quality of life for the deaf community and reduce the communication barriers they currently experience.