A Gaussian mixture representation of gesture kinematics for on-line Sign Language video annotation

Fabio Martínez (1, 2) , Antoine Manzanera (1) , Michèle Gouiffès (2) , Annelies Braffort (3)

1 RV - Robotique et Vision
2 AMI - Architectures et Modèles pour l'Interaction
3 ILES - Information, Langue Ecrite et Signée

Sign languages (SLs) are visuo-gestural representations used by deaf communities. Recognition of SLs usually requires manual annotations, which are expert dependent, prone to errors and time consuming. This work introduces a method to support SL annotations based on a motion descriptor that characterizes dynamic gestures in videos. The proposed approach starts by computing local kinematic cues, represented as mixtures of Gaussians which together correspond to gestures with a semantic equivalence in the sign language corpora. At each frame, a spatial pyramid partition allows a fine-to-coarse sub-regional description of motion-cues distribution. Then for each sub-region, a histogram of motion-cues occurrence is built, forming a frame-gesture descriptor which can be used for on-line annotation. The proposed approach is evaluated using a bag-of-features framework, in which every frame-level histogram is mapped to an SVM. Experimental results show competitive results in terms of accuracy and time computation for a signing dataset.

Domaines: Vision par ordinateur et reconnaissance de formes [cs.CV]; Traitement des images [eess.IV]

Fabio Martínez, Antoine Manzanera, Michèle Gouiffès, Annelies Braffort. A Gaussian mixture representation of gesture kinematics for on-line Sign Language video annotation. International Symposium on Visual Computing ISVC'15, Dec 2015, Las Vegas, United States. ⟨hal-01245123⟩