SRG3: Speech-driven Robot Gesture Generation with GAN

Chuang Yu (1) , Adriana Tapus (1)

1 U2IS - Unité d'Informatique et d'Ingénierie des Systèmes

The human gestures occur spontaneously and usually they are aligned with speech, which leads to a natural and expressive interaction. Speech-driven gesture generation is important in order to enable a social robot to exhibit social cues and conduct a successful human-robot interaction. In this paper, the generation process involves mapping acoustic speech representation to the corresponding gestures for a humanoid robot. The paper proposes a new GAN (Generative Adversarial Network) architecture for speech to gesture generation. Instead of the fixed mapping from one speech to one gesture pattern, our end-to-end GAN structure can generate multiple mapped gestures patterns from one speech (with multiple noises) just like humans do. The generated gestures can be applied to social robots with arms. The evaluation result shows the effectiveness of our generative model for speech-driven robot gesture generation.

Domaines: Robotique [cs.RO]

Chuang Yu, Adriana Tapus. SRG3: Speech-driven Robot Gesture Generation with GAN. 16th International Conference on Control, Automation, Robotics and Vision, Dec 2020, Shenzhen (virtual), China. ⟨hal-03047565⟩