Gesture Based Word (Re)Acquisition with a Virtual Agent in Augmented Reality: a Preliminary Study

EasyChair Preprint 6957

Abstract

From an evolutionary perspective, language and gesture belong together as a system serving communication on both an abstract and a physical level. In aphasia, when language is impaired, patients make use of gestures. Laboratory research has provided evidence that gesture can support aphasia rehabilitation, more specifically anomia rehabilitation. Here, we test an anomia gesture-based rehabilitation scenario with a virtual trainer (VT) in augmented reality (AR) as a therapy simulation. Thirty German speaking participants were trained to 27 bi- and three-syllabic words of Vimmi, an artificial language. Each Vimmi word was paired to a function word in German. The participants were divided into two Groups of 15 and 15 persons. Group A learned words pairs by observing the gestures performed by the VT and additionally imitating them. Group B learned 27 word pairs by observing the VT standing still and listening to it. Participants were trained singularly on 3 days alternating one day of training with one day of rest for memory consolidation. Word retention was assessed immediately after each training session by means of free and cued recall tests subministered electronically. Group A and Group B did not differ in word retention. When subdividing participants in high and low performers interactions showed that high performers benefitted more of gesture based training than low performers. The data in this preliminary study do not speak in favour of VTs as possible tool in gesture based AR language rehabilitation. Technology might have in this case detrimental effects on word learning.

Keyphrases: Rehabilitation, Technology, aphasia, embodiment, gesture, language, therapy, virtual training

@booklet{EasyChair:6957,
  author    = {Manuela Macedonia and Joachim Greiner and Claudia Repetto and Selina Wriessnegger},
  title     = {Gesture Based Word (Re)Acquisition with a Virtual Agent in Augmented Reality: a Preliminary Study},
  howpublished = {EasyChair Preprint 6957},
  year      = {EasyChair, 2021}}