Seminar from Cécile Macaire, PhD student in the GETALP team at the LIG

Date: 22/03/2024
Time: 10h15
Localization: IC2, Boardroom
Speaker: Cécile Macaire
 
 

Automatic translation of speech into pictograms to facilitate communication for people with language disabilities

When a person’s language skills (speech production and comprehension, reading, writing) are impaired or absent due to a genetic disease, or an autism spectrum disorder for example, Augmentative and Alternative Communication (AAC) can be put in place. AAC encompasses a range of tools and strategies designed to facilitate communication for individuals with language disorders. These include the use of pictograms to convey messages in everyday situations. At present, AAC faces a number of environmental obstacles that limit its use and dissemination (lack of awareness among carers of the potential of AAC and difficulty of access to tools). We believe that the development of systems capable of automatically translating speech into a sequence of pictograms would help to overcome these challenges.

I will first introduce this new Speech-to-Pictos NLP task, as part of the ANR PROPICTO project. In particular, I will focus on the various issues and pictograms in AAC. I will detail the work on the construction of speech-text-pictogram aligned data. I will continue by presenting two approaches to the Speech-to-Pictograms task, which adapt state-of-the-art Automatic Speech Recognition, Machine Translation and Speech Translation systems to this data. To conclude, I will report the results of a first automatic and human evaluation of these models, while illustrating my remarks with several examples of translation into pictograms.