pp. 2001-2006
S&M2949 Research Paper of Special Issue https://doi.org/10.18494/SAM3773 Published: May 31, 2022 Picture-to-text Translation-based Assistive Communication System for People with Autism Spectrum Disorder [PDF] Yeou-Jiunn Chen (Received December 5, 2021; Accepted February 15, 2022) Keywords: assistive communication system, word prediction, picture-to-text translation, autism spectrum disorder
People with autism spectrum disorder (ASD) have different degrees of communication disorder, reducing the quality of their daily lives. Therefore, an assistive communication system could effectively help people with ASD to communicate with other people or with devices. In this study, an assistive communication system with word prediction and picture-to-text conversion is proposed to help people with ASD. To help them use the system, touch sensors are adopted at the operating interfaces. To overcome the problem of the limited number of items that can be displayed by the operating interfaces, recurrent neural networks are adopted to predict the next input picture. To help people understand the meaning of the picture sequences inputted by people with ASD, a picture-to-text translation system, which is based on bidirectional encoder representations from transformers (BERTs), is integrated to convert picture sequences to sentences. Experimental results showed that the proposed assistive communication system can effectively predict the next picture and exactly convert picture sequences to sentences. Thus, the proposed system can effectively help people with ASD communicate with others.
Corresponding author: Yeou-Jiunn ChenThis work is licensed under a Creative Commons Attribution 4.0 International License. Cite this article Yeou-Jiunn Chen, Picture-to-text Translation-based Assistive Communication System for People with Autism Spectrum Disorder, Sens. Mater., Vol. 34, No. 5, 2022, p. 2001-2006. |