Guzmán Robador |
Alicante (EFE).- Guaranteeing the right to inclusion is vital in all groups of society, as is the ability to understand and be understood, regardless of whether or not they have a disability. Assistive robotics is becoming a support with great potential to help deaf people to cope with communication challenges.
An advance in this field is now the first application capable of recognizing and interpreting the Spanish Sign Language alphabet in real time, recently designed, without any public subsidy, by two researchers from the Robotics and Three-Dimensional Vision Group (RoViT ) from the University of Alicante (UA).
Professor Ester Martínez and doctoral student Francisco Morillas got involved in this initiative without knowing, at first, Sign Language and have managed to materialize an application that, according to UA sources, contributes to breaking communication barriers between deaf and listeners in everyday situations such as going to the doctor’s office or eating in a restaurant.
Thanks to the use of different computer vision and deep learning techniques, Ester Martínez, who works in care robotics, and Francisco Morillas have developed this low-cost tool, named ‘Sign4all’, in order to to be able to offer assistance to deaf people when they cannot be accompanied by an interpreter.
Speaking to EFE, Martínez explained that this project began two years ago, coinciding with the end-of-degree project of Francisco Morillas, who is now doing his doctorate, and it arose after they designed an application in 2018 to monitor and carry out physical exercise with older people and an interface so that these people could interact with it.
Developing this interface, they asked themselves “what happens if the person the robot is trying to monitor has some kind of visual or hearing disability”, revealed Martínez.
UA sources have detailed how this tool works: ‘Sign4all’, after capturing the person and extracting the detail of the skeleton of the arms and hands, codes the left part of the body in blue and the right part in red, keeping the the anonymity of the user at all times.
An innovative application due to its bidirectional quality
From this moment on, the application has a bidirectional quality, which makes it innovative at an international level, since it translates the sign made by the deaf person in real time and, in the opposite direction, it is capable of signing, through a virtual avatar , Spanish words typed by the hearing person.
“The idea is that this entire process can be done by downloading an application with the camera of the mobile phone or a tablet so that it can be used anywhere and easily,” explained Martínez, who explained that, after many tests, ‘Sign4all’ manages to interpret and recognize the alphabet of the Spanish Sign Language (LSE) with an accuracy of 80%.
Although this result corresponds to the dactylological alphabet (each sign represents a letter), “we are working on a version with a specific vocabulary belonging to the field of daily tasks where we can interpret complete sentences”, he announced.
Collaboration with the University of Vigo
The UA work team has been “training” this new system for months, introducing more and more signs. In this sense, a collaboration has been sealed with the Research Group of the Spanish Language and Signed Languages (GRILES) of the University of Vigo, with extensive experience in the study of this language and its use in different territories.
The application that they have created for the deaf allows for now to recognize the alphabet of sign language and sign it, but the two UA researchers already have a new corpus of food vocabulary data and they will add, with some tests that they are doing his colleagues from the University of Vigo, expressions of social interaction.
“At the moment, the only thing that is finalized and completed is what concerns the alphabet, although we are trying to expand it so that the application is really useful and reaches as many people as possible,” said Martínez.
To date, this project has not received any public subsidy, except for the granting of a research grant for Francisco Morillas to do his doctorate in this line.
