The Visual Dictionary teaches the basic vocabularies of American Sign Language (ASL) through visual-gesture associations.
It employs machine learning technology to recognise hand movements that, if recognised correctly, will generate an image that matches the meaning of the word.
It is designed for children and anyone who wants to learn and practise sign language at home, at school, or in a therapeutic setting.
It aims to build a more inclusive society by facilitating easier and freer interaction between the general public and people with speech and hearing impairments.
Self-learning sign language is a unique process that is more challenging than learning a spoken language.
However, according to our research, most self-study tutorials on the market have two drawbacks: they are one-sided and do not provide interactive feedback; they only teach by showing words and gestures, without showing pictures of what they are.
To address these issues, we have taken a gamification approach and designed the Visual Dictionary that includes real-time interaction and image feedback, which makes learning sign language more engaging.
In this sample experience, we have selected three words: bear, chicken, and rabbit.
Users learn one word at a time and then make the corresponding sign.
The program determines the similarity of the gesture, and if it is correct, generates a picture of the animal to affirm it and help reinforce the memory.
We have conducted two user tests with participants from different age groups, ranging from children to elderly people. Based on the feedback received from the tests, we made updates to the usage flow and web design of our project.
Although the demo version of the project still has some flaws in terms of recognition accuracy and page size response on different devices, users have found the design to be useful and interesting. The product has shown potential for further development.
