Deaf-Mute Communication – Part III: The Solution

Existing Solutions

Obviously, we are not the only ones who had the idea of using a gesture recognition device such as leap motion to read and interpret sign language. We have found two applications which are being developed at the moment that use Leap motion to read sign language.

• Motionsavvy intends to translate sign language to text.
• chat software which can be used with sign language.

Our Solution

The main purpose of our solution is to ease communication for deaf people in situations where they have to interact with people who are not able to speak sign language. According to our background research we have identified a set of qualities that need to be fulfilled in order to provide a usable device and with a good user experience.

Qualities

• Reads sign language and transforms it into spoken language or text
• Dictation feature to store a set of often used questions / phrases to increase communication speed
• Has an aesthetic design e.g. can be worn like a necklace
• Communication between device and mobile phone to make voice calls

Sketched device

The outlined solution could be prototyped by using a device called Leap Motion. Leap Motion is a small device that can recognize hands with high accuracy allowing interactions through gestures.

We have build a demo to demonstrate the reading of American Sign Language (ASL) and the transformation into text output. The demo uses Leap Motion’s Javascript API and is able to recognize the following words and numbers:

• What?
• here
• Book
• Sorry
• You
• 1
• 2
• 3
• 4
• 5

If you have a Leap Motion and want to try it out you can download the following zip file which includes an html file as well as the jquery and leap motion javascript libraries.

Download Demo

Potential problems

As promising as our solution might sound there are also certain problems attached. In order to operate accurately the device relies on image recognition. That means that the device needs to have a direct view on the hands of the user. Some words in ASL require the positioning of hands above each other which cannot be detected by Leap Motion. Additionally, different light conditions e.g. direct / bright sunlight can influence the accuracy of the device up to making it completely unusable.

Another problem can be described as the King Midas problem. King Midas had the ability to turn everything he touches into gold. The same problem occurs with such a device as certain everyday gestures or hand movement might be interpreted as a word resulting in unwanted voice or text output.

Potential for universal usage

The main purpose of our solution is to include deaf people which does not mean that the device cannot be used by other people as well. We can think of applications that enhance the day-to-day experience of all people.

• Dictation of confidential information in public places
• Writing text messages while walking