Tech for good: Class12 students develop an app to help the deaf, blind and mute communicate with ease

Developed by three Class 12 students from Amity International, Delhi, the Practikality app provides an interactive mechanism for the differently abled - the deaf, the blind, and the mute - and is in its final stages of testing.

For seven-year-old Rohan Velanki, a partially mute and deaf child, the joy of social interaction was a limited experience. A quiet and reticent child, Rohan began interacting with Sugam, a non-governmental organisation (NGO), which works towards the betterment of abandoned and differently-abled children. Over the course of the next six months, he became a far more sociable and confident child. The key to this transformation was his new-found ability to communicate efficiently.

“Rohan carried a tremendous amount of emotional baggage. He would hardly attempt to mingle with other children. However, after using the ‘Practikality’ application, what seemed impossible became possible. He turned out to be a more outgoing and light-hearted child,” says Saritha C, Vice-President, Sugam NGO.

Developed by Padam Chopra, Keshav Maheshwari and Aryaman Agrawal, three Class 12 students from Amity International, New Delhi, the ‘Practikality’ app aims to empower people with disabilities.

While Braille helps the visually impaired to read and write, sign language aids the deaf and mute in expressing themselves. But, these can sometimes be a form of one-way communication. The machine learning-based Practikality app provides a fully-interactive mechanism for the deaf, blind, and mute to communicate with ease, and is in its final stages of testing. The app is scheduled to be launched on Android and on the web in March this year.

How it all started

Padam and Aaryaman are classmates who had earlier worked together as a team for an event in 2016. They have also represented their college together in multiple competitions. Padam says that the push to develop an app like this came about when his grandfather lost his ability to speak clearly.

“My grandfather could not even go to the chemist by himself. Since he couldn’t communicate properly, he used to depend on me for everything. He was feeling very helpless, and I wanted to do something about it. That is when I started thinking about the bigger picture, and of those people who do not have the ability to see, hear or speak,” says Padam.

A few months later, Padam got to visit a blind school, an experience that spurred him to commit himself to making the idea a reality. Both Aaryaman and Padam started on the research and development needed to build the application. After a few months, their classmate Keshav joined them to bring in some business acumen to the team.

“We spent sleepless nights learning to build algorithms and developing machine-learning models. But I am glad our efforts bore some fruit,” Padam recollects.

Three solutions, one app

[Also check out our collection of stories on Inclusion in India]

The Practikality app has three features - Voice, Easi, and Vision - to assist the differently abled.

VOICE: This solution can be used by the mute to convert text into voice. The user has to key in the message that he or she wants to communicate. The app converts the input which can be either words or Morse Code (a character encoding scheme) into audio, thereby facilitating communication between the user and the recipient. The icing on the cake is that VOICE produces the audio by closely matching it to human speech after taking into account five emotions – disgust, hate, anger, sorrow and joy.

EASI: EASI, which stands for ‘Electronic Assist for Sign Interpretation’, is for people who have a hearing impairment. It helps the user to convert the recipient’s voice into text, which is in turn displayed on the app. To add to that, EASI also has an option to customise the database, which allows the user to add personal signs or gestures so as to reduce the time lag between interpretations.

VISION: VISION allows safe navigation for the visually impaired by allowing them to detect objects and people in the surrounding environment. It does this through environmental analysis and facial recognition. This solution also offers educational assistance by allowing the blind to store content within the app and refer to it later.

Padam says the app will be open for free subscription for the first 100,000 users. A minimal fee will be charged later for certain special features.

Understanding the challenges 

During the course of testing the app, the founders collaborated with many government schools and colleges meant for the differently abled. “We interacted with about 150 deaf, blind, and mute students in order to understand their communication needs and difficulties, smartphone usage patterns, as well as their expectations from an app like Practikality,” says Padam.

After collating data points and feedback from them, the team used machine learning and a self-trained ‘Convolutional Neural Network’ to constantly improvise the app's features to reach one step closer to their goal.

“I was in constant touch with the co-founders throughout the testing of the app. As and when I was exposed to the voice-to-text feature, I felt independent since I didn’t have to rely on anybody’s help. The real-time conversion of ongoing conversations was great,” says Alok Bansal, a user with a hearing impairment.

According to a report published by the World Health Organisation (WHO), nearly 16.5 crore people across the globe suffer from some form of disability or the other. In India, the number stands at 2.68 crore as per the 2011 Census. Of this, 1.2 crore people have problems with either speech, hearing or visual impairment.

Another app available to help the differently abled is ‘Good Vibes’, developed by Cheil WW, an advertising agency. The app enables two-way communication among the deaf-blind by syncing Morse Code and mobile phone vibrations.

Last year, the Practikality app was also recognised at the Microsoft Imagine Cup Indian Nationals as the ‘Best Project on Accessibility Track’. The team was also a national finalist at the Intel IRIS National Science Fair in 2017.

[Also read: The Raised Lines Foundation at IIT Delhi helps blind students ‘see’ diagrams in their science books]