Feel the Rhythm

We took up the Next Sensory Augmented Hearing Aid project (NSAHA) as part of the Japan-India Transformative Technology Network, a multi-year series organized by Salzburg Global Seminar and the Nippon Foundation.

NSAHA is a technology working towards assisting people with hearing disabilities to communicate easily with others. We have developed an AI-enabled hardware device and application that would allow differently-abled individuals to live an everyday life with dignity and pride. It is a project that aligns with the mission statement behind Sustainable Development Goal 3 - Good Health and Well-being – “ensure healthy lives and promote well-being for all at all ages.”

Since some people with hearing impairments cannot fully hear sound, they cannot understand some sound characteristics, such as rhythm and patterns. This project aims to use technology to convert sound into vibrations, light, and letters to convey sound characteristics to the hearing impaired and to create opportunities for them to experience the world of sound. The project not only enhances the senses but also augments the learning ability of the individuals.

NSAHA solution is a combination of a breakthrough hardware device, Ontenna, and a complementary AI application, Lip-re.

Ontenna is an accessory-type device that converts the volume of sound into vibrations and light intensity to convey sound characteristics to the hearing impaired. 60dB to 90dB of sound pressure is converted into 256 light and vibration intensity levels, allowing users to sense rhythms and patterns. Currently, about 80 percent of the schools for the deaf in Japan have this device, which is used for speech and rhythm practice.

By introducing Ontenna + Lip-re to schools for the deaf in India, we will create an opportunity for them to experience the world of sound. We will also create opportunities for children of deaf schools in Japan and India to connect by holding workshops in Japan and India where they can dance and perform using Ontenna + Lip-re.

Lip-re, short for Lip Reading, is an application that traces the movements of the lips and displays them in a large, enlarged format, making it easier for the hearing impaired to understand their movements. It also creates an opportunity for people with normal hearing to consciously move their mouths in a larger size when talking to the hearing impaired. These can be used to create smoother communication, regardless of disability. The application can be used more during pandemics when school classes are held online. Lip-re can be integrated with standard online communication tools such as Zoom or Google Meet.

Ontenna allows people of all nationalities, ages, and disabilities to feel the exact characteristics of sound, such as rhythm and pattern. By using this product, we will create a cultural and language connection between the young people of Japan and India. In addition, Lip-re can be used to facilitate communication between the deaf and the hearing impaired. We hope to use this product to create a connection between the children of deaf schools in Japan and India and between the deaf and hard-of-hearing people involved in the schools.

The technology will be helpful for individuals born with hearing disabilities – they can learn to read and understand words much quicker. For people with acquired disabilities (through old age or by accident), the technology will serve as an additional aid for regular communication. The technology will benefit family members, friends, and other stakeholders to communicate freely with people with hearing disabilities.

In the future, Ontenna + Lip-re can be integrated with wearables such as Google Glass to have a host of other benefits such as real-time language support. Glass-wearables can be an all-in-one device integrating the rhythm and pattern and Lip-re display on the glass in front of the eye. Integrating with glass-wearables will be a breakthrough as it would allow people to move and work freely without the fear of non-communication.

Deaf schools tend to have a relatively small budget compared to public schools. Therefore, although they may change blackboard erasers, chalk, desks, and chairs, they do not have the budget to buy things like Ontenna. The same situation can be expected in India, and the challenge is how to raise funds to implement such a program. The project has received a small grant from Salzburg Global and the Nippon Foundation for us to work in one school each in India and Japan.

Between April and May 2022, one school in Tokyo and one school in New Delhi were contacted to conduct the workshop on their respective premises. The support from the authorities, teachers, and students was overwhelming. Students participated in the workshop with great enthusiasm. Many kids could feel the sound for the first time in their lives. Imagine living without being able to hear the barking of dogs, singing of the birds, or the sound of firecrackers.

Through the workshop, kids were exposed to Ontenna and Lip-re, where they could appreciate and comprehend the vastness of the sound universe. For them, it opened a new dimension that most of them could not reach. The workshops in schools in India and Japan have allowed children to interact with others without any fear.

Potentially for the first time in their lives, children from the school were able to speak to their foreign counterparts over Zoom. It was a gratifying experience to see the young kids able to communicate without any language barrier.

Stephen L. Salyer, President and CEO of Salzburg Global Seminar, said, “I’m delighted to see Tatsuya and Nikhil develop this breakthrough project through Salzburg Global’s Japan-India Transformative Technology Network. Their work exemplifies what we and our partner, The Nippon Foundation set out to achieve — i.e., a platform to maximize the benefits of technology for social good.

“Like others in the Network, Tatsuya and Nikhil have shown how we can use technology to provide solutions to everyday challenges people face. They’ve shown how an existing hardware device and a complementary AI application can better enable people to interact – not only in one classroom but in multiple classrooms simultaneously across borders. As a hearing-impaired person, I am deeply interested to see where their momentum takes them next.”

Tatsuya and Nikhil are now looking for additional funding support from organizations to further assist them in developing the technology. In a joint message, both said, “We have already shown in two schools the power of this technology, but we need additional support to grow and take this project further.”

In addition, Tatsuya said, “It’s my dream to share the technology around the world. I wish to empower the deaf by giving them the opportunity to live a normal life”.

Nikhil, who has worked with women self-help groups, entrepreneurs, and youth worldwide, emphasized, “It’s our moral duty to build technologies and methods to bring people into the main fold. Inclusivity is not a fashion statement, it’s our ethical responsibility – we cannot leave people living on the fringe forever. I strongly believe technologies such as glass-wearables can provide such opportunity for deaf people to integrate with society.”

If you’re in a position to provide Tatsuya and Nikhil additional support, please email Nikhil.