Submitted
Learning for Civic Action Challenge

AI-based solutions for Deaf people

Team Leader
Mehdi Masoumi
Solution Overview & Team Lead Details
Our Organization
Deaf AI
What is the name of your solution?
AI-based solutions for Deaf people
Provide a one-line summary of your solution.
Deaf AI: AI-based digital humans as sign language interpreters
Film your elevator pitch.
What specific problem are you solving?

There are more than 400 million people who only communicate using sign language. They desire equitable access to all aspects of civic life, including critical services that are frequently unavailable on-demand. While unemployment and underemployment continue to affect them disproportionately, those in the deaf community are relegated to the role of factory worker or hard laborer. About 8% of deaf or hard-of-hearing adults in the United States actively sought work in 2018, but fewer found it, with most only finding part-time positions—and only 39.5% employed full-time, compared to 57.5% of hearing peers. The same disparities exist in education. Despite the Americans with Disabilities Act (ADA) rules, normal schools and colleges are rarely structured in a way that allows Deaf and hard-of-hearing students to thrive, and only a few deaf and hard-of-hearing educational institutions exist. Only 18% of deaf or hard of hearing (DHOH) people hold a bachelor's degree or above, compared to 33 percent of working-age hearing individuals. These job and educational obstacles have a rippling effect. Deaf and hard-of-hearing people are already at a higher risk of depression and anxiety. However, data from psychologists and sociologists suggest that insufficient work is associated with a variety of mental health problems, as well as chronic diseases and substance abuse. Poverty has its own influence on health. According to studies, low-income people with minimal education are routinely less healthy than their more educated, richer colleagues, particularly among minority groups. Socioeconomic status and education levels have been connected to a variety of health consequences, ranging from low birth weight to diabetes. Because many of these issues are linked, resolving them will not be a straightforward legislative remedy. While many deaf and hard of hearing people get financial assistance from programs such as Social Security and Assistance and Disability Benefits Income, more may be done to promote equitable access to work, society, and education.

Factual information and numbers in ASL

  • There is no universal sign language in the world. 
  • The linguistic structure of English differs from the linguistic structure of American Sign Language. As a result, English is not the first language of d/Deaf and hard of hearing. For that reason, closed captioning and subtitles are in English and are not fully functional to Deaf people.
  • Sign Language has face movements and facial expression which are a way to convey emotion to Deaf people. This makes the technology development harder to understand the emotion from the text.
  • Developing Large Language Model and NLP in ASL and make AI-based technology for interpretation is so sophisticated. Google, Microsoft, and Open AI (chatGPT) can't offer such services because only in ASL, there are many variables, localized signs, pop-culture signs. These companies (e.g. Google) do not have datasets of all signs. 
What is your solution?

For people who only communicate in sign language and seek equal access to all aspects of civil life, such as essential services, we train machines with artificial intelligence to do sign language in order to improve the experience of communicating with people who don’t know sign language and make the real world (society) and virtual world (digital world) more inclusive. We assist service providers, organizations, and businesses to be more accessible to Deaf people and to serve these people in sign language on demand.

Our mission is to serve deaf people so they have better experiences in public life and the digital world through AI-based solutions with an ultimate vision is to develop digital humans as sign language interpreters for real-time interpretation of voice into sign languages.

As our vision is to make the transition from human-based services to AI-based services in sign language by learning machines to translate the voice into sign language. Sign language would be affordable, available, and accessible to service users (deaf people) and service providers (any organizations, businesses, or companies that need to be more accessible). Picture this, you watch a video on YouTube, and instead of using a subtitle, there is another option. By clicking on this option, a virtual avatar pops up in the corner of the screen and translates your voice to sign language. We would diversify options for Deaf people to better access the digital world. Online learning platforms like Coursera, webinars, online meetings, and the workplace would also be a better place for deaf people as they could communicate with their colleagues, especially when working remotely.
Here in this section are the most unfair advantages of Deaf AI, which not only positions itself against competitors but also steps into some markets that were neglected by others, and these markets have always served deaf people. 

When it comes to AI in ASL, the first solution is to make a huge dataset for the universal application of translating any sentence, which requires an enormous amount of time and money. However, sign language is so complex that it involves more than just the movement of hands and fingers. Facial expressions are an integral part of ASL and are used to convey emotions to Deaf people. Moreover, there are some localized signs that each deaf community has for itself. In order to understand the customer behavior of Deaf people, we should understand their culture, which has a direct bond between them.
Consequently, we at Deaf AI selected niche markets to align the limitations of technology with market demands, mitigate the risk, respect the Deaf culture, and create a few localized signs and facial expressions. The first market is airport terminals, followed by sports video games and live sports events.

Who does your solution serve, and in what ways will the solution impact their lives?

Deaf AI is committed to improving the lives of Deaf individuals and the d/Deaf community by providing a solution that can enhance communication access in public spaces. Our target population is primarily Deaf and hard-of-hearing individuals who rely on American Sign Language (ASL) for communication. This group has traditionally been underserved due to a lack of accessibility options in public spaces, which can lead to social isolation, reduced access to information, and difficulties with daily activities such as navigating public transportation or attending medical appointments.

Our solution addresses these needs by providing a virtual avatar that can translate public announcements and other important information into real-time ASL. This technology aims to improve communication accessibility in public spaces, reducing the barriers that Deaf and hard-of-hearing individuals face. By making public spaces more accessible, we aim to increase social inclusion, provide equal access to information, and enable individuals to participate more fully in daily activities.

In addition to directly serving Deaf and hard-of-hearing individuals, our solution can also benefit the wider community by promoting accessibility and inclusion. Our technology can help create a more inclusive environment for all individuals, regardless of their communication needs, especially in the future when Deaf AI covers more sign languages than ASL. By increasing accessibility in public spaces, we aim to promote greater understanding and respect for Deaf culture and the importance of communication and equal access to essential information. 

Overall, our solution has the potential to significantly impact the lives of Deaf and hard-of-hearing individuals by providing a tool that can enhance communication access and improve overall quality of life. By addressing the needs of this underserved population, we aim to promote greater accessibility and inclusion for all individuals in public spaces.

How are you and your team well-positioned to deliver this solution?

Our small team includes diverse people with disabilities from underrepresented communities and people of color with complementary skills, i.e., the minimum viable team; an entrepreneur, a seasoned manager, an AI scientist, a deaf AI/ML engineer, an ASL interpreter, and AI interns. However, Deaf AI went beyond forming a solid team because developing such sophisticated technology requires strategic partnerships with a group of people rather than individuals. This technology is a combination of ASL knowledge and d/Deaf culture, NLP, computer vision, Machine learning/Deep Learning, computer graphics, and animation and visual production. 
This is the reason our product is "deep tech." 

Therefore, we partner up with the Screen Industries Research and Training Centre (SIRT) to support Ontario’s film and television, startup, and interactive media clusters. SIRT is a government-funded Technology Access Center that includes dedicated staff, as well as faculty and students to conduct world-class applied research, training, and other industry support activities that enable the adoption of new technology.

SIRT supports Deaf AI in product development. Besides, Transport Canada is another partner for Deaf AI, committed to making airports more Deaf-friendly and improving accessibility for Deaf passengers at the airports. 

We also work with a few ASL interpreting consultation firms to support us on quality controls and feedback on the product.  

We are actively searching for connections with other NGOs and philanthropic foundations, such as the Ford Foundation, airlines, and local governments. 

Which dimension of the Challenge does your solution most closely address?
  • Help learners acquire key civic skills and knowledge, including how to assess credibility of information, engage across differences, understand one’s own agency, and engage with issues of power, privilege, and injustice.
In what city, town, or region is your solution team headquartered?
Toronto
In what country is your solution team headquartered?
  • Canada
What is your solution’s stage of development?
  • Pilot: An organization testing a product, service, or business model with a small number of users
How many people does your solution currently serve?
we are at the pilot study
Why are you applying to Solve?

Companies with social impact need a network of like-minded people, with beliefs about making the world a better place for everyone. 

We are looking for a network of various stakeholders, such as local governments, impact investors, global leaders, and foundations. Raising money is the least of your concerns if you have a strong network of like-minded people who share your vision. 

In which of the following areas do you most need partners or support?
  • Business Model (e.g. product-market fit, strategy & development)
  • Financial (e.g. accounting practices, pitching to investors)
  • Human Capital (e.g. sourcing talent, board development)
  • Legal or Regulatory Matters
Who is the Team Lead for your solution?
Mehdi Masoumi
More About Your Solution
Your Team
Your Business Model & Funding
Solution Team:
Mehdi Masoumi
Mehdi Masoumi