MathMusic

MathMusic enables access for visually impaired students to STEM education by conveying vision through sound in a dedicated platform.

It is a widely-accepted notion that the skills provided by Science, Technology, Engineering and Mathematics (STEM) education are immensely crucial for our innovation-dependent, tech-driven world. However, despite the significant efforts in the integration of students with disabilities in general education activities – teaching STEM and related fields to visually impaired people (VIPs) still remains a huge challenge.

With significantly lower representation of VIPs in STEM fields, and with pin-boards and swell papers still being the most widespread method to convey mathematical visualizations to VIPs – it is clear that there is a huge need in developing innovative methods in order to finally create accessible education and enable people with visual impairment reach their full potential.

With the MathMusic, we aim to do just that. The MathMusic is a platform both for the educator and for the student that is built around a proprietary, innovative algorithm to convey mathematical visualizations (such as graphs and geometry) through a combination of sound and speech. The user, whether an educator or a student, can create/upload images of mathematical visualizations and hear them through our vision-to-sound algorithm, with the help of computer vision and image processing methods. in the website there will be also a community area with teachers and students uploading lessons.

This method was developed after years of research with VIPs on the topic of sensory substitution in the lab of Prof. Amir Amedi, which is a co-founder, in the Hebrew University. There, we have created a new way to substitute vision through sound using computer vision algorithm and showed how areas in the visual cortex of people blind from birth are activated when they "see" through sound.

We have already developed the prototype as well as a training program for learning the algorithm. Initial testing with visually impaired and blind participants showed great promise. These participants managed to solve Real graph problems with our online platform, using only sound. Participants with no knowledge on graphs (as they were exempted from it in high school like most VIPs nowadays) managed to solve these problems in 2-3 hours of training. Participants rated the platform and its potential very high, and were extremely enthusiastic about it, saying they wished their mathematics teacher would see them just now.

We believe that the reason for the lack of accessibility of mathematical content to VIPs is actually a technical problem – a problem of conveying content that is considered "visual" through other, intact senses in a way that will enable to use it successfully. After years of research resulting in deep know-how, we have found the most efficient way to convey visual information through sound in an online, accessible platform. We believe that through this method we can create the primary platform and community to teach STEM subjects to blind and visually impaired people worldwide and by that increase independence and participation in STEM professions among VIPs. We believe funding and support like the SOLVE initiative offer can dramatically increase our chances to reach that goal.

The MathMusic is a technology based on the approach of sensory substitution and on previous prototypes that were developed as a mobile application and online. The training program for the MathMusic is also partially based on previously developed applications. However, This is the first time the methods of visual-to-auditory sensory substitution and topographic speech are combined in the context of STEM education. Also, we have developed a new method to convey graphics using OCR to soundscape. Lastly, we have developed a dedicated platform specifically for this project.

Our proprietary technology is what enables to create a new kind of eyes-free access to visual information, through sound. Our algorithm enables to efficiently percieve the mathematical visualziation through soundscapes. Also, the platform we have developed enable us to create a community of users and educators who can upload different lessons, as well as upload images from different sources (e.g. a text book) and hear them. Lastly, using a graph generator, VI users can create their own graphs.

In the next 12 months we would like to pilot the solution both in Israel and in Boston by creating partnerships with the relevant organizations, e.g. the Perkins School for the Blind and the American Printing House for the blind, and to develop and imrpove the solution based on the feedback. in the end of the next 12 month we would like to already have paying users/schools on the technology.

we plan to scale our solution in two ways:

1. Regions - we will start with the area of MA and in Israel but later on will expand to other states in the US as well as other some countries in Europe by using partnerships we are already working on creating and new collaborations.

2. a future direction will be to use the same solution for people with dyslexia and dyscalculia, enabling a new form of access to mathematical visualizations which might benefit in better performance.

The solution is an online platform, and we will reach customers through organizations for the blind and visually impaired. We already have in our advisory board the head of technology of one of the biggest organizations in the US. These organizations will be part of the development process and will help us to access potential users.

It is a prototype, so not yet.

in 12 months we believe we can get around 1000 users, and in 3 years we would like to be a mainstream technology for the education of people with blindness and visually impaired. we believe this will yield around 100000 users.

we are combining both experts in the field of sensory substitution with blind people themselves who are at the center of the development process.

We would like to sell through a subscription model that will enable access to the platform, content and of course the algorithm itself.

The model will be selling directly to costumers (mainly students and parents of VI children) and to schools. Any school/state is obliged to cover expenses for assistive technology for visually impaired children.

We believe the recognition from Solve and the partnership and mentors Solve has will enable us to pilot our technology as soon as possible and to connect to key opinion leaders which can help us in penetrating the market and advance our technology.

the main barriers we see are (1) costumer acquisition - it might be hard to reach potential users, thus we plan to do it through organizations for the blind and KOLs - and we will need contact to both, which we believe Solve can help with, and (2) Market education - the need to create awareness and knowledge about the amazing possibility in access to STEM. This "education" of the market needs to be both to the potential users and to their parents and teachers which got used to the fact they have an exempt for many of these topics.