The MathMusic platform

Currently, there is a significantly lower representation of people who are visually impaired in STEM fields. This is a result of inadequate tools to provide mathematical content - especially visualizations like graphs and geometry.

Today, we present you with the MathMusic platform. Using a unique combination of AI and vision-to-sound algorithms, we were able to come up with the first, most advanced method of conveying any graphical content to students who are visually impaired. Our algorithms are scientifically validated, stemming from years of research in the Hebrew University of Jerusalem, and our approach, which is demonstrated in the attached presentation, was tested with actual users.

We firmly believe that the MathMusic will revolutionize the field of accessibility of STEM content to students who are visually impaired – enabling them for the first time a much wider set of opportunities for employment, for achieving their goals and ambitions with no barriers.

We are working towards solving the inequality of access to STEM content faced by students and pupils with visual impairment. These students are facing huge challenges in access to visual mathematical content because of inadequate tools today - which in turn creates a drastically lower representation of this population in STEM fields (see here and here)

This is a huge problem of inequality and social justice, both for students, and their parents and teachers which are eager for solutions to advance the student in STEM. This lack of access, especially compared to other peers in class - create a straightforward inequality and a barrier for children to fulfill their amazing potential.

Recent studies show that in the US alone there were above 547,000 children with vision difficulty (2019) among them, according to the American Priniting House for the Blind, in 2018 were registered 55,249 U.S. children, youth, and adult students in educational settings who were legally blind.

Globally there are more than 330 Million people who are blind and visually impaired - with more children dealing with visual impairment in developing countries.

Please find our presentation here, describing our solution visually.

, describing our solution visually.

The MathMusic is a tablet application that enables true access to any kind of STEM graphics for students who are visually impaired, by conveying the visual content through a method combining sound and speech. Using advanced artificial intelligence, the platform can take any graph, and translate it into a "soundscape" - thus enabling users to hear the graph and solve STEM exercises without the need of seeing the content.

This is done by a platform with an accessible, simple interface relevant for our audience, that enables students, teachers and caregivers to upload graphs or geometry exercises and hear them. Our AI engine knows how to take scans and even graphs captured in real-time by the camera, analyze these images, and turns them into comprehensive content for our users. In addition, a resource portal for caregivers is provided, enabling uploading lessons from around the world. In addition games our provided for the students for better learning experience.

Our algorithms are scientifically validated, stemming from years of research in the Hebrew University of Jerusalem, and our approach, which is demonstrated in the attached presentation, was tested with actual users.

Our target population is divided into 3 groups:

- children with visual impairment which are currently severely underserved with virtually no access to advanced graphic STEM content. Answering this dramatic unmet need will provide them with equal access to these important fields, just like their peers in class, and will enable them to later pursue employment and careers in whatever direction they choose with much less barriers.

- Parent of these children which are eagerly looking for solution to help advance their children and save from them the feeling of being disadvantaged with no reason.

- Teachers of children with visual impairment which most of the time are teachers of regular classes and face for the first time a student with visual impairment. The platform will provide them with the most advanced tool for teaching this population, including exploring exercises and lessons uploaded by users from around the world, a platform to create and share exercises to the students themselves - to make sure the student is processing just like the rest of the class.

The Mathmusic enables true access to STEM content to students with visual impairment, hence drastically increasing their engagement with STEM fields, while enabling them the amazing feeling of independence.

In addition, the platform provides teachers with novel tools to adapt their educational methods to this population.

We have tested our methods with users who are fully blind. We are also researching and progressing our platform in tight collaboration with the Israeli School for the Blind.

Our unique and innovative approach was tested with people who are fully blind, from birth. In the first stage, we started with adults.

In the experiments, the results were remarkable. Showing how, after a short practice, users were able to solve real graph problems, through sound.

Here is a link to a video of our participants' responses.

Please see our presentation for a walkthrough.

Our solution is innovative in almost all of its aspects:
- Our vision-to-audio sensory substitution method enables users to hear the graphs "soundscapes" through an IP-protected combination of sound and speech. This sensory substitution approach was researched in lab settings, also showing activation in the visual cortex for people who are blind from birth (to read more about sensory substitution approaches

- Our AI engine enables for the first time users to upload any kind of graph and even capture an image of a graph in real-time and have access to it - even while being in lessons. The engine analyzes the text and shapes in the image to provide users access.

 - Our resource portal enables for the first time a resource center for teachers, students, and caregivers for children with visual impairment, easing for them the process of supporting the student in studying.

- Our gaming center provides for the first time games for our visually impaired students to make STEM more fun.

Currently, our solution is in prototype testing mode.

However, in 12 months we believe we can reach with the platform to 850 users in Israel and in a selected beta site in the US (this is after pilots in Israel and US in the next 12 months). Following this large-scale pilots, we will expand to other sites, attempting to reach, in five years, to 95,000 users in the US and European markets.

We will measure the amount of exercises students are capable of doing following the use of the platform compared to before.

We will measure grades on certain topics and will measure how much of the students choose to solve graphic content-related exercises on tests.

Finally we will test how many of our students will proceed to STEM-related fields.

We are four people, 2 developers (one serves as the CEO),1 product manager and 1 biz dev.

Half of our team is with visual impairment.

we are combining both experts in the field of sensory substitution with
blind people themselves who are at the center of the development
process.

Half of our team are people with visual impairment, which provides us with the most valuable information on the development of products for our population. Oren Ganor, our product manager, also has years of experience in working with children during his role at the Israeli School for the Blind.

As said, our approach for the company is to strive for at least 50% of the team being people with blindness/visual impairment or their caregivers. This is not only for agenda. We believe this kind of expertise is crucial for building an actually meaningful product for our audience.

We appreciate the grant purpose the MIT SOLVE organization for such an amazing opportunity for projects like ours from around the world.

Project like ours deals with numerous challenges: from financial needs through product development and penetrating different markets, especially governmental institution which can be challenging – while overcoming cultural barriers.

We strongly believe joining the MIT SOLVE community will be crucial for the MathMusic to come to life as a product and to create a new form of independence for students with visual impairment. Besides of funding, SOLVE can dramatically help us by the network it provides, and advice from the community (which we would like to contribute to as well) regarding penetrating markets like the US, while overcoming cultural barriers.

1.     Organizations for people who are visually impaired focusing on education, like the Perkins School for the Blind and The Texas School for the blind

Partnership would include piloting the device and assistance in integrating it into schools while improving furthermore the accessibility and the value it brings to our users.

2.     Experts in public policy and distribution of application and devices to the education system in the US

Partnership will include advice and guidance in preparing a product for the market

3.     STEM Teachers’ organizations

Partnership will include suiting the platform according to k-12 curriculum and creating pre-made lessons for users

4.     AI experts to further improve our algorithms

Partnership will include data labeling and improving algorithms

5.     Researchers in the field of special education – partnership will include help in making an unbiased, controlled research in the impact of our platform on students

6.     Accessibility experts in universities – partnership will include working together to make sure all content is accessible.

With years of inequality of students with visual impairment in their access to STEM materials, with the MathMusic platform we can finally encourage them to take the path of STEM fields and career with more safety. As part of the platform, we will promote, together with schools across the country, a campaign for raising awareness for the abilities of student with visual impairment to strive in STEM careers, encouraging and empowering students in primary and secondary school to continue and advance in STEM despite challenges they might be facing. The resources portal in the platform, together with the unique unprecedented abilities it provides - will further help them do that. We will be related that ASA and MIT SOLVE will be part of this campaign and the message behind it.

The MathMusic platform main target is to make STEM education much more accessible and equitable. Providing true, unprecedented access to graphics in STEM materials, Students who are blind can solve exercises just like their peers – and further pursue careers in STEM fields with more confidence than ever. Through the MathMusic, we will make STEM education finally equal to students with visual impairment, with the target to raise the participation of this community in STEM fields, and empower the community by drastically reducing barriers.

In the MathMusic platform, we innovatively use AI and advanced image processing techniques to extract important shape and text data from images. Our AI-engine enable users to scan images and even take a picture of a worksheet, to gain accessibility to this exercise. This is especially important in a class environment: when a student is in class and the rest of the classmates get a physical worksheet. Being able to have access to the worksheet, instead of waiting for the topic to change to things she is now capable of doing - will enable the student a true feeling of independence and inclusion – and will drastically change her life. There is no reason, besides of technical reasons, children with visual impairment can’t do STEM topics. And this is where AI fits in. With our AI expertise (developing the first mobility assistance for people who are blind) we are taking up the challenge of using AI for drastically increasing STEM education for students who are blind or visually impaired

In the MathMusic platform, we innovatively use AI and advanced image processing techniques to extract important shape and text data from images. Our AI-engine enable users to scan images and even take a picture of a worksheet, to gain accessibility to this exercise. This is especially important in a class environment: when a student is in class and the rest of the classmates get a physical worksheet. Being able to have access to the worksheet, instead of waiting for the topic to change to things she is now capable of doing - will enable the student a true feeling of independence and inclusion – and will drastically change her life. There is no reason, besides of technical reasons, children with visual impairment can’t do STEM topics. And this is where AI fits in. With our AI expertise (developing the first mobility assistance for people who are blind) we are taking up the challenge of using AI for drastically increasing STEM education for students who are blind or visually impaired

With the MathMusic we use an innovative combination of AI, and a concept called sensory substitution, to enable students who are blind first-time access to graphical STEM materials.

Our vision-to-audio sensory substitution method is a unique conversion algorithm that takes actual pixels in the image and convert them into “soundscapes”, enabling users to actually perceive the shape as if they were visual. And this is also reflected in their brains – numerus papers have shown how areas in the visual cortex of people who are blind from birth are activated using this kind of algorithms.

With this innovative approach, we were able to actually convey real graph exercises to people who are blind from birth, and they solved these graphs after a short training session.

Our AI, from the other hand, enables users to upload and scan graphs in real-time to enable them feel just like the other classmates: getting an exercise and solving it – equally to everyone else.