Language Jungle

A gap exists in the home environment for childhood language acquisition. Some parents can afford to hire a professional babysitter who could make positive impact on language acquisition. Some have sufficient time and knowledge to engage in desirable linguistic interaction with their children. But others don’t. Closing this gap is a massive problem to solve. Existing solutions need a labor-intensive approach and consideration of parents’ socioeconomic situation. Our solution could help the have-nots by partially automating linguistic interactions with their child and raising their awareness.

Recent research (Suskind, 2015) suggests that less opportunities for linguistic interaction in early childhood lead to negative impact in a child’s intelligence. We would like to address this problem seen in poverty class in American and Japanese parents who cannot afford to give sufficient linguistic interaction with their children due to a lack of time, money, or knowledge. Our solution is to create an environment with affluent multi-modality and interaction to facilitate language acquisition.

We are mainly working with parents who have their children under 3. From a socio economic viewpoint, our target population includes families in low-income class, as well as other families who can't give sufficient linguistic interaction with their children due to a lack of knowledge, time, and motivation. Geographically, our target audiences are mainly in Japan, the US, and Europe.

To understand their needs, user interviews have been conducted with families in the US and Japan. User tests to receive feedback on the solution are being conducted with families in Japan at their home or facilities like childcare center where children from the community gathers. 

As we aim at providing a solution with multi-languages such as English and Japanese, localization work and local user tests has to be done basically. However, we believe some findings in user tests such as intuitiveness of the system usage for child will be commonly utilized beyond differences in language.

Language Jungle (LJ) is a product that creates interactive linguistic environments at home or facilities and facilitates language acquisition in early childhood. Image 1 explains overall product concept with required technologies and the scope of the current prototype. LJ mainly consists of the following elements: - a microphone to record a child’s utterances (see a1 in Image 2) - an algorithm to analyze the recorded child’s utterances and determine the timing and the content of the feedback (b) - display(s) (e.g. speaker, projector, tablet, toy) to give multimodal feedback (e.g. vision, hearing, olfaction) based on the analyzed result (a2, c) The flow of LJ usage is as follows: 0. (If necessary) LJ or parent feeds some inputs to trigger the baby's utterance 1. The child gives utterance which is recorded with the microphone 2. Analyzed results of the child’s utterances are converted to effective outputs rendered on the displays in a multi-modal way 3. The rendered outputs are reflected to the child as feedback to facilitate further utterances 4. Audio feedback can also be given so that the child can learn from it 5. Insights gained through the linguistic interaction between the baby and LJ are shared so that the parent can raise their children. Through interactions with LJ, the child is actively encouraged by multimodal feedback to vocally communicate with the world around them . Interactions directed by the child would create more opportunities for utterances and learning based on enriched feedback. Language and even self-efficacy are expected to be acquired eventually by using Language Jungle.

Our solution is innovative in terms of responsiveness and awareness of child's action. In the positioning map below, our solution's positioning is depicted in comparison with relevant products for language acquisition using voice related product/technology. 

(1) is a type of products called "word pedometer" to count and analyze parent's talk to their child e.g. Oyalabs' Wordle and VersaMe's Starling. This would help parents keep motivation but would not access to the parents who still need to raise awareness on the language acquisition issues.

(2) is a product type to enhance story time experience by adding sound effect and background music in response to parent's reading e.g. Novel Effect , Google Home's similar app. This would entertain parents and their child but would not access to the parents who don't have sufficient time to engage with their child.

(3) is represented with SoapBox Labs' Children's Speech Recognition technology. It has an advantage to directly engage with children and enable them to search the Web, give a command, and correct their speech by themselves. However, they target children over 3, focus on speech recognition, and don't provide proactive response.  

Our solution (4) is designed based on the “guided play” concept (Hirsh-Pasek et al., 2009) to initiate "play" as a proactive response, where the system is aware of parent's and child's voice. With this, children's attention can be attracted by showing reactions to their voice. Also, it can trigger interaction with the children even when their parents doesn't stand by closely. 

Language Jungle has an AI agent like Alexa to listen and talk to the parents and the child, and a visual image that reacts with it to attract the kid’s attention. - Throughout the session, Language Jungle get the parents involved with the child’s learning.

Our AI distinguishes voice of parents and utterance of infants under circumstance of daily communication. The uniqueness of our AI dedicates on the recognition and analysis of utterance of infants less than three years old. Although technology of voice recognition AI is matured and becomes commodity especially for voice of adults,  voice  recognition AI of infants less than three years old needs to be developed.

First, in order to collect teaching data, we gather audio interactions between parents and an infant. Teaching data set includes utterance of Infants, voice of parent, and recognized label based on the voice of parent.

Next, our technology learns the interaction between parents and a baby. AI of language jungle shall understand which patterns of voices of parents are triggering utterance of a baby, and level of emotion of the baby. 

Finally, language jungle generates a sound with characteristic of parents’ sounds and relevant images based on label, and the baby responds the sound and image.

Our system iterates the process above and make the intervention of parents less from the entire process. Ultimately our technology achieves interaction with an infant without existence of the parents while increasing the level of language acquisition of the infant.

As the Activities, our product Language Jungle is provided through libraries or childcare centers to influence families. By using our product, parents would be able to have sufficient knowledge (e.g. 1)existence of word gap, 2)practical tips on parent talk, 3)importance of conversational turns) and better engagement with their child for language acquisition, which is regarded as the Outputs. For Short Term Outcomes, not only the parents who uses our products but also the other parents in the same local communities would appreciate the benefit of efforts toward child's language acquisition through viral communication by the influencer in each community. Eventually, as the Long Term Outcomes, the statistical variability of NAEP vocabulary assessment for 4th grade students would be reduced due to improvement of language environment at home.

Reference:

1) Hart, B., & Risley, T. R. (1995). Meaningful differences in the everyday experience of young American children. Baltimore, MD, US: Paul H Brookes Publishing.

2) Suskind, D., Suskind, B., & Lewinter-Suskind, L. (2015). Thirty million words: Building a child's brain: tune in, talk more, take turns. Dutton Books

3) Romeo, R. R., Leonard, J. A., Robinson, S. T., West, M. R., Mackey, A. P., Rowe, M. L., & Gabrieli, J. D. E. (2018). Beyond the 30-Million-Word Gap: Children’s Conversational Exposure Is Associated With Language-Related Brain Function. Psychological Science, 29(5), 700–710. https://doi.org/10.1177/0956797617742725

the current number of people you’re serving: 5 families

the number you’ll be serving in one year: 10,000 families

the number you’ll be serving in five years: 10,000,000 families

Short-term = 1 year: Launch our service in the US and Japan

Long-term = 5 years: Launch our service in Europe

1 year 

financial:

technical:

legal:

cultural: Parents feels guilty or laziness about themselves if they use any tablet or digital services to educate their infants without their physical presence with their babies.

market barriers: This is our biggest barrier how to enter the US market as we are based in Japan and consisted of Japanese members currently.

5 years

financial:

technical:

legal:

cultural:

market barriers: how to reach millions of families in the US, Japan and EU to serve 10M families.

1 year

financial:

technical:

legal:

cultural: partnership with institutions to prove scientific impact from our service

market barriers: Applying SOLVE to find our market dynamics in the US.

5 years

financial:

technical:

legal:

cultural:

market barriers: 

1 full-time staff, and 4 part-time staffs. As part of support from SONY's accelerator, one advisor on business development is assigned to us from SONY. In addition, one advisor is in San Francisco to give advise on how to address the US market.

MITSUGO Project is a project team with collection of individuals who have different expertise such as robotics, learning design, and learning science. As most of us have own child, we have strong passion to work on issues in early childhood especially for language acquisition. When knowing the fact that “word gap” problem can’t be overlooked, we thought there may be some way to solve the problem at least partially using technology. As it is quite natural for us to have user’s perspective against the problems in early childhood, we can come up with ideas and test them easily and quickly. We also have motivation to solve own problems or problems we can emphasize. Our team owns technical competence at least for prototyping and domain knowledge in early childhood development. We have a robotics expert who had experiences to build interactive systems, a learning scientist who is studying about confirmation bias, conceptual change and knowledge system in childhood, and a learning designer.

We have been participating SONY's Acceleration Program based in Tokyo, which provides an office space, regular mentoring, and other services such as access to their own crowd funding platform.
https://www.sony.net/SonyInfo/csr_report/innovation/index3.html

Also, NEDO (The New Energy and Industrial Technology Development Organization, Japan's largest public management organization promoting research and development as well as deployment of industrial, energy and environmental technologies) is going to offer funding for further product development.
https://en.wikipedia.org/wiki/New_Energy_and_Industrial_Technology_Development_Organization

In addition, we are listed in Promise Venture Studio's Venture Index. Within the community, findings and resources to understand how to address issues on Early Childhood Development in US market is provided. 
https://ecdecosystem.knack.com/app#ventures/

To address both families in low-income class and in the other classes who need of language acquisition aid, two different business model are under investigation: When approaching to the former segment, we shall need to deliver it to the end users in an affordable way. We would like to incorporate government support to solve this social economical problem together. In addition, as collected usage data of Language Jungle can be invaluable for child development research, Language Jungle could be a part of reserach program and distributed as a tool to collect data for research. When approaching to the latter segment, a simple business model, where Language Jungle would be sold as a commercial product and end users pay the fee in return, would be applied. We will build more concrete business models and verify them for both US and Japan market as the next step.

We aim organizational support model. Our users are both low-income families and non-low income families. With regards to low-income families, a government or other social supporting organization such as libraries and/or childcare centers at local will be a customer who purchase our service and provide user license for free to the low-income users.

SOLVE can help us entry to US market.

Concretely;

Advance the development of our AI that adapts to multi-language base.

Access to users, commnuities, partnering with public sectors

To conduct market research and user research, we still need partners who could support the following actions:

 - Investigation of US’s social economic situation related to language acquisition

- Access to target audience (e.g. child facilities or parents who has child from 0 to 3)

AI to make the interaction adoptive

We utilize the prize to the following activities:

We collect teaching dataset of interactions of parents and infants in order to develop our AI in multiple languages.

We develop our AI to work under any circumstances in daily communication.

We enrich generated sounds and images to keep attention and communication with infants longer than 15 minutes.

As the usage data of Language Jungle stored as part of life-long data could be invaluable for research on early childhood development, our idea have potential to impact the science community positively by sharing the collected data.