Click Learning

Who (first and last name) is the Team Lead for your application and LEAP Project?

Rob urquhart

Describe the role the Team Lead plays in your organization. [100-200 words recommended]

Rob is Click Learning's Chief Research and Impact Officer and is a member of the executive team. Rob's responsibilities include:

• Developing and implementing Click Learning's monitoring and evaluation framework in order to understand the impact of the organisation's work
• Undertaking a range of qualitative and quantitative research activities - both internally and in partnership with others - that support the organisation's mission and monitoring and evaluation objectives
• Developing an evidence base of what works; to support Click's journey for impact and scale and the broader eco-system
• Defining and answering key learning questions that inform operations but also contribute to a broader body of knowledge
• Provide evaluative support to Click Learning's sandbox - a ringfenced operation set up to trial and iterate promising new Edtech products

Explain how your Team Lead and supporting team members are well-positioned to effectively support the LEAP Project, given other priorities within your organization. [200-500 words - recommended]

The potential questions to explore in the LEAP project in the next section, all form part of Click Learning's learning agenda. Moreover they all are integral to the sandbox that is being launched, which is a strategic initiative by the organization and has considerable resources (personnel and financial) associated with it. To that end, the LEAP project would not sit outside of Click Learning's work as a 'project' but would be integral to accelerating its objectives. The LEAP project would provide evidence that is critical to scaling Click Learning's impact. Click Learning's executive team would therefore be fully committed to supporting the project's work and avail necessary resources as appropriate. As team lead, Rob Urquhart, sits on the executive management team and will champion support and integration with the rest of the executive team. Rob's responsibilities for research, impact, and monitoring and evaluation ensures that the LEAP project has internal consistency, and that the necessary resources and access to data is provided.

One-line solution summary: In 20 words or less, summarize your organization's product or program that is the focus of your LEAP Project.

A sandbox to trial three new products and build an evidence base of what works, for whom, in what circumstances

Define the problem that your solution seeks to solve. [200-500 words recommended]

While the array of EdTech products available is increasing rapidly, very little quantitative evidence exists on which products deliver what outcome and under what circumstances. Furthermore, the vast majority of products are developed with a focus on high-income households and additional support is required to make them effective in under-resourced environments. 

In South Africa various products are being implemented by different funders and partners, each with their own evaluation criteria and generally all trying to grapple with how to measure impact in the absence of standardized testing. 

In order for edtech to have an impact on education at scale more research is required on what works in what circumstances. In line with a report by Central Square Foundation in India, “Decision-makers not only need better evidence on suitable approaches that would work and have an impact in their respective contexts, but they also need guidance and support on deploying EdTech products in ways that increase student learning.”

In response to these gaps Click Learning has established a sandbox (Double Click)  to:

• Identify promising edtech products that improve literacy, numeracy, and digital skills
• Test these products by implementing them in a controlled environment
• Understand their relative efficacy
• Define a path for scaling their implementation
• Evaluate their impact
• Build an evidence base of what works to shape wider adoption and financing

The sandbox provides a contained real-life space to test proposed interventions that allows Click to safely learn and adapt before rolling out promising ideas or products more widely. By working in conditions of uncertainty, the sandbox allows for ideas to be tested and grown through iterative experiments driven by a commitment to learn, and adapt to problems as they present themselves. The sandbox approach is designed to help generate evidence in real-time, and draws on concepts such as  lean impact and user-centered design to get to learning as fast as possible to make interventions as robust as possible.

Describe your solution and how it works in simple terms. [200-500 words recommended]

Our solution is to identify three initial products that we want to test in the Sandbox: Mindspark (numeracy), Amira (literacy), and MyLexia (literacy.) We will deploy these three products as three different treatments arms, with a supporting control groups in a set of no-fee schools randomly selected from populations of schools available to us. We will:

1. Start with a hypothesis about what learning gains these numeracy and literacy products will realise under what conditions and what circumstances. In the course of doing so, identify the assumptions for this hypothesis to be true (e.g. absence of loadshedding, sufficient connectivity for the products to function.)
2. Implement the products whilst acknowledging the assumptions, to test the hypotheses as  experiments. Based on the results, either persevere with the trial or pivot to either tweak implementation or test new hypotheses/conditions. 
3. Use a process of validated learning by collecting data from different sources for measurable success metrics to test the hypotheses. The validated learning process isn’t simply a case of ‘was metric x achieved?’ but asking questions through the data of why this was or wasn’t the case. This active, reflective learning can then be used to run more complex trials or adapt implementation.
4. At regular intervals have a decision point to pivot or persevere; to step back and consider whether the product trials are at a point of diminishing returns and need to consider a new direction, or are gaining traction and can proceed with more elaborate experiments.
5. Track the outcomes for panels of learners working with the products through each cycle of iteration to build a longitudinal view of impact over time.
6. Work iteratively to do so in order to build the learning and confidence that is needed, the rigour of the evidence base for wider adoption, and paths to scaling before making deeper investments in infrastructure and resources for implementation.

Learning variability is at the heart of the sandbox. We don't believe that there is one product for all, but rather that its important to understand what products, work for whom, under what circumstances and hence the choice of multiple products to test. Our sandbox design allows for responsiveness; to pivot and test different programming as we learn from implementation. There is heterogeneity in our treatment groups: across different grades (grades 1 to 7) in schools of different poverty levels, in different geographies (provinces and economic wealth.) And one of the products we are testing and are eager to learn from is an adaptive learning product that allows learners to advance their reading development at their own pace and at the right level of instruction.

Select the key characteristics of your target population. Select all that apply.

• Women & Girls
• Primary school children (ages 5-12)
• Rural
• Peri-Urban
• Poor
• Low-Income
• Minorities & Previously Excluded Populations

If you selected “other” please indicate which populations you serve.

N/A

In which country or countries does your solution currently operate?

South Africa

Upload your solution's Theory of Change or Logic Model.

Where would you place your solution on Nesta's Standards of Evidence?

To date, what research/studies has your organization conducted that have helped demonstrate the effectiveness of your solution? [200-500 words recommended]

Our work encompasses both formative and summative research. At the core of our research has been our eQuiz. Click Learning has developed with support from literacy experts and in alignment with nationally administered EGRS's, a digital assessment (e-quiz) that enables it to measure learner progress at baseline and endline. At the end of 2022, nearly 140,000 learners completed the eQuiz. A similar eQuiz for numeracy has recently been finalised. Examples of some of the work we have undertaken include:

• An independent evaluation of our eQuiz against a 2017 EGRA (early grade reading assessment) finding moderate correlation (the EGRA included an oral comprehension which our eQuiz isn't yet able to do)
• An independent analysis of Click’s data to explore whether learners who spent more time on the apps had higher literacy levels whilst taking into account the learners’ school environment and demographics. The analysis focused on learners that were active on the apps prior to the impact of COVID-19 and the associated school shutdowns. Using a regression analysis  and after controlling for various factors results show that learners’ literacy eQuiz results increased as their cumulative hours on the literacy apps increased.
• An external evaluation of Click Learning’s eQuiz to understand whether it was internally valid; whether proficiency in alphabetic knowledge leads to proficiency in word reading which in turn leads to proficiency in reading comprehension. The results show that for grade three learners, a 1% in their score in alphabetic knowledge results in a 0.71% increase in their word reading score. And similarly for grade five learners, a 1%  increase in their word reading score results in a  0.51% increase in their comprehension score. The evaluation finds that the results presented from Click’s eQuiz align with those from publicly available literature using both the EGRA and other tests which also show moderate positive correlations between these competencies.
• An internally conduct analysis the eQuiz results of a cohort of learners who participated in the programme in grades one and two in 2019, and the results of the same learners in grades 4 and 5 again in 2022. The analysis shows the benefit of sustained participation in Click Learning’s programme; with literacy for the panel of learners in grade 1 (2019) and grade 4 (2022) increasing by 55%. And literacy for the panel of learners in grade 2 (2019) and grade 5 (2022) increasing by 40%. 

What has the research/studies you have conducted revealed about your solution and how did it inform your work moving forward? [200-500 words recommended]

• The research confirms the validity of the eQuiz as digital assessment as an alternative to paper-based early grade assessments that have been the norm in South Africa but which are expensive and time-intensive to implement. This has allowed us to largely 'settle' our assessment instrument and in turn the frame by which we can understand impact and build evidence.
• The research points to the importance of time for learning outcomes; both in the amount of hours spent using products, as well as longitudinal time. One of the elements of variability the sandbox is going to test as a result, is a doubling of in-school learner time on product.
• The research results at the minimum show that the products we are currently using (i.e. not yet tested in the the sandbox) are associated with learning gains (even if there is still work to do in isolating this from natural grade progression but which we hypothesize in unlikely given the recently released 2021 PIRLS results for South Africa which indicate learning losses of a year for no-fee schools.) Importantly and not indicated above, is that we see these gains happening across the three domains of literacy that stack: foundations of literacy (e.g. letter recognition), word reading, and reading comprehension. This therefore shows us that product efficacy doesn't stop at the foundations, but feeds through to the more complex and ultimately desirable reading comprehension.

Describe your organization's need to strengthen the evidence base of your solution. [200-500 words recommended]

Click’s implementation and scale is supported by a platform that manages and tracks end-to-end delivery, from early design and school selection to implementation and monitoring. The platform extracts learner-level data from the government’s learner management systems, the South African School Administration and Management System (SA-SAMS) (via the Data-Driven Districts platform (DDD)), and from the Centralised Education Management Information System (CEMIS), used by the Western Cape Education Department (WCED). 

Click collects data on individual learner environments, usage and progression, A primary source of data for ongoing monitoring, evaluation, and learning is ClickMart, the data warehouse that gathers information on learners and their schools, learner progress through the programme and their literacy-assessment results (measured through  digital eQuizzes). The digital eQuizzes (numeracy and literacy) can be conducted quickly and at minimal cost, and results are available in real-time. This presents opportunities for dynamic numeracy and literacy assessment and research in South Africa, allows for baseline and mid/end-line measurement, and can be conducted at various points throughout an intervention.

In its 11 years of operations and work with over 200,000 learners Click Learning has successfully demonstrated:

• That is is possible to get learners in front of a digital device, in their schools, during school hours, at scale;

• And in so doing provide them with sustained access and interaction across a school year to numeracy and literacy improvement programs that provide personalized learning journeys;

• That learning outcomes for learners engaging with the product are improved;

• That basic digital skills (mouse navigation, engaging with devices, using a keyboard) can be provided in a short period of time; and

• That additionality is created when local, unemployed youth can be screened, trained, and deployed as lab facilitators - providing them with a first work experience and opportunities for future employment.

Now is the right time for Click to engage with a LEAP project because:

• It has the structures to support the collection and storage of data;
• It has the scale, organizational capacity, and will to use the outcomes of a LEAP engagement to shape its operations and strategy;
• It recognizes that the path to further scale and impact is contingent on an increasing evidence base; 
• It's sandbox and the underlying philosophy of iterating and lean experimentation with new products is an ideal space for a LEAP project.

What are 2-3 research questions that you would like your LEAP Project to help you answer? [100 words recommended]

The following are options:

1.  Who benefits from which literacy or numeracy products and are there predictors of success? 

2. Are there transfer effects between the literacy and numeracy products?

3. Can the eQuiz track performance in literacy domains, and measure the extent to which learners develop mastery of each domain during their time as Click learners?

4. When comparing eQuiz performance with fluency benchmarks, do patterns emerge that enable us to map performance on specific eQuiz items with fluency benchmarks?

5. Could the results of the eQuiz be mapped to grade level competencies?

What type of research/studies do you think will help answer your stated questions? Select all that apply.

• Formative research (e.g. usability studies; feasibility studies; case studies; user interviews; implementation studies; pre-post or multi-measure research; correlational studies)
• Summative research (e.g. correlational studies; quasi-experimental studies; randomized control studies)

Please elaborate on your selection above by describing your desired outputs of the 12-week LEAP Project sprint. [200 - 500 words recommended]

Possible outputs - depending on the research question(s) selected would include:

• Analysis of data we have previously collected on products (where applicable) or analysis of data collected for the products we are testing in the sandbox (correlational studies);
• Co-design of practical experiments (correlational, quais-experimental) that can be implemented by us and - ideally but not essential - which produce data within the 12 week timeframe for products we are testing in the sandbox;
  • Analysis of data emerging from those experiments if possible;
  • Recommendations from the analysis;
• If its not possible for the data to be collected in the 12 week timeframe, guidance on the process for collecting and analysing the data;
• If any of questions 3-5 were selected above, what can be said in answer to these questions with regards to data we have and if not, what data we would need to collect;
• A review of our equizzes (numeracy and literacy) and guidance on additions;
• A review of what analysis we have undertaken, what data we are collecting, and advice on a strategy to build out our impact further

How will your organization put these outputs into action? [200-500 words recommended]

The questions posed above form part of Click Learning's impact and learning agenda. Answering (or providing steering on the answers to these questions) will (1) provide directional information about the edtech products we are testing in the sandbox and a datapoint about how we need to either pivot or sustain implementation for impact, (2) Validate the assessment instrument we are using, (3) Build out evidence base of what works, for whom, and when. Our ultimate objective is to use the answers to these questions to have a solid foundation of evidence to be able to advocate with and engage provincial government on the use of edtech products that are proven to work to address South Africa's literacy gaps (South Africa performed the worst in the recently released 2021 PIRLS results with a year of learning lost due to COVID). And separately and more ambitiously, to use the evidence to engage education stakeholders more generally regarding alternative outcomes funding solutions.

Describe your desired short-term and long-term outcomes of the 12-week LEAP Project sprint for both your organization and solution. [200-500 words recommended]

- Better program targeting on literacy and numeracy products being tested, so that learners are able to improve their outcomes based on where they are at and their needs. (Put differently, targeting that addresses variability in learners.)

- More effective program design that considers the transfer effects of numeracy and literacy.

- Validation of or changes to the equiz so that it has a more rigorously validated foundation of evidence for assessment

- Adoption of the digital equiz as a recognised and validated tool of assessment by government (currently only paper-based assessments are used, are costly, and have limitations in scale and timeous feedback.)

- As a result of adoption of equiz, more real-time systemic data on learner progress which in turn shapes government and social sector responses.

- Use of the equiz by teachers as a tool to understand individual learning gaps and respond accordingly

-Adoption of literacy and numeracy products by government and others as a result of the evidence produced.