MediBot

The specific problem we are addressing is the significant gap in clinical competency and the inability for effective clinical decision-making among the newly expanded healthcare workforce in Primary Health Care (PHC) in Timor-Leste. Despite successful efforts to increase the number of healthcare professionals, which has improved access to PHC, the quality of care remains suboptimal. This issue is primarily due to limited opportunities in workforce development particularly for less experienced health workers, many of whom are physicians practicing in remote municipalities with limited access to clinical oversight, supportive supervision, or continuous professional development opportunities.

A key aspect of this challenge is the isolation of these healthcare workers from the essential clinical guidance needed for quality patient care. Without the presence of experienced professionals or access to evidence-based medical guidelines, these individuals often rely on informal, peer-to-peer advice through platforms such as WhatsApp. This approach results in inconsistent and inaccurate medical advice, further exacerbating the problem of inadequate medical decision support for physicians working in the Timor Leste health system.

The scale of this problem is significant within the communities of Timor-Leste, directly impacting the quality of healthcare delivery across the nation's population. It presents a considerable barrier to improving health outcomes and efficiently utilizing healthcare resources. The reliance on informal networks for clinical decision support not only risks patient safety but also reflects a broader issue of access to and familiarity with current medical guidelines and standards.

Moreover, the problem is compounded by several factors, including language barriers with many essential documents and guidelines available only in English and not in the local language, Tetun. This situation is aggravated by the limited physical and digital access to these resources and a lack of targeted training or 'socialization' of these guidelines among the healthcare workforce. The culture within the medical practice in Timor-Leste, which traditionally has not emphasized evidence-based or guideline-based approaches, further dilutes the potential impact of existing resources meant to guide clinical decision-making.

Globally, this issue resonates beyond Timor-Leste, reflecting a common challenge in healthcare systems experiencing rapid workforce expansion without commensurate increases in training, supervision, and resources for professional development. In Timor-Leste, this challenge directly affects the healthcare delivery to its population, estimated at around 1.3 million, underscoring the urgent need for interventions that can bridge the gap between healthcare worker competencies and the demands of quality patient care.

In summary, the problem we aim to address is the critical gap in clinical competency and decision-making support among the newly expanded healthcare workforce in Timor-Leste, compounded by barriers to accessing and utilizing existing healthcare guidelines and resources, significantly impacting the quality of PHC delivery and patient care outcomes.

MediBot is a clinical decision support AI chatbot that enables primary care providers (doctors and nurses) in Timor Leste to query in their native language Tetun for medical decision support. To facilitate easy access, the bot is integrated with Telegram and Whatsapp - communication channels widely used by physicians in Timor Leste. We have experimented with a variety of Large Language Models (OpenAI, Claude, Gemini), and have trained the bot on local clinical care documents, national guidelines, and standards approved by Timor-Leste’s Ministry of Health and the World Health Organisation. This enables the provision of medical decision support that is localised and well-contextualised for Timor-Leste’s primary care practice. 

The bot is currently embedded in community chat groups via Telegram and Whatsapp - a format that local doctors are familiar with and can easily access. At the technical level, we enable the bot to perform the following functions to provide accurate, relevant, and accessible medical decision support through:

1. Peer review and community moderation of AI-generated responses - this provides a level of verification and review by more senior physicians to correct potential factual inaccuracies, in order to encourage safe use of AI-generated responses. This is done through a blend of community moderation and a structured process for experienced clinicians to review and refine responses. 

2. Maximizing context-relevant responses - our solution tags different kinds of questions into categories that enable branching of bot responses. For example, if a query is detected as a query intended to aid with diagnosis, the bot returns questions instead of immediate answers, so as to narrow responses to the most contextually relevant and actionable parts.

3. Self-critique and other techniques for prevention of hallucinations - hallucinations being a known problem of AI-bots, we have been working with experts to apply techniques (e.g. self-critique and role assignment) to reduce the risk and incidences of hallucination. 

4. Cost optimization to enable scalability and accessibility - for scalability beyond the pilot group of doctors, we will be optimizing the cost-efficiency of the bot through caching repeated or frequent queries and training doctors in effective prompting.

Our solution is designed to serve two key groups within the healthcare ecosystem of Timor-Leste: direct beneficiaries, who are government primary healthcare (PHC) providers (estimated total 10,000 persons), and indirect beneficiaries, who are patients accessing government PHC services (estimated total 1.3M persons).

Direct Beneficiaries: Government Primary Healthcare Providers

The direct beneficiaries of our solution are the government PHC providers, including physicians, nurses, and other healthcare workers operating in the public health sector across Timor-Leste. This group primarily consists of newly recruited healthcare professionals deployed across various municipalities to deliver primary healthcare services. Gaps in medical knowledge and training have resulted in the need for these healthcare professionals to rely on medical advice from other doctors on Whatsapp chat groups. This practice, while effective in improving clinical support and confidence of healthcare professionals who are administering interventions at point of care, can be fraught with inconsistencies and inaccuracies. 

Our solution will provide immediate, reliable, and easily accessible guidance based on Ministry of Health-approved documents and guidelines, thereby enhancing the clinical decision-making capabilities of PHC providers. This will, in turn, increase their confidence in providing care, reduce their reliance on informal and often unreliable sources of medical advice, and foster a culture of evidence-based medical practice.

Indirect Beneficiaries: Patients of Government PHC Services

Indirectly, our solution serves the 1.3 million patients who rely on government PHC services across Timor-Leste. Patients often experience the repercussions of insufficient clinical support among healthcare workers through misdiagnoses, unnecessary referrals, delayed treatment, and overall poor health outcomes.

By empowering PHC providers with the tools and resources to improve their clinical decision-making, our solution directly impacts patient care. Improved competence and confidence among healthcare workers are expected to lead to more accurate diagnoses, appropriate and timely treatment, and a reduction in unnecessary hospital referrals. This shift will enhance the overall quality of PHC services, leading to better health outcomes for patients and a more efficient use of healthcare resources.

The EquiTech Collective’s mission is to advance equity through technology in Southeast Asia, by building technology with catalytic impact in the fields of healthcare and education. We focus on solutions that have high impact and have the potential to deliver systemic and sustainable change. Our work is propelled by strong partnerships with organizations on the ground, and local capacity efforts.  We are deploying various technologies, such as electronic health records, remote monitoring systems for health, and AI bots in lower-resourced settings across Southeast Asia. Our operations currently span across Thailand, Cambodia, the Philippines and Timor Leste, and our approach has been to build hyper-local teams that live within the very communities we serve. This allows us to combine global expertise with local ingenuity - our engineers come not just from our headquarter in Singapore but also from the very communities we deploy technology and systems in. In other words, the solutions we incorporate are not only guided by communities’ input, ideas, and agenda, they are built and co-created with them through integral teams. 

For MediBot, we are collaborating with Associação Maluk Timor (MT), a local NGO registered in Timor Leste. MT supports Timor Leste’s Ministry of Health by strengthening primary health care. MT has  a presence in all 14 municipalities of Timor Leste and has a track record of strengthening the local health system through workforce development, service delivery and design, and community health. Our modus operandi in this collaboration with MT has been to deploy engineers to work in integrated teams with ground operational staff, both remotely and eventually on-site so that we can gather timely user feedback for both the technology development for Medibot, and to refine the ongoing pilot with local healthcare professionals. Finally, MT has also been proactive about advocating for the use of this Medibot to Timor-Leste’s Ministry of Health leadership, and the reception has thus far been positive. Post-pilot, we plan to work jointly to scale this beyond the initial pilot group, to possibly deploy this tool at the national level.

The prototype of MediBot has been created on Telegram using large language models. This generative AI bot has been trained on locally approved guidelines and has been piloted by the Equitech Collective’s technical team and MT’s main directors for basic testing and hallucination identification. 

We are in the midst of completing our first round of user testing among 15 healthcare providers in Timor Leste. This phase allowed providers to query MediBot about real-life use cases and questions that are commonly encountered in clinical practice. They documented any inaccuracies and suggested edits to MediBot’s answers and outputs. Simultaneously, the Equitech Collective’s technical team tested the bot for edge cases, hallucinations, and adherence to clinical guidelines. All of these input-output pairs will help to further train the AI algorithm. Many subsequent rounds of user testing and iterations are planned; however, initial results indicate that the prototype is working well and that the approach is feasible, effective, and cost-effective.

The technical team is currently exploring the concept of self-criticism in the bot (a technique that helps LLMs improve their logic by evaluation and refining their own outputs). In addition, we are developing a tagging system so the bot can categorize questions (into for example, diagnosis and treatment questions) so that it can deliver more targeted answers. We are also building  a novel model of community moderation whereby qualified doctors will be able to peer-review the bot’s answers and provide feedback for the bot to be retrained on in future mass rollouts of user testing. Our technical team is currently training the bot to ask follow-up clinical questions to doctors if required in order to gather all the relevant information and provide more nuanced clinical answers. 

The Equitech Collective was established in 2022. We are a not-for-profit enterprise and most of our projects are funded through philanthropic foundations and grants. As a young enterprise, we can benefit significantly from program funding that could help to implement and scale pilots, as well as mentoring and capacity building support. The current Medibot project is entirely funded from internal funding as part of internal experimentation, and will require funding at the pilot and rollout phase which is where we are presently at. We would also benefit from MIT SOLVE’s advice on other organizations, health systems, and jurisdictions that could benefit from such a solution. As such, MIT SOLVE’s ability to provide advisory support in areas such as technology, market entry, monitoring and evaluation of impact, could have catalytic impact for Equitech’s capacity to serve more communities across Southeast Asia. In addition, networking with companies and talents in AI and other entrepreneurs in similar domains or on similar trajectories would be beneficial.

Use cases of AI are rarely implemented in low and middle income country settings. MediBot is the first AI-driven clinical decision support tool in Timor Leste, thereby providing access to and bringing the latest technologies to resource-poor settings.

MediBot is a hyper local AI bot. MediBot is trained on nationally approved clinical guidelines, ensuring that its recommendations are highly relevant and specific to the local context of disease patterns, available treatments, and healthcare infrastructure. This ensures the delivery of targeted care that is context sensitive.

Additionally, our solution's integration with widely used platforms like WhatsApp and Telegram leverages existing digital habits, significantly reducing new user adoption barriers and enhancing accessibility. This strategic use ensures that every connected healthcare provider in Timor Leste can access sophisticated clinical decision support effortlessly, right from their phones.

To address AI reliability, we are implementing robust mechanisms including self-critique and peer review processes that continuously validate the AI’s recommendations against medical standards. This not only builds trust and credibility but also promotes the tool’s safe integration into daily medical practice.

By demonstrating the effectiveness of context-sensitive AI tools in improving healthcare delivery, our project aims to catalyze broader technological adoption in the region, potentially transforming the primary healthcare landscape in Timor Leste and setting a precedent for other low-and-middle-income countries.

Our Theory of Change is described below:

Activities:

• Bot Development and Training: Build and train MediBot with local clinical guidelines and diverse comprehensive clinical data. Develop safeguarding mechanisms, such as self-criticism and peer-review, to reduce errors and hallucinations.

• Integration: Seamlessly integrate the chatbot with existing health IT systems and communication platforms (e.g., WhatsApp).

• Training for PHC Workforce: Conduct workshops and ongoing training sessions for the PHC workforce on how to use the chatbot effectively.

• Monitoring and Evaluation: Continuously monitor MediBot’s performance and impact, adjusting protocols and content as needed.

Outputs:

• AI Chatbot Deployed: MediBot is functional and accessible to PHC workforce.

• Training Sessions Completed: PHC doctors and nurses are trained and able to use MediBot efficiently.

• Regular Updates and Improvements: MediBot is routinely updated based on feedback and evolving medical standards.

Immediate Outcomes:

• Increased Access to Clinical Guidelines: Doctors have at-hand local guidelines via MediBot, thereby ensuring clinical practice evidence-based. 

Intermediate Outcomes:

• Enhanced Clinical Decision Making

  • Improved Diagnostic Accuracy: Enhanced precision in diagnosing patient conditions.

  • Enhanced Treatment Efficacy: More effective and tailored treatment plans.

• Reduced Time to Treatment: Faster diagnosis and initiation of treatment.

• Increased Provider Confidence and Capability: Doctors feel more supported and knowledgeable.

Long-term Outcomes:

• Improved Patient Health Outcomes: Better patient management leads to improved health statuses.

• Elevated Healthcare Quality and Satisfaction: Overall improvements in service delivery and patient satisfaction.

• Decreased Healthcare Costs: Efficiency reduces unnecessary testing and misdiagnosis costs.

Long Term Goal:

The ultimate goal of MediBot is to improve healthcare outcomes in Timor Leste by supporting primary care doctors with accurate, timely, and locally relevant clinical decision support.

1. Impact goal: Increased access to digital health tools
   
   1. Metrics: Adoption and scale (adoption rate by PHC workforce, number of active users, geographical coverage)
2. Impact goal: Improved PHC clinical decision making capability and accuracy
   1. Metrics: Clinical decision concordance (compare clinical decisions made by PHC equipped with and without MediBot to trained doctors’ decisions
3. Impact goal: Reduced time to treatment
   1. Metrics: compare average time from patient presentation to initiation of appropriate treatment before and after MediBot implementation
4. Impact goal: Enhanced patient and provider satisfaction
   1. Metrics: Surveys and qualitative interviews with PHC workforce and patients before and after MediBot implementation
5. Impact goal: Improved quality and effectiveness of PHC 
   1. Metrics: Preventive Care: Uptake rates of preventive measures such as immunizations, screenings (e.g., mammograms, colonoscopies), and lifestyle counseling.
   2. Metrics: Patient Health Outcomes: Monitor improvements in patients’ clinical conditions, management of communicable and chronic diseases, and recovery rates.
6. Impact goal: Development of novel system-level public health insights
   1. Elaboration: A secondary use-case of MediBot is its application as a system-wide data collection tool. For example, the proportion of clinical questions about specific patient profiles and disease conditions queried by doctors in MediBot could inform disease control and outbreak policies, especially when married with geospatial data. Another example is the inability of MediBot to answer specific clinical questions could suggest that local guidelines need to be updated to represent modern disease patterns and patient presentations. 
   2. Metrics: Qualitative interviews and focus group discussions with policymakers from the Timor Leste Ministry of Health

Our core technology is generative artificial intelligence. We utilize large language models for assisting with clinical decision making. While we will initially rely on general purpose, out-of-the-box foundational models such as OpenAI’s GPT, we plan to build AI agents, empowered with other tools as we see fit, to solve the problem more comprehensively. For example, we might give our AI agent access to a medical history database or a web-browser. We will also consider fine tuning our own custom models or using smaller, specialized large language models to optimize for cost and create more sustainable solutions.

1 full-time staff, 3 part-time staff

4 months, starting January 2024

Our team is made up of passionate people from diverse backgrounds - across ethnicities, geographies, cultures, and genders. We implement regular bonding activities to promote camaraderie. We've also created opportunities for people from marginalised groups and the communities we aim to serve to become part of our team.

EquiTech Collective is a non-profit organisation incorporated in Singapore. Our revenue sources include grants and project fees that we charge clients such as World Health Organisation in strategic partnerships for health systems transformation. For WHO, we serve as innovation capacity building partner to develop innovation sandboxes for health systems transformation. This involves extensive understanding of internal innovation mandate and priorities within WHO (Western Pacific Region), the work of country focal points, as well as government agencies. Our role is to provide technology teams for the whole innovation cycle, from discovery to implementation and evaluation, as well as digital health products designed to advance equitable access to healthcare. 

In addition, we work with government ministries and agencies to address barriers to innovation, such as the lack of technical capability and capacity to continuously improve existing operations and technology to suit evolving needs of health. We provide more cost-sustainable technology tools and systems that serve as alternatives to commercial enterprise solutions that may be prohibitively expensive. In addition, to ensure that solutions are well embedded in the places they are deployed, we support government agencies and non-profit organisations in change management by shepherding them through organisational changes, technical retooling, retraining of staff, and strategic alignment across policy, technology, and operations.

In so doing, health systems - both in public and people sectors - can accelerate their transformation journey through more sustainable digitalisation efforts.

Equitech Collective has been cashflow positive and profitable, with revenue streams mainly from contracts by international NGOs, government contracts, and non-profit contracts. While we do rely on grants, we see these grants as catalytic in nature rather than a long term revenue model. In general grants go to support initial pilots that would not have been possible without initial catalytic funding, but our general rule of thumb has been to ensure that longer term costs post grant period constitute no more than 15% of initial cost, within a ballpark that would be affordable to partner organisations. 

Our early grantor has included China Medical Board (a US-based foundation), and clients include World Health Organisation (Western Pacific Region), SaveTheChildren International (Thailand), Mae Tao Clinic (Thailand), Maluk Timor (Timor-Leste), AHA! Behavioural Design (Manila).