Humans in the Loop
One-line solution summary:
Providing refugees with work and skills to drive forward the Artificial Intelligence Industry.
Pitch your solution.
65 million people worldwide have been displaced by conflict or violence, giving up their homes, education and work (Source:UNHCR). Due to language and administrative barriers, many are unable to access education or work in their new homes.
Humans in the Loop provides conflict-affected people with training and employment opportunities in the Artificial Intelligence Industry. We are building the workforce of the future which annotates datasets and corrects model predictions, helping AI learn to think like a human. This provides much-needed income as well as new skills and experience as professional "humans in the loop".
The company was founded in Bulgaria in 2017. Since then we have provided employment opportunities to more than 300 refugees in Bulgaria, Turkey, Syria and Iraq via local partners. There is huge potential to scale up our work by 1) deepening the involvement of humans in the AI pipeline and 2) working in more countries.
What specific problem are you solving?
We refer to the target community that we are working with as "conflict-affected" people, which includes not only refugees and asylum-seekers but also internally displaced people and those currently living in places of armed conflict. Our target region are the Balkans and the Middle East, with 3.5 million refugees living in Turkey alone and 6 million people who are internally displaced in Syria and Iraq (source: UN OCHA).
Armed conflict has caused a rise in unemployment levels in Syria and Iraq (currently at 8%) while refugees and asylum-seekers in Turkey and Balkan countries are facing severe administrative restrictions (only 50,000 people out of 2 million working age ones have a work permit in Turkey). Youth unemployment is even higher, reaching 20%, while female unemployment has been an enduring challenge in the whole region.
Due to work permit issues or lack of opportunities, many conflict-affected people end up working in the informal sector and are vulnerable to exploitation. Unfortunately, they have also been among the most hit by the COVID-19 economic downturn, with 200,000 people reverting to using cash support (UNHCR). This reliance on humanitarian assistance, even if extremely necessary in the short-term, can be harmful in the long term.
What is your solution?
Our solution is to provide conflict-affected people with training and work opportunities that are entirely digital and can be accessed remotely. The work that we offer is extremely easy but it taps into one of the most innovative industries: Artificial Intelligence.
We work as a B2B outsourcing company which connects directly with computer vision startups and researchers and supplies human input in order to help them improve their models. That includes activities such as dataset collection and annotation, edge case handling, and output verification.
Essentially, we train our workers to understand how AI works and to teach it how to think like a human by providing examples and correcting its mistakes. Our professional "humans in the loop" are trained to use a variety of annotation tools and platforms and to understand machine learning bias and how to avoid it.
The work that we offer does not require English capabilities and can be done from home so it's very appropriate for women and is resilient to situations like the COVID-19 lockdown during which our workers had uninterrupted access to work. In addition, it helps workers acquire essential computer and workplace skills that increase their professional opportunities in the future.
Who does your solution serve, and in what ways will the solution impact their lives?
We primarily work with conflict-affected people living in Bulgaria (104 refugees and asylum-seekers), Turkey (39 refugees, including people with disabilities), Syria (80 internally displaced people and locals) and Iraq (95 internally displaced people and locals). The overall ratio of women in our workforce is 52% and the average age is 26, even though we have annotators who are 50+ as well and we do not focus on a particular age group.
Thanks to our work with local NGOs, primarily the Roia Foundation and WorkWell, we are able to adjust our offering to the local context in each country, addressing issues such as internet connectivity and access to technology, payment processing, and training needs. We conduct monthly measurements of hours of work, wages paid, and trainings received across our entire workforce to make sure we are hitting our impact targets. In addition, most of our supervisors come from our target group and ensure a feedback loop.
In order to better understand the needs of our community, we conduct a yearly impact assessment which includes qualitative research through interviews and quantitative KPI evaluation. This year, the research is conducted in partnership with the Refugee Digital Livelihoods project at the University of Edinburgh.
Which dimension of the Challenge does your solution most closely address?Equip workers with technological and digital literacy as well as the durable skills needed to stay apace with the changing job market
Explain how the problem, your solution, and your solution’s target population relate to the Challenge and your selected dimension.
Humans in the Loop and its partners provide training for hundreds of conflict-affected people in basic digital skills as well as more specific technical skills that are required for the projects that we work on. This enables them to work for us on a variety of projects and gives them financial security in the short term. In the long term, we also equip them with digital expertise and other transferable skills and knowledge that they will need to secure other freelance or contract work and give them the best opportunity to access the job market in the future.
In what city, town, or region is your solution team headquartered?Sofia, България
What is your solution’s stage of development?Growth: An organization with an established product, service, or business model rolled out in one or, ideally, several communities, which is poised for further growth
Who is the primary delegate for your solution?
Iva Gumnishka, CEO
If you have additional video content that explains your solution, provide a YouTube or Vimeo link here:
Which of the following categories best describes your solution?A new business model or process
Describe what makes your solution innovative.
We are building a completely new job category: a professional "human in the loop" who works to improve AI systems and teach them what it means to be human. Currently, Artificial Intelligence requires a lot of manual input in the form of annotations, such as bounding boxes or image tags, in order to recognize objects in an image. However, as AI advances, such tasks will be automated and humans will be required to teach even more advanced skills to AI, such as decision-making, reasoning, and emotions.
Using the theory of “leapfrogging” that communities can make a quick jump in economic development by harnessing technological innovation, we aim to train those who are in the biggest need of job opportunities and make them an active part of the latest technological advancements. For example, together with our partners at Roia we are opening Syria's first Annotation Hub in Aleppo in July 2020.
There are many initiatives in the region doing amazing work in the livelihoods field which are focusing on other types of remote work such as Natakallam and Chatterbox (language teaching). There are also two similar refugee annotation platforms and apps like Taqaddam and WorkAround. What distinguishes us is that we couple training and employment through a unique partnership model with NGOs on the ground. In addition, while most similar livelihoods initiatives focus on refugees only, we also provide employment opportunities in the countries of origin and are currently the only organisation providing such work opportunities in Syria.
Describe the core technology that powers your solution.
Our offering to clients is setting up an Active Learning "human-in-the-loop" pipeline in which humans can participate in the training of an AI system at various stages: by collecting the initial training data and annotating it, by receiving edge cases that the system is unsure about and labeling them, and by correcting mistakes and helping to backpropagate the errors by identifying what biases in the training dataset led to wrong interpretations.
Our main value hides in our dedicated and trained workforce and in our project management, since we have not yet found a "one size fits all" solution that would work for all of our clients in terms of a technology. We have decided against building our own software, at least for the time being, since our flexibility is one of our key selling points for clients. Instead, we partner with leading annotation platforms, such as Supervise.ly, Hasty, Diffgram, Trainingdata.io, and others. Some of our clients also have their own in-house platforms, in which case we train our workers to use their tools.
The fact that we are "tool agnostic" means that we also train our staff to use a variety of tools and platforms already on the market, which makes them more flexible, resilient and employable. However, when existing tools don't suit our purposes, we do have the readiness to develop new tools, and to date have developed our own proprietary app for image collection, as well as a platform for video rating.
Provide evidence that this technology works.
Image and data annotation is a growing market, estimated to reach 1 Billion USD by 2023 (Cognylitica) with more than 30 tools available for different specialties and uses, including video annotation, 3D annotation, Radar and LiDar annotation, text annotation, etc. The Deep Learning boom itself in the 2000s was powered not only by the increases in computational power but also by the availability of large-scale crowdsourced datasets through Amazon's Mechanical Turk.
Today, many companies are turning against crowdsourcing due to the need of highly accurate and consistent interpretation of the data that only trained dedicated teams can ensure. Human input like collection and manual annotation is at the core of fields like self-driving cars, drones and satellites, retail industry, face recognition and medical imagery. Our annotators are trained to provide datasets for all of these use cases, by collecting data from different sources, classifying it, annotating it and creating segmentation masks.
According to our observations, a large part of the market is still not mature enough for the human-in-the-loop pipeline that we are offering, since most companies are still focused on building gold-truth datasets and halting the model training process once it achieves a certain accuracy. However, more and more clients are now reaching out with requests for human verification of pre-annotated datasets and are interested in applying Active Learning to accelerate model training. And by all means more and more companies will be turning to such processes in the future, especially when models are dealing with real-life data.
Please select the technologies currently used in your solution:
What is your theory of change?
To help conflict-affected communities be self-sufficient through digital employment and training in the field of "human-in-the-loop" jobs for AI
Over the next 5 years, at least 3000 conflict-affected persons (refugees, asylum-seekers, IDPs, locals, 50% of whom women) across the Balkans and the Middle East:
1. are working on digital "human-in-the-loop" jobs and earning a living wage for part-time work
2. achieve proficiency in transferable "human-in-the-loop" skills, including basic English, basic computer literacy and data labeling skills, different annotation platforms, and knowledge of the AI industry
At least 3000 conflict-affected persons (at least 50% women) across the Balkans and the Middle East:
1. are working 40 hours per month on average on "human-in-the-loop" projects for continuous AI improvement through human input
2. have received 40 hours per month on average in trainings in basic English, basic computer literacy and data labeling skills, different annotation platforms, and knowledge of the AI industry
1.1. Create partnerships and work actively with NGO partners in 8 countries across the Balkans and the Middle East (Bulgaria, Turkey, Iraq, Syria, Palestine, Greece, Jordan, Lebanon)
1.2. Onboard 3000 workers who qualify as conflict-affected (either refugees, asylum-seekers, internally displaced or living in conflict zones) over the next 5 years
1.3. Implement specific measures for gender inclusion in order to ensure at least 50% women representation in all recruitment and training activities
1.4. Secure a minimum of $1,000,000 per year from clients through business development activities and partnerships with research institutions
1.5. Provide at least 120,000 hours of work per year to conflict-affected people on a variety of platforms and tools for AI model training and verification
2.1. Develop and deliver 20 hours/week of trainings for basic English and general computer skills for the entire workforce in offline and online modules through NGO partners
2.2. Deliver 20 hours/week of trainings for specific AI annotation projects, including data labeling skills, working with annotation platforms, and understanding project guidelines
2.3. Deliver a series of trainings to ensure a deep understanding of the AI industry among the workforce, including AI and computer vision basics, understanding and recognizing bias, and the "human-in-the-loop" model
Select the key characteristics of your target population.
Which of the UN Sustainable Development Goals does your solution address?
In which countries do you currently operate?
In which countries will you be operating within the next year?
How many people does your solution currently serve? How many will it serve in one year? In five years?
In our lifetime, since the beginning of our operations in March 2018, we have served 321 conflict-affected people across 4 countries in total. In 2020 only (up until end of May), we have provided 154 people with training and work.
In 2021, our goal is to provide training and employment opportunities for 400 people and to expand our partnerships to Greece and Palestine, where we are already in touch with potential partners.
In 2025, we will work with 1000 people in eight countries, scaling to Jordan and Lebanon.
Our aim is to provide a minimum of 3000 unique beneficiaries with work throughout this five year period.
What are your goals within the next year and within the next five years?
We have had partnership inquiries around the world but we are keen on balancing expansion with maximizing our impact in the countries that we already work in, recognizing our role in contributing to the the post-conflict reconstruction of Syria and Iraq.
In 2021, our plan is to:
- Provide training and employment opportunities for 400 people (at least 50% of them women), impacting up to 1200 indirect beneficiaries such as dependents and family members
- Work with partners in 5 countries (Turkey, Syria, Iraq, Palestine, Greece)
Within the next five years, we will:
- Provide employment and training for 3000 people (at least 50% of them women), impacting up to 9000 indirect beneficiaries such as dependents and family members
- Work with partners in 7 countries (based on comprehensive needs assessment and partner selection processes)
In terms of our business model, we are planning to continue structuring our operations around remote work, which allows a great degree of flexibility to workers. However, we are exploring the idea of setting up Annotation hubs, such as our HQ in Bulgaria and the hub opening soon in Syria, where workers are able to attend in-person trainings and socialize, as well as access computers and reliable internet.
In terms of technology, we are extremely interested in exploring more ways in which humans can be plugged into the AI training pipeline and we are not ruling out the option to develop our own software if we see that it would bring a considerable business advantage.
What barriers currently exist for you to accomplish your goals in the next year and in the next five years?
1. We face a variety of challenges to our work on the ground which are influenced by the local economic situation and infrastructure, such as:
- Payment processing and legal constraints to employment, in addition to freelance work not being recognized officially
- Access to internet, computer equipment and reliable electricity
- Language barriers and low levels of skills, education and digital awareness among our target group
2. Given these challenges, we require very reliable partners on the ground which is a very hard thing to ensure. We have reviewed more than 10 NGOs in our target region but have so far only established a partnership with two, because we have very high standards and very specific requirements for the partners to fulfill.
3. Business Development is also a challenge because the industry that we are working in is very dynamic and growing very fast, and it is important for us to keep pace. We are competing against massive crowdsourcing platforms like Amazon's Mechanical Turk or Crowdflower, and also against large delivery centers who come from a traditional BPO background and are now shifting to AI data annotation.
4. In addition, companies are looking into ways to automate the work that our annotators do in order to reduce their costs for manual pre-processing, so we are developing ways in which our annotators can still be kept in the loop of developing technology and we can make sure their work is relevant. This will also create more long-term opportunities for them.
How do you plan to overcome these barriers?
1. We believe the most effective way to address challenges on the ground is to work in partnership with local NGOs who know their context and people well. This is why the NGO partnership model is part and parcel of our work and central to our vision going forward.
2. We are continuously looking to identify new appropriate partners, and we believe that if we are successful in the SOLVE program, this will give us exposure and partnership opportunities to reach a number of other suitable NGOs in our region.
3. In terms of business development, we are planning to approach more and more the academic sector, where we have seen a much more open reception of our social impact model and where we think we are well positioned to replace Mechanical Turk as the source of reference for data annotation. Through Solve, we are hoping to extend our network in the academic community and to become involved in exciting new research projects.
4. Our focus is currently on developing ways in which our annotators can be involved in every stage of the AI training process, from collecting data and annotating it, to ongoing review and correction of machine-produced results.
This is a promising development not only because it will keep our workers relevant to our client's business by providing continuous improvement of their models through human supervision, but it will also ensure more long-term employment for our workers and more opportunities for us as an organisation.
What type of organization is your solution team?For-profit, including B-Corp or similar models
If you selected Other, please explain here.
How many people work on your solution team?
We have four full time staff and two part time staff.
Full time staff include:
- Chief Operations Officer
- Chief Impact Officer
- Project Manager
Part time staff include:
- Sales Development Representative (0.5 FTE)
- Business Development Associate (0.25 FTE)
How many years have you worked on your solution?
Why are you and your team well-positioned to deliver this solution?
Our CEO and Founder Iva Gumnishka founded Humans in the Loop in 2017 in order to support Middle Eastern refugees and asylum-seekers who had newly arrived in Bulgaria and who were affected by serious discrimination. In addition to being the founder of HITL, Iva is also the director of the Impact Tech Foundation which develops projects to connect technologies with social good. Iva has a degree in Human rights from Columbia University and was named Forbes 30 under 30 for Bulgaria in 2018.
Our Chief Operations Officer Tess Valbuena has more than 10 years of experience in Operations in the Business Process Outsourcing sector in Australia and the Philippines. As a COO of Humans in the Loop, Tess has successfully managed and delivered more than 50 projects for business and academic clients.
Our Chief Impact Officer Zoe Holliday is no stranger to ambitious programmes to support refugees, having previously set up the first ever full time educational activities across four refugee centres and two detention centres in Bulgaria, as part of her role with Caritas Sofia. Previously, she was the Coordinator of the Refugee Survival Trust in Scotland. Zoe is responsible for ensuring that we measure and maximise our impact at HITL.
Our Project Manager Raghda Al Salman is originally from Gaza, Palestine, and has a background in education and social work. Raghda performs supervision, project management, trainings and community management, in addition to translating all of the training materials into Arabic and delivering trainings in Arabic where necessary.
What organizations do you currently partner with, if any? How are you working with them?
We currently partner with two organisations for project delivery and work on the ground in the Middle East. These partners are responsible for recruitment, training and supervision of annotators; project management; and distributing payments.
- Roia, based in Istanbul and with operations in two cities in Turkey (Gaziantep and Istanbul) and three in the Syrian Arabic Republic (Aleppo, Raqqa, Idlib). They focus on ICT trainings and together we are opening the first ever annotation hub in Syria, based in Aleppo.
- Workwell, a programme of the Premptive Love Coalition in Iraq, with operations in three cities across Iraqi Kurdistan (Erbil, Suleimaniyah, Duhok) as well as Mosul. They also focus on ICT and microwork trainings.
In addition, we have established partnerships with several of the companies offering annotation tools, such as:
- Darwin V7
Since the services that we offer are complementary to theirs (we offer workforce, they offer software), these partnerships are a win-win situation for both parties. Through these partnerships, we receive free access to their annotation tools as well as introductions to prospects. We regularly publish reviews of annotation tools and we actively engage with the annotation landscape so as to keep up with the latest trends in the AI training data space. With many of these partners we have set up custom infrastructures for clients where our workforce has been incorporated in a pipeline for training data annotation and verification.
What is your business model?
We work primarily as an outsourcing company, providing B2B services. Our clients are start-ups, medium sized companies and corporations, working in the artificial intelligence and computer vision fields. The majority of our clients are based in Western Europe and the United States, but also have clients in Australia, Singapore and Dubai.
We provide a complete suite of services for the machine learning model development process, including: collecting the ground truth data set; annotating data; verification of machine outputs; and edge case handling.
Many of our projects are one-off, because they depend on our partners' machine learning development cycle, but many of our clients have recurring needs as they improve and hone their models, and they need more batches of work to be prepared and labelled by us.
Our clients appreciate the fact that we work with dedicated teams who are trained specifically for their projects and are available on demand. Our teams are also scalable and can go from 2-3 people up to 40-50 depending on the needs of the client. We are also very flexible in terms of tooling and can use the client's in-house tools if they prefer.
When we start the project, we provide a free trial during which we evaluate the pricing and difficulty, after which we reach out to our partners, identify the most appropriate team and a dedicated project manager, and conduct trainings. We deliver projects in several iterations to allow clients to give ongoing feedback on our work.
Do you primarily provide products or services directly to individuals, or to other organizations?Organizations (B2B)
What is your path to financial sustainability?
Our goal is to be completely self-sustaining, and to rely on earned revenue through services. We have a very clear revenue stream by charging clients for every unit of service delivered (e.g. pricing per image, label, or hour of work). This is then broken down in the following way:
- 50% goes directly to the worker
- 5-10% goes to the partner NGO to cover project management and supervision.
- 40-45% remains within HITL to finance overheads and business development opportunities.
Because we hire annotators on a freelance basis, we have the capacity to take on a variety of projects at short notice, and our aim is to focus on business development to be able to offer work to as many people as possible. In addition, by working with several partners and our in-house team in Bulgaria, we are able to manage multiple projects in parallel without an excessive operations burden.
In the short-to medium- term we hope to support our business development work by securing grant funding and prize money to cover the time and costs of our core staff to develop new partnerships and scope new possibilities for scaling our activities. However, even without external funding our organization is already self-sustaining and able to cover all of its operations and sales costs.
Why are you applying to Solve?
Being selected as a Solve solution would give Humans in the Loop an excellent platform on which to build awareness of our activities amongst potential partners, clients and collaborators. In particular:
- We hope that we will find potential partner organisations in other countries who are interested to work with us to roll out similar models amongst different conflict-affected communities. These partners may be other Solvers or may simply hear about us due to publicity generated by our success.
- We believe that becoming a Solver would help us to reach more clients, as well as to give them confidence in us as an organisation. By securing more clients we will be able to offer employment opportunities to more people, helping us to scale up our activities and impact.
- Being a relatively new organisation, we would also hugely value the opportunity for mentorship, expert advice and guidance from Solve and MIT networks, which will help us to plan the future direction and development of Humans in the Loop.
- Any prize money would be directed into our partnership, business development and sales activities, with a particular focus on developing new partnerships in different geographical regions, and moving into new areas of growth
In which of the following areas do you most need partners or support?
Please explain in more detail here.
We are looking for mentorship on our business model, in order to analyze a potential shift to becoming a product company, since both grantmakers and investors have suggested that this might be a better way to scalability in both economic and social impact aspect. We would also love to discuss ways in which we can maximize the relevance of our work to companies working in the computer vision domain with subject matter experts.
We currently do not have a board of directors or an advisory board and we would love to set up one with the help of the SOLVE network and community. We would love to establish one with a balance of experiences in scaling social impact solutions and launching and solidifying strong companies.
Finally, we would really appreciate the opportunity for marketing, media and exposure that SOLVE provides in order to spread the word about our work.
What organizations would you like to partner with, and how would you like to partner with them?
We have identified a number of Solve applicants who might be potential partners as we look to expand our work into different countries, such as Dignify (Columbia), Moringa School (Kenya), D2 (Bangladesh), Skills and Entrepreneurship (Nigeria), Butterframe (Uganda), Establishment of Digital Skill Centre (Tanzania) and Douar Tech (Morocco). We would be interested to initiate conversations to see whether our activities and goals align.
In terms of MIT Faculty and Initiatives, we are really interested in collaborating with the MIT Media Lab on exploring human input in AI, human-AI interaction, diversity in computer vision, and avoiding harmful biases. We would love to collaborate with computer vision and AI researchers across the MIT faculty, especially at the Computer Science & Artificial Intelligence Lab and the Center for Brains, Minds and Machines.
We have been working on a project for cardiac CT and Xray annotation with the University of Leeds (UK) and we are negotiating a project for prostate lesions detection with UCL. We have a dedicated team of Syrian doctors who are refugees in Turkey and we are excited to explore more ways in which we can use their capabilities to advance medical research. That is why we are especially interested in connecting with the Medical Vision Group at MIT.