DISASTER METAVERSE WITH MACHINE VISION AERIAL DRONES

Floods and fires are a new norm which is causing billions of dollars in damages and in the last 5 years has displaced more than 300000 Malaysians. This is especially a stark reality for those from the rural and indigenous communities who often do not have any social net to fall back on to mitigate the impact of these natural disasters on their homes and live hood especially those in agriculture and livestock farming for example in Borneo Malaysia in the 2020 floods over 150,000 people victimized, and over 50,000 people were displaced from their homes.

Poor drainage, deforestation, the country's topography and climate change coupled together have rendered the national agencies who are meant to offer aid and response paralyzed and often busy leveraging supply chains to determine the scale and accessibility options to facilitate response to these communities.

Often the flood or disaster alerts come a little too late and people are trapped in the midst of the disasters with no viable solution, which could be easily tackled with a reliable early warning system. Moreover, the overlapping jurisdictions of the many disaster agencies, often entails the response being rather lackluster and flatfooted. Although they have satellite data, the information is not shared through a common platform, often entailing in contradicting response strategies.

This situation is the same across the world, as sudden freak storms and super hurricanes wreak unfathomable destruction in their wake and strike with such frequency, that they render the antiquated notions of 100 years squalls and 1000 year storm the new reality. It is to cope with this new disaster landscape that a new more proactive, resilient and swift solution is needed, hence our Rescueye. 

RescuEye is a disaster metaverse platform with capabilities to predict, manage and respond to natural disaster victims through leveraging AI machine vision on autonomous drone with sensor fusion disaster digital twins. large language model disaster chatbot which communicates with humans and the devices/drones. Sensor fusion data populates a digital twin on the area monitored. The sensor fusion consists of long range Lorawan sensors that collect water level and other critical parameters to determine a disaster event, which trigger a drone to autonomously fly and survey the effect areas , to serve as an infallible early warning system.

Based live sensor fusion and drone verification real-time simulations are carried out on the digital twin to gauge the impact of heavy rain and trigger alert events when they breach the safe threshold. The chatbot then auto triggers an alert response to all the community members in the areas with instructions and live feedback of safe havens and water levels.The chatbot requests the swarm drones to do an autonomous flyby to detect safe haven, any victims in harm and other defined parameters and populates the digital twin with this live alerts. Victims are given AR route to safe haven.

The system then alerts all key agencies via the chatbot it and manages all their response and satellite data on a common platform. It informs the authorities of access route/obstacles and gives situational awareness to maximize response. All actions are tracked and analysed by the disaster chatbot for live feedback recommendation based on trained rescue missions.

Finally post damage assessment is done by the drone and auto populated by the disaster chatbot to give accurate estimate for disaster relief and post insurance damage payout calculations, streamlining the entire process.   

1.The indigenous and rural communities will be able to get early warning to make risk assessment to purchase disaster insurance or make financial arrangement if the flood risk is higher changing their risk profile and mitigating damages. These communities will also be better prepared to evacuate early and seal their properties with the effective early warning systems.Relief will be auto generated based on life damages insurance assessment.

2.The urban poor will utilize these information to be better prepared for flash floods and seek shelter early. They will avoid getting trapped in the flood in their vehicles or in their residence due to the live feed update. 

3.Disaster responders can coordinate their approach earlier and more efficiently to prioritize red zones and manage the yellow zones. All multi agency efforts can be better managed and get situational awareness to access sites and victims better.

4.Government can assess and create a social property insurance for high risk low lying areas based on active risk indicators  and share the risk costs and manage the reconstruction costs better.

We are based in the same state as the indigenous and rural communities. Currently we are engaging with the rescue services in Malaysia and the National disaster agency(NADMA) to use drone technology to scout and give situational awareness to rescue operations as part of the Malysian Drone2PTK taskforce during the last flood season in 2022.We also helped the scout and location of landslide victims in Batang Kali Landslide in late 2022.Hence we are part of the taskforce and currently are inline to test our rescuEye system after a demo full system test at the rescue services full mock flood demo in Q3 this year.

We have a fully functioning system with the computer vision integrated into the drones(3 units) and the metaverse with the mobile app and chatbot fully functioning. We have tested it in the lab enviroment and now wish to test it at the mock flood demo with all the agencies in 3 months.We are testing the fire fighting capabilities once we have approval to use the fire brigade test site.  

Currently it serves two rural indigenous communities in the Sepang area  Kampung Sungai Buah Dalam dan Kampung Bukit Tunggul which has around 200 villagers.

For technical, financial and cultural help.

1.Technical - To help with the technical complexity in a state level pilot  system in a state level scenario with various agencies.

2.Cultural - To gain acceptance and acknowledgement to empower and engender the indigenous community to use our solutions.

3.Finacial - to build network to venture capital and other funds to helps us at the seed stage capital later. 

We leverage scalable technology with a local flavou as all our hardware and software are opensource but custom fitted to our use case. Through a multi facet approach leveraging drone tech and satellite tech with long range and mmwave sensortech. All this with deep learning can prepare a better early warning system ,which can better equip response and risk management of disaster events before they reach the point of no return. Once these risks can be predetermined with some accuracy, market models to create financial instrument to manage and control them can be introduced with innovative multitiered insurance products to spread the risk factor and aid reconstruction costs.

In the next 12 months:

1.To test and perfect our prototype to be able to manage at a district level disaster which can involve 1000 people.

In the next three years:

1.To Manage a state level event of 100,000 individuals.

In the next five years:

1.to manage a nation level event , with over 1 million individuals. 

1. Number of successful drone missions: Keep track of the number of drone missions carried out for disaster management purposes, such as search and rescue, damage assessment, and delivery of supplies. This can help gauge the effectiveness of your drone operations.

2. Time saved: Drones can help speed up disaster response times by providing real-time information and reducing the need for ground-based personnel. Tracking the amount of time saved through the use of drones can help demonstrate their impact.

3. Lives saved: One of the most important impact goals in disaster management is to save lives. Keep track of the number of people rescued or aided by drones during disaster response operations.

4. Cost savings: Using drones for disaster management can be more cost-effective than traditional methods, such as using helicopters or ground-based personnel. Keep track of the cost savings achieved through the use of drones to demonstrate their economic impact.

5. Community feedback: Gathering feedback from communities affected by disasters can help assess the impact of drone operations. This can include feedback on the effectiveness of drone operations, the quality of information provided, and the overall response to the disaster.

Inputs:

• Resources: Funding, equipment, and personnel are required to develop and operate drone-based disaster management systems.
• Expertise: Skilled personnel are required to operate drones and analyze the data they collect.
• Partnerships: Collaboration with government agencies, non-governmental organizations (NGOs), and communities is necessary to ensure effective disaster response.

Activities:

• Develop and deploy drone-based disaster management systems, including drones equipped with cameras, sensors, and other technologies.
• Train personnel to operate drones and analyze the data they collect.
• Establish partnerships with government agencies, NGOs, and communities to coordinate disaster response efforts.

Outputs:

• Real-time information on disaster impacts, including the extent of damage, location of survivors, and needs of affected communities.
• Improved response times, with drones able to rapidly assess disaster impacts and deliver critical supplies.
• Reduced risks to disaster response personnel, with drones able to access hazardous areas and provide remote sensing capabilities.

Outcomes:

• Enhanced situational awareness and decision-making capabilities, with real-time data informing disaster response efforts.
• Increased efficiency and effectiveness of disaster response efforts, leading to faster recovery and reduced impacts on affected communities.
• Improved resiliency of communities, with early warning systems and more effective response efforts reducing the impact of future disasters.

Impact:

• Lives saved and injuries avoided through faster and more effective disaster response efforts.
• Reduced economic losses and environmental impacts resulting from disasters.
• Improved social outcomes, including stronger community partnerships and increased trust in government and disaster response agencies.

The solution is designed to use computer vision with edge detection to identify pre flood and post flood conditions by the implementation of swarm drones. The system sets several drones which will function as a swarm and collect data over the floodzone  radius at different flight elevation for a complete disaster site digital twin map. All the collected data will populate the disaster digital twin which will be updated in realtime based on the edge and fog computing unit at the disaster site.The drones preflood and post flood data gathering will be triggered based on flood preconditions and after flood alerts from the Government MyRTK system and the LORA sensors in the flood zones.

The drones are triggered by the information received from the LORA sensors on site and the MYrtk system as well as alerts from the BOMBA and other authorities.The Universal traffic management system through the metahuman AI chatbot collates the data and gets immediate permission from authorities for a flight to the flood zone. The information is collated and send in realtime to all stakeholders along with the weather data, flight path and other critical data as well as the aerial units in operation. The swarm drones are deployed and the universal traffic management hub is deployed to the flood zone central point to provide on field realtime support to the swarm drone units.

The search patterns are optimized based on the on site and JUPEM MyRTK data and communicated to all parties and the drone by the ai chatbot too. Real time detection and analysis on the edge devices are done via the onboard thermal and stereo cameras, where victims are located ,geotagged and the access routes are routed in realtime for emergency service access. All these data are populated on the digital twin 3D disaster map and updated in realtime with uniform access by all stakeholders to synergize the efforts. Image recognition to identify  victims through the post processing in cloud can generate their identities and bring up any medical needs which can be met in realtime by the drone via the AI chatbot.If cellular network is disrupted the drones can carry 5G repeaters and loud speakers to communicate with victims.

The metahuman disaster  chatbot autonomously populate more info in the digital twin map  based on flood alerts from social media API and instruct the drones to fly autonomously to the new disaster points in realtime without any disruption due to multi agency delays. The system would also identify elevated rally points and respond  to assist victims of fast rising flood zones to these safe zones, with loudspeakers mapping an obstruction free route. The system can also communicate with a Metahuman AI chatbot to send food to the geotagged victims and coordinate with emergency services

Post flood analysis of damage can also be done via the post processing of the flood through cloud, with the metahuman AI chatbot which automatically populates the necessary information for flood insurance and flood damage assessment with the required evidence gathered autonomously by the drone for a swift and concise damage assessment.

we are a spinoff startup from the center of iot reserach from the Asia pacific university

we have a diverse teams of various nationalities and ethnicities as the we have a diverse pool of international and malaysian students who are in our organization.

currently we are based on selling our services to the agencies involved and the packages as a solution to entity with an interest in safeguarding their assets such as mining and oil and gas.

We are currently planning on securing a service contract to the government and build our disaster metaverse technology to maturity through the increase on the scalability of our solution. Once we achieve that we will be better prepared to package and scale our operations.