## Inspiration

After the recent hurricanes in Florida, one of our team members who is from Florida had damage to their house. Normally, the process for filing an insurance claim after disaster is extremely cumbersome, requiring proof of all items and their value that were in the house before the disaster. This is very hard for disaster victims to prove in such a tough situation, which dissuades many people from filing insurance claims that cover the full value of all the items they lost.

## What it does

Our app allows homeowners to record a video of their house, while our innovative AI automatically catalogues all items in your house and their approximate value. After a disaster, you an rescan your house and we automatically suggest items that might be lost to add to your insurance claim, with image proof of the before and after.

## How we built it

Our frontend is built with next.js. Our backend uses Flask and sqllite. We also use xrpl in order to put items on the blockchain for proof of their existence before the disaster. Here are the steps our AI uses to process videos and pictures into items:

1. The video is stitched together into a large panorama with OpenCV
2. We use YOLOv8 and Meta's segment anything model to generate areas of interest on the image.
3. We use Hyperbolic's inference (Qwen2 VL7B) to analyze each area and decide if it is an object to record and find its estimated monetary value
4. We place all images in ChromaDB to group/count items of the same type and match images before and after disaster. We originally used ResNet for our embedding for the vector database in order to compare items, but later found all-MiniLM-L6-v2 to work better for vector encoding. The ResNet embeddings were too high dimensional.
5. We generate a pdf claim to submit to the insurance company based on items that didn't appear after the disaster and were added by the user.

## Challenges we ran into

1. Segmentation problem: We tried a lot of different things in order to segment images well. We tried the segment everything model, SAM+yolo, and tried asking gemini to bounding boxes (it's the only vision language model that advertises this capability). We found SAM+yolo worked the best by far but only after lots of iteration and tuning.
2. System Design: lots of iteration required and thinking about the customer. For simplicity chose to host a single AWS EC2 instance although seriously considered hosting multiple servers for each general task like segmentation.
3. Video Panorama: How do you do segmentation on a video without getting duplicate items across frames? Our solution was to turn the video into a large image and run our machine learning on this image.
4. CORS struck us once again.

## Accomplishments that we're proud of

Super fast Segment Everything + Yolo + recognition + grouping pipeline that robustly recognized and catalogued items from video.
Well designed and aesthetic front-end.

## What we learned

There are no power outlets in SF. We learned how to deploy an EC2 instance, and also how to use vector databases.

## What's next for CLaiM

We would like to
## Inspiration

I got this idea because of the current hurricane Milton causing devastation across Florida.
The inspiration behind *Autonomous AI Society* stems from the need for faster, more efficient, and autonomous systems that can make critical decisions during disaster situations. With multiple sponsors like Fetch.ai, Groq, Deepgram, Hyperbolic, and Vapi providing powerful tools, I envisioned an intelligent system of AI agents capable of handling a disaster response chain—from analyzing distress calls to dispatching drones and contacting rescue teams. The goal was to build an AI-driven solution that can streamline emergency responses, save lives, and minimize risks.

## What it does

*Autonomous AI Society* is a fully autonomous multi-agent system that performs disaster response tasks in the following workflow:

1. **Distress Call Analysis**: The system first analyzes distress calls using Deepgram for speech-to-text and Hume AI to score distress levels. Based on the analysis, the agent identifies the most urgent calls and the city.
2. **Drone Dispatch**: The distress analyzer agent communicates with the drone agent (built using Fetch.ai) to dispatch drones to specific locations, assisting with flood and rescue operations.
3. **Human Detection**: Drones capture aerial images, which are analyzed by the human detection agent using Hyperbolic's LLaMA Vision model to detect humans in distress. The agent provides a description and coordinates.
4. **Priority-Based Action**: The drone results are displayed on a dashboard, ranked based on priority using Groq. Higher priority areas receive faster dispatches, and this is determined dynamically.
5. **Rescue Call**: The final agent, built using Vapi, places an emergency call to the rescue team. It uses instructions generated by Hyperbolic’s text model to give precise directions based on the detected individuals and their location.

## How I built it

The system consists of five agents, all built using **Fetch.ai**’s framework, allowing them to interact autonomously and make real-time decisions:

* **Request-sender agent** sends the initial requests.
* **Distress analyzer agent** uses **Hume AI** to analyze calls and **Groq** to generate dramatic messages.
* **Drone agent** dispatches drones to designated areas based on the distress score.
* **Human detection agent** uses **Hyperbolic’s LLaMA Vision** to process images and detect humans in danger.
* **Call rescue agent** sends audio instructions using **Deepgram**’s TTS and **Vapi** for automated phone calls.

## Challenges I ran into

* **Simulating a drone movement on florida map**: The lat\_lon\_to\_pixel function converts latitude and longitude coordinates to pixel positions on the screen. The drone starts at the center of Florida.
Its movement is calculated using trigonometry. The angle to the target city is calculated using math.atan2.
The drone moves towards the target using sin and cos functions.This allows placing cities and the drone accurately on the map.
* **Callibrating the map to right coordinates**: I had manually experiment with increasing and decreasing the coordinates to fit them at right spots on the florida map.
* **Coordinating AI agents**: Getting agents to communicate effectively while working autonomously was a challenge.
* **Handling dynamic priorities**: Ensuring real-time analysis and updating the priority of drone dispatch based on Groq's risk assessment was tricky.
* **Integration of multiple APIs**: Each sponsor's tools had specific nuances, and integrating all of them smoothly, especially with Fetch.ai, required careful handling.

## Accomplishments that I am proud of

* Successfully built an end-to-end autonomous system where AI agents can make intelligent decisions during a disaster, from distress call analysis to rescue actions.
* Integrated cutting-edge technologies like **Fetch.ai**, **Groq**, **Hyperbolic**, **Deepgram**, and **Vapi** in a single project to create a highly functional and real-time response system.

## What I learned

* **AI for disaster response**: Building systems that leverage multimodal AI agents can significantly improve response times and decision-making in life-critical scenarios.
* **Cross-platform integration**: We learned how to seamlessly integrate various tools, from vision AI to TTS to drone dispatch, using **Fetch.ai** and sponsor technologies.
* **Working with real-time data**: Developing an autonomous system that processes data in real-time provided insights into handling complex workflows.

## What's next for Autonomous AI Society

* **Scaling to more disasters**: Expanding the system to handle other types of natural disasters like wildfires or earthquakes.
* **Edge deployment**: Enabling drones and agents to run on the edge to reduce response times further.
* **Improved human detection**: Enhancing human detection with more precise models to handle low-light or difficult visual conditions.
* **Expanded rescue communication**: Integrating real-time communication with the victims themselves using Deepgram’s speech technology.
## Inspiration

Fire Insurance Finder or FIF is inspired the large amount of Wildfires in Canada which burn approximately 2.5 million ha of land each year according to Canadian Wildlife Federation. We wanted to design FIF to identify areas of crisis, as well as help deal with insurance claims efficiently by showing a map of the damaged areas.

## What it does

FIF has a 97% confusion matrix true positive accuracy rate and provides an automation of arduous structure damage inspection process which is currently a manual process. Through features such as real-time damage inspection, it can improve post-fire recovery, help homeowners, as well as insurance policy makers. FIF allows users to view a real time map of the area affected by fires and shows which houses were affected by the fire and which were not as the user uses a slider to move across the screen. The houses affected by the fire show up as red in colour.

## How we built it

Fire Insurance Finder does the following when a new user uploads an aerial survey:

1. FIF takes in an aerial survey through a dedicated Google Drive and stores it in the required format for running the model.
2. Partitioning: In this step, FIF uses a pre-trained model to segment all structures in a landscape.
3. Clipping: Once the structures are segmented, then based on the average building size in scene, square scenes centering the structure are cropped from the landscape
4. Classification: The cropped images are then classified as “damaged building” or “not damaged building”. For our classification model we performed transfer learning using Pytorch. More specifically, we used a ResNet18 network architecture pre-trained on Imagenet.
5. Rebuilding: After it has classified each cropped image, FIF moves the images from pixel-space to geospace by remapping the cropped images onto the original landscape scene.
6. Visualization: The web-app is renovated with the new aerial survey and with damaged shown in red colour.

## Challenges we ran into

It was initially hard to pick an idea because we wanted to make something that can be used for insurance as well as for the social good. We also had trouble initially in image processing and learning Google's API but we overcame that.

## Accomplishments that we're proud of

We are proud of being able to provide a real-time aerial view that can provide a clear big picture in the aftermath of a devastating fire. We are also proud of coming together as a team in a short amount of team to build the FIF.

## What we learned

We learned how to train models, as well as use Google's API. Some of our teammates also learned how to do react programming for the first time.

## What's next for Fire Insurance Finder (FIF)

As a team, we would like to continue developing FIF as an open-source platform since this project can truly help multiple stakeholders affected by the devastating wildfires. Some of our plans include the followings:
->Reach out to policy makers as well as first responders to test and improve FIF
->Use data driven approaches to augment training examples to better the detection accuracy
->Improve image resolutions