## 📞 The problem at hand.

How might we leverage an alert agent to ensure **24/7 student safety** from school shootings?

## 🗝️ Where did this stem from?

**School shootings. There are 2 per week**…and this number is NOT decreasing. Students are now worried about their safety but have no way to indicate potential danger. Just two days ago, a suspected shootout at UPenn left witnesses terrified, as suspects made sounds resembling gunshots while exiting. This ongoing sense of fear and uncertainty is unacceptable, especially when lives are at risk.

## 💻 What does Watchful.AI do?

1. **Instant alert** of weapons and suspicious activity, with the versatility and intelligence of GPT-4o
2. Search for and track suspicious activity through video semantic search
3. Monitor multiple video feeds and visualize position of threats in real time
4. **Alert local security** to resolve suspicious activity

## How does it work?

GPT-4o is used in harmony with its distant cousin, CLIP, to progressively distill gigabytes of simultaneous video footage into key anomalies and threat events, which can be efficiently actioned on by a human in the loop. Utilizing CLIP also has the added benefit of enabling semantic search over CCTV footage, so officers can spend less time scrolling through reels of video footage.

## 🛠️ How we built it

**UI/UX Design**
We used Figma and FigJam to lay out the user flow, using research from notable newspaper stations like The Daily Pennsylvanian and CNN to support our visual designs and design system with data.

**Frontend**
Next.js, ShadCN, Tailwind CSS, MappedIn

**Backend**
CLIP (PyTorch + Mac M2), GPT-4o, ChromaDB, FastAPI, OpenCV

## 🕵🏼 Challenges we ran into

A challenge we ran into was collecting data and mapping out the Penn Engineering Building manually. Many tools we wanted to use were unavailable, but we ended up contacting Mappedin and learning their software to create this crucial component of our product

Another significant challenge was utilizing multithreading to processes multiple video streams at a time on a single device. The backend has to ingest, embed, analyze, and serve huge amounts of data.

## ✔️ Accomplishments that we're proud of

We are proud to have built an impactful and well flushed out product in 36 hours! We were able to combine the knowledge of both our frontend team and backend team to work to our strengths and produce a fully functional tool that can be used to prevent an urgent and widespread issue – school shootings.

We were also proud to implement a new technique for cheap but accurate inference on large amounts of video data. Obviously calling GPT-4 for every frame of multiple video streams would be highly cumbersome, but by prescreening with embeddings (which run for free on local GPUs), then checking with GPT-4o, we get the accessibility of on-device AI with the power of huge hosted models.

## ☕️ What we learned

We learned that many students are anxious about school shootings in the US considering there’s an average of 2 shootings per week. Surprisingly, there aren’t any publicly available tools to alert citizens about potential shootings accurately, which is quite concerning. We also learned that we are able to create that tool we wished existed to ensure our safety. Learning how to use MappedIn was something new were discovered to do so.

## ⏭ What's next for Watchful.AI

We strive to implement Watchful.AI in every security system in the US to watch for potential shootouts and suspicious activity. We want to further develop the accuracy of our cameras and description search to ensure safety of not just students, but all citizens in the US.
# ThreatTrace AI: Your Solution for Swift Violence Detection

## ThreatTrace AI eliminates the need for constant human monitoring of camera footage, facilitating faster threat identification and enhanced safety.

## Inspiration

In response to the rising weapon-related crimes in Toronto, we developed a weapon detection tool using artificial intelligence to reduce the potential for violence.

## What It Does

ThreatTrace AI is a peacekeeping platform aiming to utilize AI object detection to identify weapons and alert the authorities regarding potential threats and violence. Our ultimate vision was to monitor real-time security footage for violence detection, eliminating the need for human oversight across multiple screens. However, due to time constraints, we focused on training our machine learning model solely to detect weapons, namely pistols and guns from images.

## How We Built It

Our frontend utilizes Python's Flask library with HTML and CSS, while the backend is powered by TensorFlow and various other libraries and repositories. We trained our machine learning model with a specific dataset and organized selected images in a folder, which we iterate through to create a 'slideshow' on the frontend.

## Challenges We Ran Into

Throughout the event, we encountered numerous challenges that shaped the course of our inspiration development as a team. We started off as two separate groups of three people, but then one person from one team fell ill on the day of the event, and the other team was looking to form a team of four. However, since there were many teams of two or three looking for more members, one team had to disassemble into a group of two and join another group of two to finally form a group of four. During our setup phase, our computers ran into a lot of troubles while trying to install the necessary packages for the machine learning model to function. Our original goal was to have three to all four of us train the learning model with datasets, but unfortunately, we spent half of the time, and only two members managed to set up the machine learning model by the end of the event. In the end, only one member managed to train the model, making this the greatest technical challenge we've encountered in the event and slowing our progress by a margin.

## Accomplishments That We're Proud Of

We're proud of the progress made from utilizing machine learning APIs such as TensorFlow and object detection for the very first time. Additionally, two members had no experience with Flask, and yet we were still able to develop a functional frontend in conjunction with HTML and CSS.

## What We Learned

We have learned about numerous new technical knowledge regarding libraries, packages, and technologies such as TensorFlow, virtual environments, GitHub, CocoAPI, and ObjectDetectionAPI. Since three of our members use Windows as the main operating system, we had to utilize both the Windows and the bash terminals to set up our repository. Through our numerous trials, we have also learned about the vast amount of time it takes to train a machine learning model and dataset before being able to make use of it. Finally, the most important lesson we have learned was team collaboration, as each of us has made use of our strengths, and we utilized our abilities well to facilitate this project.

## What's Next for ThreatTrace AI

Our goal is to continue training our learning model to accept more weapons at different lighting levels and angles, so that our dataset can be more refined as time goes. Then we will transition into training video footages with the learning model. Ultimately, we will reach our original vision of a real-time video violence detection AI.
## Inspiration

Natural disasters do more than just destroy property—they disrupt lives, tear apart communities, and hinder our progress toward a sustainable future. One of our team members from Rice University experienced this firsthand during a recent hurricane in Houston. Trees were uprooted, infrastructure was destroyed, and delayed response times put countless lives at risk.

* **Emotional Impact**: The chaos and helplessness during such events are overwhelming.
* **Urgency for Change**: We recognized the need for swift damage assessment to aid authorities in locating those in need and deploying appropriate services.
* **Sustainability Concerns**: Rebuilding efforts often use non-eco-friendly methods, leading to significant carbon footprints.

Inspired by these challenges, we aim to leverage AI, computer vision, and peer networks to provide rapid, actionable damage assessments. Our AI assistant can detect people in distress and deliver crucial information swiftly, bridging the gap between disaster and recovery.

## What it Does

The Garuda Dashboard offers a comprehensive view of current, upcoming, and past disasters across the country:

* **Live Dashboard**: Displays a heatmap of affected areas updated via a peer-to-peer network.
* **Drones Damage Analysis**: Deploy drones to survey and mark damaged neighborhoods using the Llava Vision-Language Model and generate reports for the Recovery Team.
* **Detailed Reporting**: Reports have annotations to classify damage types [tree, road, roof, water], human rescue needs, site accessibility [Can response team get to the site by land], and suggest equipment dispatch [Cranes, Ambulance, Fire Control].
* **Drowning Alert**: The drone footage can detect when it identifies a drowning subject and immediately call rescue teams
* **AI-Generated Summary**: Reports on past disasters include recovery costs, carbon footprint, and total asset/life damage.

## How We Built It

* **Front End**: Developed with Next.js for an intuitive user interface tailored for emergency use.
* **Data Integration**: Utilized Google Maps API for heatmaps and energy-efficient routing.
* **Real-Time Updates**: Custom Flask API records hot zones when users upload disaster videos.
* **AI Models**: Employed MSNet for real-time damage assessment on GPUs and Llava VLM for detailed video analysis.
* **Secure Storage**: Images and videos stored on Firebase database.

## Challenges We Faced

* **Model Integration**: Adapting MSNet with outdated dependencies required deep understanding of technical papers.
* **VLM Setup**: Implementing Llava VLM was challenging due to lack of prior experience.
* **Efficiency Issues**: Running models on personal computers led to inefficiencies.

## Accomplishments We're Proud Of

* **Technical Skills**: Mastered API integration, technical paper analysis, and new technologies like VLMs.
* **Innovative Impact**: Combined emerging technologies for disaster detection and recovery measures.
* **Complex Integration**: Successfully merged backend, frontend, and GPU components under time constraints.

## What We Learned

* Expanded full-stack development skills and explored new AI models.
* Realized the potential of coding experience in tackling real-world problems with interdisciplinary solutions.
* Balanced MVP features with user needs throughout development.

## What's Next for Garuda

* **Drone Integration**: Enable drones to autonomously call EMS services and deploy life-saving equipment.
* **Collaboration with EMS**: Partner with emergency services for widespread national and global adoption.
* **Broader Impact**: Expand software capabilities to address various natural disasters beyond hurricanes.