## Inspiration

We are heading to the wireless future where human and machine are united together

## What it does

Our program can analysis and record the brain wave and control the the Tello to taking off and landing.

## How we built it

First, we find a starter kit for brainwave detection called MindWave Mobile 2. A program on pc with GUI and read the serial information, and perform the data analysis and category. A Tello drone is connected to same PC through the network. and program can send command through WIFI to control the drone directly.

## Challenges we ran into

1. Set up the connection between PC and MindWave devices: there is no very detailed documents about how the communication process being performed. We spent a lot of time to analysis the serial communication and its output format.
2. Set up the comment control for Tello: we also spend some time in this step due to the package online mostly based on the python2, and our program is running by python3, so we have to debug and change some code to make it work.
3. GUI design and Concurrent programming: we spent significant amount of time on debugging the logic of the GUI, also, the program flow to read and write and share data between threads.
4. Detection

## Accomplishments that we're proud of

Being about to write software to communicate with software, and being something cool

## What we learned

Team Collaboration, Working under stress, Have Fun !!!

## What's next for Mind Tello

We are going to improve the control accuracy, and the detection.
## What is it?

Using the Intel RealSense, we first extracted a user's hand movements and mapped certain gestures to specific actions for the drone. Then we transmitted the data to an Arduino using low energy bluetooth. The Arduinos generated the signals readable by the transmitter chip using digital potentiometers. Having never used FPV before, we included a GoPro with a 5.8 GHz transmitter to stream live first person view to a wireless monitor.

## Inspiration

We have always been interested in drones and their increased prevalence in our culture. We love reverse engineering commercial products, and we wanted to experiment with the Intel RealSense.

## How We built it

In order to extract gestures, we used the Realsense SDK in C++ to extract the coordinates on a user's hand. Doing so, we were able to match certain combinations of coordinates to different instructions for the drone. The camera first tracks where the user's hand is and will move the drone according to where that is. More complex custom instructions such as landing the drone and havering can be called by using two hands. The drone was flown by taking apart its transmitter and completely disassembling it. We used digital potentiometers to digitally and autonomously emulate the transmitter joysticks. Out homemade transmitter is controlled by four Arduino Unos, which was necessary because the digital potentiometers were controlled by SPI and we had to work with a limited number of SPI inputs in the Arduino Uno.

## Challenges I ran into

The most difficult part of this project was emulating the drone control system. Due to the tight specifications of the controller, it was very difficult to create signals that fit within its tolerance boundaries. It took many hours of calibration for us to figure out exactly what signals were allowed. After that, we had to map controls to these signals. An additional challenge was generating data using Intel RealSense that was then transmitted to the master Arduino device using a Sparkfun low energy Bluetooth board.
## 📞 The problem at hand.

How might we leverage an alert agent to ensure **24/7 student safety** from school shootings?

## 🗝️ Where did this stem from?

**School shootings. There are 2 per week**…and this number is NOT decreasing. Students are now worried about their safety but have no way to indicate potential danger. Just two days ago, a suspected shootout at UPenn left witnesses terrified, as suspects made sounds resembling gunshots while exiting. This ongoing sense of fear and uncertainty is unacceptable, especially when lives are at risk.

## 💻 What does Watchful.AI do?

1. **Instant alert** of weapons and suspicious activity, with the versatility and intelligence of GPT-4o
2. Search for and track suspicious activity through video semantic search
3. Monitor multiple video feeds and visualize position of threats in real time
4. **Alert local security** to resolve suspicious activity

## How does it work?

GPT-4o is used in harmony with its distant cousin, CLIP, to progressively distill gigabytes of simultaneous video footage into key anomalies and threat events, which can be efficiently actioned on by a human in the loop. Utilizing CLIP also has the added benefit of enabling semantic search over CCTV footage, so officers can spend less time scrolling through reels of video footage.

## 🛠️ How we built it

**UI/UX Design**
We used Figma and FigJam to lay out the user flow, using research from notable newspaper stations like The Daily Pennsylvanian and CNN to support our visual designs and design system with data.

**Frontend**
Next.js, ShadCN, Tailwind CSS, MappedIn

**Backend**
CLIP (PyTorch + Mac M2), GPT-4o, ChromaDB, FastAPI, OpenCV

## 🕵🏼 Challenges we ran into

A challenge we ran into was collecting data and mapping out the Penn Engineering Building manually. Many tools we wanted to use were unavailable, but we ended up contacting Mappedin and learning their software to create this crucial component of our product

Another significant challenge was utilizing multithreading to processes multiple video streams at a time on a single device. The backend has to ingest, embed, analyze, and serve huge amounts of data.

## ✔️ Accomplishments that we're proud of

We are proud to have built an impactful and well flushed out product in 36 hours! We were able to combine the knowledge of both our frontend team and backend team to work to our strengths and produce a fully functional tool that can be used to prevent an urgent and widespread issue – school shootings.

We were also proud to implement a new technique for cheap but accurate inference on large amounts of video data. Obviously calling GPT-4 for every frame of multiple video streams would be highly cumbersome, but by prescreening with embeddings (which run for free on local GPUs), then checking with GPT-4o, we get the accessibility of on-device AI with the power of huge hosted models.

## ☕️ What we learned

We learned that many students are anxious about school shootings in the US considering there’s an average of 2 shootings per week. Surprisingly, there aren’t any publicly available tools to alert citizens about potential shootings accurately, which is quite concerning. We also learned that we are able to create that tool we wished existed to ensure our safety. Learning how to use MappedIn was something new were discovered to do so.

## ⏭ What's next for Watchful.AI

We strive to implement Watchful.AI in every security system in the US to watch for potential shootouts and suspicious activity. We want to further develop the accuracy of our cameras and description search to ensure safety of not just students, but all citizens in the US.