## Inspiration

Positive cases are spreading really quickly on campus, and many students decide against going to dormitories because of ineffectively mask enforcements. We hope to make these places safer so college students can come back to university sooner and enjoy their precious student-life experience more!

## What it does

Cozy Koalas allows for the identification of people as well as whether or not they’re currently wearing a mask, using YOLOv5. It would allow dormitories to monitor statistics such as number of people with/without mask throughout the weeks. Another feature of our application is our infrared sensor which detects the temperature of an individual (<https://ieeexplore.ieee.org/document/9530864>) . If that person’s temperature is abnormally high and may have a fever, a notification is sent using Twilio so that they are aware of this and will take action to reduce risk.

## How we built it

*Machine Learning*: We use images from a camera feed to detect a) whenever a person comes into the frame, and b) whether that person is either:

1. Not wearing a mask
2. Wearing a mask incorrectly
3. Wearing a mask correctly

We use YOLOv5 (You Only Look Once) model, a real-time object detection model based on convolutional neural networks ([https://arxiv.org/pdf/2102.05402.pdf)](https://arxiv.org/pdf/2102.05402.pdf) and incorporated a Python script to help label our data.

We ran multiple iterations through YOLO to improve our model and labelling. Initially, we only had 2 labels: mask or no\_mask. Howeverk, this was ultimately problematic as our model was unable to detect when someone wore there mask incorrectly (e.g. doesn’t cover nose). That’s why after multiple iterations, we added another label using a Python script. While that was our main change, our multiple iterations helped us balance our data and tune our hyperparameters, leading to a greater accuracy. This backend is connected to a server and a database via Google Cloud’s Firebase for the moment.

*Front End:* The front end fetches its information from the Google Cloud’s Firebase and displays it in an interactive dashboard. The dashboard and its following pages were done using Material UI, a front-end library in React. Furthermore, a number of other libraries or tools were used to help sort the data/ beautify the application, such as lodash, iconify, ant-design, faker, etc.

## Challenges we ran into

We struggled on really understanding the YOLO model and convolutional networks before implementing it. We initially tried to implement parts of it without understanding this nor torch as this was our first time working with it. However, to actually improve our model, we really needed to understand the parts to change. We also struggled immensely on connecting the two parts of our projects together. In the front end, we ran into multiple typeerrors that had to do with states and usages of props.

## Accomplishments that we're proud of

We’re proud of how we worked as a team and leveraged our different specialities and managed to create a working product together once we stitched different parts of our project together.

The computer vision and machine learning modules and libraries are state-of-the-art and very much used in current technologies. Our model is even able to differentiate very corner case situations, such as when one has their hand to cover their face instead of a real mask or when the mask is worn incorrectly. The dashboard also turned out to be simple, clean and elegant and reflected what we initially went for.

## What we learned

ML models are easy to use yet hard at the same time. Documentation and APIs are useful, but a large part of the understanding is understanding your data and what changes need to be made to improve your model. Data visualisation and metrics were very helpful for this part! Another great thing was the potential and opportunity which comes from already pre-existing and labeling datasets. In our project, Roboflow and Kaggle proved to be hugely useful and saved us a lot of time.

## What's next for Cozy Koalas

A mask recognition system can be used in multiple other fields. For instance, airports, hospitals, quarantine centers, malls, schools and offices could reinforce their mask mandates without having to buy extra hardware. A software that can be used in conjunction with existing camera feeds would simplify this task and the analytics provided could also help those organizations better plan their resources.

In the technical aspect, a clear next step would be to incorporate face recognition into our model using FaceNet library (<https://ieeexplore.ieee.org/document/9451684>). This would essentially map the face images it gets from the cameras’ feed to identified individuals inside our deep convolutional network. Whether or not this feature will be used is up to the user’s discretion; however, it would be necessary for our system to automatically send a text message to the right person. Other next steps include increasing accuracy through a more balanced dataset (SMOTE balancing can only do so much...), adding distance measurements, and providing more analytics in the dashboard.
## Team 13

* Jayden Chiu - Jayden#3838
* Marcus Chok - Ace no ™#8264
* Steven Yau - stevenn#8855
* Brittany Wong - Brittany#8474

## Inspiration

We wanted to learn something related to ML/AI as we all thought it was an interesting topic we can learn for QHacks and we also wanted to tackle an issue related to the current pandemic, so we decided to make a website for business monitors to monitor if customers are wearing masks properly. We decided to use Tensorflow and OpenCV along with Flask to tackle these issues and create a website. Now, business owners can track who's wearing a mask properly depending on where they set up their camera and monitor it from a single website, making it much easier monitor potential exposure to COVID-19 in their business!

## What it does

* Detects if user is wearing a mask, wearing a mask incorrectly, or not wearing a mask at all from webcam or uploaded video
* Displays live webcam footage on home page
* Allows users to upload their own videos and download them with detected masks and violations

## How we built it

### Datasets

* [MaskedFace-Net](https://github.com/cabani/MaskedFace-Net), AI-generated face mask dataset
* [RWMFD](https://github.com/X-zhangyang/Real-World-Masked-Face-Dataset), real-world face mask dataset
* [WIDERFACE Dataset](http://shuoyang1213.me/WIDERFACE/), face dataset

### Models

* Finetuned model based off of [MobileNetV2](https://keras.io/api/applications/mobilenet/) for detecting mask status of face
* [OpenCV DNN sample](https://github.com/opencv/opencv/tree/master/samples/dnn) for face detection model

### Technologies

* Google Colab GPU runtime used to train models

### Stack

* Flask for website backend routing
* OpenCV used for video processing
* Tensorflow and Keras for training models
* Bootstrap CSS for UI

## Challenges we ran into

* Model inaccuracies depending on viewing angle, lighting, and mask
* Depending on epochs, initial learning rate, etc., model would overfit to data
* Choosing the best gradient descent optimization algorithm for training
* Model would sometimes break because of improper frame loading
* Google Colab would sometimes take excruciatingly long to load images from datasets
* Bootstrap CSS would not create UI as we wanted

## What we learned and our accomplishments

* Learning Tensorflow and Flask for the project
* Learning about how ML and neural networks work
* Learning how to use Google Colab to train your own ML models
* Finding proper face mask datasets to train the model with
* Running OpenCV locally compared to displaying it on a website using Flask
* Creating a responsive UI using Bootstrap CSS
* Training a model with a high enough accuracy (~93% accuracy) that we can deploy

## What's next for Mask Monitor

* Create authentication system which can email users if a mask violation is detected
* Allow users to upload webcam footage to the cloud so they can review the videos later
* Find larger datasets to increase the accuracy of our model
* Expand our app to tackle more COVID-19 related issues such as social distancing or self-isolation
* Deploy the app so owners can use it from anywhere
## What inspired us to build it

Guns are now the leading cause of death among American children and teens, with 1 in every 10 gun deaths occurring in individuals aged 19 or younger. School shootings, in particular, have become a tragic epidemic in the U.S., underscoring the urgent need for enhanced safety measures. Our team united with a shared vision to leverage AI technology to improve security in American schools, helping to protect children and ensure their safety.

## What it does

Our product leverages advanced AI technology to enhance school safety by detecting potential threats in real-time. By streaming surveillance footage, our AI system can identify weapons, providing instant alerts to security personnel and administrators. In addition to visual monitoring, we integrate audio streaming to analyze changes in sentiment, such as raised voices or signs of distress. This dual approach—combining visual and auditory cues—enables rapid response to emerging threats.

## How we built it

We partnered with incredible sponsors—Deepgram, Hyperbolic, Groq, and Fetch.AI—to develop a comprehensive security solution that uses cutting-edge AI technologies. With their support, we were able to conduct fast AI inference, deploy an emergency contact agent, and create intelligent systems capable of tracking potential threats and key variables, all to ensure the safety of our communities.

For real-time data processing, we utilized Firebase and Convex to enable rapid write-back and retrieval of critical information. Additionally, we trained our weapon detection agent using Ultralytics YOLO v8 on the Roboflow platform, achieving an impressive ~90% accuracy. This high-performance detection system, combined with AI-driven analytics, provides a robust safety infrastructure capable of identifying and responding to threats in real time.

## Challenges we ran into

Streaming a real-time AI object detection model with both low latency and high accuracy was a significant challenge. Initially, we experimented with Flask and FastAPI for serving our model, followed by trying AWS and Docker to improve performance. However, after further optimization efforts, we ultimately integrated Roboflow.js directly in the browser using a Native SDK. This approach gave us a substantial advantage, allowing us to run the model efficiently within the client environment. As a result, we achieved the ability to track weapons quickly and accurately in real time, meeting the critical demands of our security solution.

## Accomplishments that we're proud of

We are incredibly proud of the features our product offers, providing a comprehensive and fully integrated security experience. Beyond detecting weapons and issuing instant alerts to law enforcement, faculty, and students through AI-powered agents, we also implemented extensive sentiment analysis. This enables us to detect emotional escalations that may signal potential threats. All of this is supported by real-time security data displays, ensuring that key decision-makers are always informed with up-to-the-minute information. Our system seamlessly brings together cutting-edge AI and real-time data processing to deliver a robust, proactive security solution.

## What we learned

We learned that the night is darkest right before the dawn... and that we need to persevere and be steadfast as a team to see our vision come to fruition.

## What's next for Watchdog

We want to get incorporated in the American school system!