## 📞 The problem at hand.

How might we leverage an alert agent to ensure **24/7 student safety** from school shootings?

## 🗝️ Where did this stem from?

**School shootings. There are 2 per week**…and this number is NOT decreasing. Students are now worried about their safety but have no way to indicate potential danger. Just two days ago, a suspected shootout at UPenn left witnesses terrified, as suspects made sounds resembling gunshots while exiting. This ongoing sense of fear and uncertainty is unacceptable, especially when lives are at risk.

## 💻 What does Watchful.AI do?

1. **Instant alert** of weapons and suspicious activity, with the versatility and intelligence of GPT-4o
2. Search for and track suspicious activity through video semantic search
3. Monitor multiple video feeds and visualize position of threats in real time
4. **Alert local security** to resolve suspicious activity

## How does it work?

GPT-4o is used in harmony with its distant cousin, CLIP, to progressively distill gigabytes of simultaneous video footage into key anomalies and threat events, which can be efficiently actioned on by a human in the loop. Utilizing CLIP also has the added benefit of enabling semantic search over CCTV footage, so officers can spend less time scrolling through reels of video footage.

## 🛠️ How we built it

**UI/UX Design**
We used Figma and FigJam to lay out the user flow, using research from notable newspaper stations like The Daily Pennsylvanian and CNN to support our visual designs and design system with data.

**Frontend**
Next.js, ShadCN, Tailwind CSS, MappedIn

**Backend**
CLIP (PyTorch + Mac M2), GPT-4o, ChromaDB, FastAPI, OpenCV

## 🕵🏼 Challenges we ran into

A challenge we ran into was collecting data and mapping out the Penn Engineering Building manually. Many tools we wanted to use were unavailable, but we ended up contacting Mappedin and learning their software to create this crucial component of our product

Another significant challenge was utilizing multithreading to processes multiple video streams at a time on a single device. The backend has to ingest, embed, analyze, and serve huge amounts of data.

## ✔️ Accomplishments that we're proud of

We are proud to have built an impactful and well flushed out product in 36 hours! We were able to combine the knowledge of both our frontend team and backend team to work to our strengths and produce a fully functional tool that can be used to prevent an urgent and widespread issue – school shootings.

We were also proud to implement a new technique for cheap but accurate inference on large amounts of video data. Obviously calling GPT-4 for every frame of multiple video streams would be highly cumbersome, but by prescreening with embeddings (which run for free on local GPUs), then checking with GPT-4o, we get the accessibility of on-device AI with the power of huge hosted models.

## ☕️ What we learned

We learned that many students are anxious about school shootings in the US considering there’s an average of 2 shootings per week. Surprisingly, there aren’t any publicly available tools to alert citizens about potential shootings accurately, which is quite concerning. We also learned that we are able to create that tool we wished existed to ensure our safety. Learning how to use MappedIn was something new were discovered to do so.

## ⏭ What's next for Watchful.AI

We strive to implement Watchful.AI in every security system in the US to watch for potential shootouts and suspicious activity. We want to further develop the accuracy of our cameras and description search to ensure safety of not just students, but all citizens in the US.
## Inspiration

After years of teaching methods remaining constant, technology has not yet infiltrated the classroom to its full potential. One day in class, it occurred to us that there must be a correlation between students behaviour in classrooms and their level of comprehension.

## What it does

We leveraged Apple's existing API's around facial detection and combined it with the newly added Core ML features to track students emotions based on their facial queues. The app can follow and analyze up to ~ ten students and provide information in real time using our dashboard.

## How we built it

The iOS app integrated Apple's Core ML framework to run a [CNN](https://www.openu.ac.il/home/hassner/projects/cnn_emotions/) to detect people's emotions from facial queues. The model was then used in combination with Apple's Vision API to identify and extract student's face's. This data was then propagated to Firebase for it to be analyzed and displayed on a dashboard in real time.

## Challenges we ran into

Throughout this project, there were several issues regarding how to improve the accuracy of the facial results. Furthermore, there were issues regarding how to properly extract and track users throughout the length of the session. As for the dashboard, we ran into problems around how to display data in real time.

## Accomplishments that we're proud of

We are proud of the fact that we were able to build such a real-time solution. However, we are happy to have met such a great group of people to have worked with.

## What we learned

Ozzie learnt more regarding CoreML and Vision frameworks.

Haider gained more experience with front-end development as well as working on a team.

Nakul gained experience with real-time graphing as well as helped developed the dashboard.

## What's next for Flatline

In the future, Flatline could grow it's dashboard features to provide more insight for the teachers. Also, the accuracy of the results could be improved by training a model to detect emotions that are more closely related to learning and student's behaviours.
## Inspiration

Research has shown us that new hires, women and under-represented minorities in the workplace could feel intimidated or uncomfortable in team meetings. Since the start of remote work, new hires lack the in real life connections, are unable to take a pulse of the group and are fearful to speak what’s on their mind. Majority of the time this is also due to more experienced individuals interrupting them or talking over them without giving them a chance to speak up. This feeling of being left out often makes people not contribute to their highest potential. Links to the reference studies and articles are at the bottom.

As new hire interns every summer, we personally experienced the communication and participation problem in team meetings and stand ups. We were new and felt intimidated to share our thoughts in fear of them being dismissed or ignored. Even though we were new hires and had little background, we still had some sound ideas and opinions to share that were instead bottled up inside us.

We found out that the situation is the same for women in general and especially under-represented minorities. We built this tool for ourselves and to those around us to feel comfortable and inclusive in team meetings. Companies and organizations must do their part in ensuring that their workplace is an inclusive community for all and that everyone has the opportunity to participate equally in their highest potential. With the pandemic and widespread adoption of virtual meetings, this is an important problem globally that we must all address and we believe Vocal aims to help solve it.

## What it does

Vocal empowers new hires, women, and under-represented minorities to be more involved and engaged in virtual meetings for a more inclusive team.

Google Chrome is extremely prevalent and our solution is a proof-of-concept Chrome Extension and Web Dashboard that works with Google Meet meetings. Later we would support others platforms such as Zoom, Webex, Skype, and others.

When the user joins the Google Meet meeting, our Extension automatically detects it and collects statistics regarding the participation of each team member. A percentage is shown next to their name to indicate their contribution and also a ranking is shown that indicates how often you spoke compared to others. When the meeting ends, all of this data is sent to the web app dashboard using Google Cloud and Firebase database. On the web app, the users can see their participation in the current meeting and progress from the past meetings with different metrics. Plants are how we gamify participation. Your personal plant grows, the more you contribute in meetings. Meetings are organized through sprints and contribution throughout the sprint will be reflected in the growth of the plant.

**Dashboard**: You can see your personal participation statistics. It show your plant, monthly interaction level graph, percent interaction with other team members (how often and which teammates you piggy back on when responding). Lastly, it also has Overall Statistics such as percent increase in interactions compared to last week, meeting participation streak, average engagement time, and total time spoken. You can see your growth in participation reflected in the plant growth. **Vocal provides lots of priceless data for the management, HR, and for the team overall to improve productivity and inclusivity.**

**Team**: Many times our teammates are stressed or go through other feelings but simply bottle it up. In the Team page, we provide Team Sentiment Graph and Team Sentiments. The graphs shows how everyone in the team has been feeling for the current sprint. Team members would check in anonymously at the end of the every week on how they’re feeling (Stressed, Anxious, Neutral, Calm, Joyful) and the whole team can see it. If someone’s feeling low, other teammates can reach out anonymously in the chat and offer them support and they both can choose to reveal their identity if they want. **Feeling that your team cares about you and your mental health can foster an inclusive community.**

**Sprints Garden**: This includes all of the previous sprints that you completed. It also shows the whole team’s garden so you can compare across teammates on how much you have been contributing relatively.

**Profile**: This is your personal profile where you will see your personal details, the plants you have grown in the past over all the sprints you have worked on - your forest, your anonymous conversations with your team members. Your garden is here to motivate you and help you grow more plants and ultimately contribute more to meetings.

**Ethics/Privacy: We found very interesting ways to collect speaking data without being intrusive. When the user is talking only the mic pulses are recorded and analyzed as a person spoken. No voice data or transcription is done to ensure that everyone can feel safe while using the extension.**

**Sustainability/Social Good**: Companies that use Vocal can partner to plant the trees grown during sprints in real life by partnering with organizations that plant real trees under the corporate social responsibility (CSR) initiative.

## How we built it

The System is made up of three independent modules.
Chrome Extension: This module works with Google meet and calculates the statistics of the people who joined the meet and stores the information of the amount of time an individual contributes and pushes those values to the database.

Firebase: It stores the stats available for each user and their meeting attended. Percentage contribution, their role, etc.

Web Dashboard: Contains the features listed above. It fetches data from firebase and then renders it to display 3 sections on the portal. a. Personal Garden - where an individual can see their overall performance, their stats and maintain a personal plant streak. b. Group Garden - where you can see the overall performance of the team, team sentiment, anonymous chat function. After each sprint cycle, individual plants are added to the nursery. c. Profile with personal meeting logs, ideas and thoughts taken in real-time calls.

## Challenges we ran into

We had challenges while connecting the database with the chrome extension. The Google Meet statistics was also difficult to do since we needed to find clever ways to collect the speaking statistics without infringing on privacy. Also, 36 hours was a very short time span for us to implement so many features, we faced a lot of time pressure but we learned to work well under pressure!

## Accomplishments that we're proud of

This was an important problem that we all deeply cared about since we saw people around us face this on a daily basis. We come from different backgrounds, but for this project we worked as one team and used our expertise, and learned what we weren’t familiar with in this project. We are so proud to have created a tool to make under-represented minorities, women and new hires feel more inclusive and involved.

We see this product as a tool we’d love to use when we start our professional journeys. Something that brings out the benefits of remote work, at the same time being tech that is humane and delightful to use.

## What's next for Vocal

Vocal is a B2B product that companies and organizations can purchase. The chrome extension to show meeting participation would be free for everyone. The dashboard and the analytics will be priced depending on the company. The number of insights and data that can be extracted from one data point(user participation) will be beneficial to the company (HR & Management) to make their workplace more inclusive and productive. The data can also be analyzed to promote inclusion initiatives and other events to support new hires, women, and under-represented minorities.

We already have so many use cases that were hard to build in the duration of the hackathon. Our next step would be to create a Mobile app, more Video Calling platform integrations including Zoom, Microsoft Teams, Devpost video call, and implement chat features. We also see this also helping in other industries like ed-tech, where teachers and students could benefit form active participation.

## References

1. <https://www.nytimes.com/2020/04/14/us/zoom-meetings-gender.html>
2. <https://www.nature.com/articles/nature.2014.16270>
3. ​​<https://www.fastcompany.com/3030861/why-women-fail-to-speak-up-at-high-level-meetings-and-what-everyone-can-do-about>
4. <https://hbr.org/2014/06/women-find-your-voice>
5. <https://www.cnbc.com/2020/09/03/45percent-of-women-business-leaders-say-its-difficult-for-women-to-speak-up-in-virtual-meetings.html>