## Inspiration

How many times have you forgotten to take your medication and damned yourself for it? It has happened to us all, with different consequences. Indeed, missing out on a single pill can, for some of us, throw an entire treatment process out the window. Being able to keep track of our prescriptions is key in healthcare, this is why we decided to create PillsOnTime.

## What it does

PillsOnTime allows you to load your prescription information, including the daily dosage and refills, as well as reminders in your local phone calendar, simply by taking a quick photo or uploading one from the library. The app takes care of the rest!

## How we built it

We built the app with react native and expo using firebase for authentication. We used the built in expo module to access the devices camera and store the image locally. We then used the Google Cloud Vision API to extract the text from the photo. We used this data to then create a (semi-accurate) algorithm which can identify key information about the prescription/medication to be added to your calendar. Finally the event was added to the phones calendar with the built in expo module.

## Challenges we ran into

As our team has a diverse array of experiences, the same can be said about the challenges that each encountered. Some had to get accustomed to new platforms in order to design an application in less than a day, while figuring out how to build an algorithm that will efficiently analyze data from prescription labels. None of us had worked with machine learning before and it took a while for us to process the incredibly large amount of data that the API gives back to you. Also working with the permissions of writing to someones calendar was also time consuming.

## Accomplishments that we're proud of

Just going into this challenge, we faced a lot of problems that we managed to overcome, whether it was getting used to unfamiliar platforms or figuring out the design of our app.
We ended up with a result rather satisfying given the time constraints & we learned quite a lot.

## What we learned

None of us had worked with ML before but we all realized that it isn't as hard as we thought!! We will definitely be exploring more similar API's that google has to offer.

## What's next for PillsOnTime

We would like to refine the algorithm to create calendar events with more accuracy
## Inspiration

Today we live in a world that is all online with the pandemic forcing us at home. Due to this, our team and the people around us were forced to rely on video conference apps for school and work. Although these apps function well, there was always something missing and we were faced with new problems we weren't used to facing. Personally, forgetting to mute my mic when going to the door to yell at my dog, accidentally disturbing the entire video conference. For others, it was a lack of accessibility tools that made the experience more difficult. Then for some, simply scared of something embarrassing happening during class while it is being recorded to be posted and seen on repeat! We knew something had to be done to fix these issues.

## What it does

Our app essentially takes over your webcam to give the user more control of what it does and when it does. The goal of the project is to add all the missing features that we wished were available during all our past video conferences.

Features:
Webcam:
1 - Detect when user is away
This feature will automatically blur the webcam feed when a User walks away from the computer to ensure the user's privacy

2- Detect when user is sleeping
We all fear falling asleep on a video call and being recorded by others, our app will detect if the user is sleeping and will automatically blur the webcam feed.

3- Only show registered user
Our app allows the user to train a simple AI face recognition model in order to only allow the webcam feed to show if they are present. This is ideal to prevent ones children from accidentally walking in front of the camera and putting on a show for all to see :)

4- Display Custom Unavailable Image
Rather than blur the frame, we give the option to choose a custom image to pass to the webcam feed when we want to block the camera

Audio:
1- Mute Microphone when video is off
This option allows users to additionally have the app mute their microphone when the app changes the video feed to block the camera.

Accessibility:
1- ASL Subtitle
Using another AI model, our app will translate your ASL into text allowing mute people another channel of communication

2- Audio Transcriber
This option will automatically transcribe all you say to your webcam feed for anyone to read.

Concentration Tracker:
1- Tracks the user's concentration level throughout their session making them aware of the time they waste, giving them the chance to change the bad habbits.

## How we built it

The core of our app was built with Python using OpenCV to manipulate the image feed. The AI's used to detect the different visual situations are a mix of haar\_cascades from OpenCV and deep learning models that we built on Google Colab using TensorFlow and Keras.

The UI of our app was created using Electron with React.js and TypeScript using a variety of different libraries to help support our app. The two parts of the application communicate together using WebSockets from socket.io as well as synchronized python thread.

## Challenges we ran into

Dam where to start haha...

Firstly, Python is not a language any of us are too familiar with, so from the start, we knew we had a challenge ahead. Our first main problem was figuring out how to highjack the webcam video feed and to pass the feed on to be used by any video conference app, rather than make our app for a specific one.

The next challenge we faced was mainly figuring out a method of communication between our front end and our python. With none of us having too much experience in either Electron or in Python, we might have spent a bit too much time on Stack Overflow, but in the end, we figured out how to leverage socket.io to allow for continuous communication between the two apps.

Another major challenge was making the core features of our application communicate with each other. Since the major parts (speech-to-text, camera feed, camera processing, socket.io, etc) were mainly running on blocking threads, we had to figure out how to properly do multi-threading in an environment we weren't familiar with. This caused a lot of issues during the development, but we ended up having a pretty good understanding near the end and got everything working together.

## Accomplishments that we're proud of

Our team is really proud of the product we have made and have already begun proudly showing it to all of our friends!

Considering we all have an intense passion for AI, we are super proud of our project from a technical standpoint, finally getting the chance to work with it. Overall, we are extremely proud of our product and genuinely plan to better optimize it in order to use within our courses and work conference, as it is really a tool we need in our everyday lives.

## What we learned

From a technical point of view, our team has learnt an incredible amount the past few days. Each of us tackled problems using technologies we have never used before that we can now proudly say we understand how to use. For me, Jonathan, I mainly learnt how to work with OpenCV, following a 4-hour long tutorial learning the inner workings of the library and how to apply it to our project. For Quan, it was mainly creating a structure that would allow for our Electron app and python program communicate together without killing the performance. Finally, Zhi worked for the first time with the Google API in order to get our speech to text working, he also learned a lot of python and about multi-threadingbin Python to set everything up together. Together, we all had to learn the basics of AI in order to implement the various models used within our application and to finally attempt (not a perfect model by any means) to create one ourselves.

## What's next for Boom. The Meeting Enhancer

This hackathon is only the start for Boom as our team is exploding with ideas!!! We have a few ideas on where to bring the project next. Firstly, we would want to finish polishing the existing features in the app. Then we would love to make a market place that allows people to choose from any kind of trained AI to determine when to block the webcam feed. This would allow for limitless creativity from us and anyone who would want to contribute!!!!
## Inspiration

This project was inspired by a team member’s family, his grandparents always have to take medicine but often forget about it. Not only his grandparents forget the medicine also his mom. Although, his mom is very young but in a very fast paced society nowadays people always forget to do small things like taking their pills. Due to this inspiration, we decided to develop a pill reminder, but then we got inspired by a Tik Tok video about a person who has Parkinson’s disease and he couldn’t pick up an individual pill from the container. In end, we decide to create this project that will resolve the problem of people forgetting to take their pills as well as helping people to easily take individual pills.

## What it does

Our project the Delta Dispenser uses an app to communicate with the database to set up a specific time to alert users to take their pills as well as tracking their pills information in the app. The hardware of Delta Dispenser will alert the user when the time is reached and automatically dispense the correct amount of the pills into the container.

## How we built it

The frontend of the app is made with **Flutter**, the app communicates with a **firebase real-time database** to store medicinal and scheduling information for the user. The physical component uses an **embedded microcontroller** called an ESP-32 which we chose for its ability to connect to WiFi and be able to sync with the firebase database to know when to dispense the pills.

## Challenges we ran into

The time constraint was definitely a big challenge and we accounted for that by deciding which features were most important in emphasizing our main idea for this project. These parts include the mechanical indexer of the pills, the interface the user would interact with, and how the database would look for communication with the app and the embedded device.

## Accomplishments that we're proud of

We are most proud of how this project utilized many different aspects of engineering, from mechanical to electrical and software. Our team did a really good job at communicating throughout the design process which made integration at the end much easier.

## What we learned

During this project, we had learned how to flutter to create a mobile app as well as learning how firebase works. Throughout this project, although we only learned a few skills that will be very useful in the future. The most important part was that we were able to develop upon the skills we already had. For example, now we are able to develop hardware that could communicate through firebase.

## What's next for Delta Dispenser

The next steps for the Delta Dispenser include building a fully 3D printed prototype, along with the control box and hopper as shown in the CAD renders. On the software side, we would also like to add the ability for more complicated drug scheduling, while keeping the UI easy enough for anyone to set up. Having another portal that allows a doctor to directly input the information themselves is also a feature we are interested in having.