## Inspiration

Ever wish you didn’t need to purchase a stylus to handwrite your digital notes? Each person at some point hasn’t had the free hands to touch their keyboard. Whether you are a student learning to type or a parent juggling many tasks, sometimes a keyboard and stylus are not accessible. We believe the future of technology won’t even need to touch anything in order to take notes. HoverTouch utilizes touchless drawings and converts your (finger)written notes to typed text! We also have a text to speech function that is Google adjacent.

## What it does

Using your index finger as a touchless stylus, you can write new words and undo previous strokes, similar to features on popular note-taking apps like Goodnotes and OneNote. As a result, users can eat a slice of pizza or hold another device in hand while achieving their goal. HoverTouch tackles efficiency, convenience, and retention all in one.

## How we built it

Our pre-trained model from media pipe works in tandem with an Arduino nano, flex sensors, and resistors to track your index finger’s drawings. Once complete, you can tap your pinky to your thumb and HoverTouch captures a screenshot of your notes as a JPG. Afterward, the JPG undergoes a masking process where it is converted to a black and white picture. The blue ink (from the user’s pen strokes) becomes black and all other components of the screenshot such as the background become white. With our game-changing Google Cloud Vision API, custom ML model, and vertex AI vision, it reads the API and converts your text to be displayed on our web browser application.

## Challenges we ran into

Given that this was our first hackathon, we had to make many decisions regarding feasibility of our ideas and researching ways to implement them. In addition, this entire event has been an ongoing learning process where we have felt so many emotions — confusion, frustration, and excitement. This truly tested our grit but we persevered by uplifting one another’s spirits, recognizing our strengths, and helping each other out wherever we could.

One challenge we faced was importing the Google Cloud Vision API. For example, we learned that we were misusing the terminal and our disorganized downloads made it difficult to integrate the software with our backend components. Secondly, while developing the hand tracking system, we struggled with producing functional Python lists. We wanted to make line strokes when the index finger traced thin air, but we eventually transitioned to using dots instead to achieve the same outcome.

## Accomplishments that we're proud of

Ultimately, we are proud to have a working prototype that combines high-level knowledge and a solution with significance to the real world. Imagine how many students, parents, friends, in settings like your home, classroom, and workplace could benefit from HoverTouch's hands free writing technology.

This was the first hackathon for ¾ of our team, so we are thrilled to have undergone a time-bounded competition and all the stages of software development (ideation, designing, prototyping, etc.) toward a final product. We worked with many cutting-edge softwares and hardwares despite having zero experience before the hackathon.

In terms of technicals, we were able to develop varying thickness of the pen strokes based on the pressure of the index finger. This means you could write in a calligraphy style and it would be translated from image to text in the same manner.

## What we learned

This past weekend we learned that our **collaborative** efforts led to the best outcomes as our teamwork motivated us to preserve even in the face of adversity. Our continued **curiosity** led to novel ideas and encouraged new ways of thinking given our vastly different skill sets.

## What's next for HoverTouch

In the short term, we would like to develop shape recognition. This is similar to Goodnotes feature where a hand-drawn square or circle automatically corrects to perfection.

In the long term, we want to integrate our software into web-conferencing applications like Zoom. We initially tried to do this using WebRTC, something we were unfamiliar with, but the Zoom SDK had many complexities that were beyond our scope of knowledge and exceeded the amount of time we could spend on this stage.

### [HoverTouch Website](hoverpoggers.tech)
## Inspiration

One of our teammate’s grandfathers suffers from diabetic retinopathy, which causes severe vision loss.
Looking on a broader scale, over 2.2 billion people suffer from near or distant vision impairment worldwide. After examining the issue closer, it can be confirmed that the issue disproportionately affects people over the age of 50 years old. We wanted to create a solution that would help them navigate the complex world independently.

## What it does

### Object Identification:

Utilizes advanced computer vision to identify and describe objects in the user's surroundings, providing real-time audio feedback.

### Facial Recognition:

It employs machine learning for facial recognition, enabling users to recognize and remember familiar faces, and fostering a deeper connection with their environment.

### Interactive Question Answering:

Acts as an on-demand information resource, allowing users to ask questions and receive accurate answers, covering a wide range of topics.

### Voice Commands:

Features a user-friendly voice command system accessible to all, facilitating seamless interaction with the AI assistant: Sierra.

## How we built it

* Python
* OpenCV
* GCP & Firebase
* Google Maps API, Google Pyttsx3, Google’s VERTEX AI Toolkit (removed later due to inefficiency)

## Challenges we ran into

* Slow response times with Google Products, resulting in some replacements of services (e.g. Pyttsx3 was replaced by a faster, offline nlp model from Vosk)
* Due to the hardware capabilities of our low-end laptops, there is some amount of lag and slowness in the software with average response times of 7-8 seconds.
* Due to strict security measures and product design, we faced a lack of flexibility in working with the Maps API. After working together with each other and viewing some tutorials, we learned how to integrate Google Maps into the dashboard

## Accomplishments that we're proud of

We are proud that by the end of the hacking period, we had a working prototype and software. Both of these factors were able to integrate properly. The AI assistant, Sierra, can accurately recognize faces as well as detect settings in the real world. Although there were challenges along the way, the immense effort we put in paid off.

## What we learned

* How to work with a variety of Google Cloud-based tools and how to overcome potential challenges they pose to beginner users.
* How to connect a smartphone to a laptop with a remote connection to create more opportunities for practical designs and demonstrations.
How to connect a smartphone to a laptop with a remote connection to create more opportunities for practical designs and demonstrations.
How to create docker containers to deploy google cloud-based flask applications to host our dashboard.
How to develop Firebase Cloud Functions to implement cron jobs. We tried to developed a cron job that would send alerts to the user.

## What's next for Saight

### Optimizing the Response Time

Currently, the hardware limitations of our computers create a large delay in the assistant's response times. By improving the efficiency of the models used, we can improve the user experience in fast-paced environments.

### Testing Various Materials for the Mount

The physical prototype of the mount was mainly a proof-of-concept for the idea. In the future, we can conduct research and testing on various materials to find out which ones are most preferred by users. Factors such as density, cost and durability will all play a role in this decision.
## Inspiration

A majority of critical data is streaming data. One such example is the performance of factory machines over time. We wanted to devise a method in order to analyze these machines for potential performance errors and to visualize their performance over time. This would tremendously help cut business costs, as specialized technicians would be able to prevent these performance errors before they occur, in order to save the company resources.

## What it does

Our app is a platform for predictive analysis on real-time machine performance data from Black & Decker. Hence, its primary purpose is to anticipate whether a machine would fail based on the given data. A secondary objective was to build an IoT dashboard to visualize results of exploratory data analysis (e.g. summary statistics).

## How we built it

In order to perform predictive analysis, we implemented an anomaly detection algorithm that takes the temporal nature of the data into consideration. We implemented the system on top of the Flask framework, with a React front-end. In addition, we used Firebase as a central data repository and hosted the app on Google's App Engine. We also used the Twilio API in order to implement text-message alerts.

## Challenges we ran into

The dataset we were given to work with was very small and not representative of the entire problem domain. Hence, we had to design a custom algorithm in order to generate sufficient data. Also, none of us had experience working with time series models or anomaly detection, so we had to learn a lot along the way.

## Accomplishments that we're proud of

We're most proud of the fact that we were able to collaborate independently on different features of the system (e.g. algorithms, back-end, front-end) and still managed to fuse everything together into the final product. Also, we're proud that we were able to tackle such a challenging problem without any obvious solutions within such a short timespan.

## What we learned

We learned a lot about the IoT framework and predictive analysis on streaming data (e.g. online learning, time series, etc.). In addition, we learned about the importance of clean data for both visualization and predictive analysis. Last but not least, we learned that teamwork and collaboration are major keys to tackling challenging and unfamiliar problems.

## What's next for IoT Streaming and Analytics Dashboard

To infinity and beyond!