## Inspiration

Our inspiration for this project stemmed from discovering unique methods of input management, especially with the advent of tracking software. Being inspired by a story of an old HTN competitor, who used a unique method to have users interact with their virtual world differently, we thought of a way to have seamless inputs through hand gestures, providing many different combinations of inputs using a tool that most people already use everyday.

## What it does

The model places digital nodes on the major joints on your hands and provides their co-ordinates relative to the screen. Using these points, an overall action/gesture is determined by using ratios and differences to determine which fingers are open or closed. And with different patterns of open or closed fingers, the program determines what gesture each player is making.

## How we built it

We used mediapipe and opencv for the machine learning model in order to track hands The interpretation of gestures was custom, deduced by mathematically deducing the distances and ratios between points on the hand. We used aseperite for assets and godot for the assembly of the games and tracking software.

## Challenges we ran into

A major challenge we ran into was that of converting a .py file to a .exe file. Despite the simplicity of the process, the dependencies led to 2 hours of errors and failures. We ran into many unexpected errors which prevented us from porting our finished tracking software into Godot, the game making engine we used. In order to circumvent the challenge we decided to take a parallel approach where the mediapipe program detected and deduced the hand gestures, placing the data in a separate file that was being read by the engine in real-time. Despite the lengthy and technical process that took the input of the camera and transformed into the actions of the characters on-screen, the program had barely noticeable latency, allowing for accurate gameplay.

## Accomplishments that we're proud of

Being able to successfully use the model and put it to use practically through critical and creative thinking was a very gratifying experience, in addition to learning more about game design and creation through creating our own assets and putting everything together.

## What we learned

We learned the basics of using ML models in conjunction with libraries like opencv for reading and writing data.

## What's next for Pipedream

In the future, we may explore more path for models not only restricted to hand movement but potentially face and full body as well. It would also be interesting to explore practical applications of this technology for not only entertainment but other avenues as well such as daily convenience and services.
## Inspiration

Our group's initial inspiration stemmed from the monotonous task of continuously searching for complex LaTeX commands online. We could easily write down any of the mathematical equations on paper with ease and in a timely manner. This year, during our machine learning class, we finally solved our problem. The course's knowledge and tools allowed us to swiftly convert drawn mathematical equations to LaTeX code.

## What it does

The project takes input from a user via a canvas that they can draw mathematical functions on, which is then converted to latex code and the respective rendered latex text through a generative large language model.

## How we built it

Firstly, we constructed the canvas for the user to draw and scribble down any desired equation to be converted. Next, we needed the canvas and the generative LLM to communicate somehow. We recognized that the canvas can be converted into an image and then a base64 string. This string is then passed to an API call which communicates with the LLM in order to generate the LaTeX code. Once the LLM produces the desired response, we parse the output to ensure the validity of the response shown to the user. Finally, the parsed response is the desired result from the sketch provided by the user, which is then rendered to the web application.

## Challenges we ran into

Our group wanted to create our own nerual network for scribbletex, training, validating, and testing it. However, with the two day time frame, we believed that this was too ambitious. We ultimately resorted to using a generative LLM that already existed.

## Accomplishments that we're proud of

We take great pride in completing this project, taking on numerous ambitious tasks collaboratively. Our group provided support and guidance to each other throughout the entire process, resulting in scribbletex, a project we are excited to showcase.

## What we learned

How to utilize a generative LLM and communicate with it via API calls. Some of our group members had no prior experience with the React framework but developed a strong foundation for the framework. Additionally, we learned how to communicate among one another, along with compromising and making progress towards scribbletex.

## What's next for scribbletex

For scribbletex, we would to implement new features, which include a user history to store previously queried mathematical equations and their respective output. Additionally, we would like for users to upload their own images for translation to latex as well!
## Inspiration

We aims to bridge the communication gap between hearing-impaired individuals and those who don't understand sign language.

## What it does

This web app utilizes the webcam to capture hand gestures, recognizes the corresponding sign language symbols using machine learning models, and displays the result on the screen.

### Features

* Real-time hand gesture recognition
* Supports standard ASL
* Intuitive user interface
* Cross-platform compatibility (iOS and Android via web browsers)

## How we built it

We use [Hand Pose Detection Model](https://github.com/tensorflow/tfjs-models/tree/master/hand-pose-detection) and [Fingerpose](https://github.com/andypotato/fingerpose) to detect hand and its correspond gesture from webcam. For frontend, we use ReactJS and Vite as the build tool and serve our website on Netlify. There is no backend since we embed the models on client side.

## Challenges we ran into

### Model Choice

We originally used [google web ML model](https://teachablemachine.withgoogle.com/train/image) to trained with data that contains many noises such as the person and the background. This implies that the prediction will be heavily biased on objects that are not hands. Next, we thought about labeling the image to emphasize on hand. That leads to another bias on some hand gestures are weighted more, so the model tends to predict certain gestures even when we did not pose it that way. Later, we discover to use hand landscape to recognize hand only and joint pose to recognize the gesture which gives much better prediction.

### Streaming

In order to let the webcam worked for all device(mobile and desktop) of different specs and screen ratio, we struggle to find a way to enable full screen on all of them. First solution was to hard code the width and height for one device but that was hard to adjust and limited. During try and error, there's another issue that raised, the width and height is applied horizontally for mobile devices, so to work on both mobile and desktop, we dynamically check the user's screen ratio. To solve full screen issue, we used progressive web app to capture the device window.

## Accomplishments that we're proud of

* Active and accurate hand tracking from webcam streaming
* Finding ways to translate different gestures from ASL to English
* Being able to use across mobile and desktop
* Intuitive yet functional design

## What we learned

* Learned about American Sign Language
* Deploy progressive web app
* How machine learning model takes inputs and make prediction
* How to stream from webcam to inputs to our model
* Variations of machine learning model and how to fine-tune them

## What's next for Signado

Next step, we plan to add two hand and motion based gestures support since many words in sign language require the use of these two properties. Also to improve on model accuracy, we can use the latest [handpose]{<https://blog.tensorflow.org/2021/11/3D-handpose.html>} model that transform the hand into a 3D mesh. This will provide more possibility and variation to the gesture that can be performed.