## signAR - *ASL Translator and Learning Tool*

# Inspiration

The idea from this project came about from a very simple question that we could not think of an answer for: why doesn't Google Translate work for sign language? There are estimated to be [around 70 million deaf people in the world](https://wfdeaf.org/faq/) yet the biggest tool for translation doesn't target this demographic. We felt that translation is not only a means of communication between speakers of different languages but also a way for people to share their cultural background. As such, we decided to tackle the problem of creating a translation engine that could convert ASL (American Sign Language) to English with signAR! (pronounced sigh-ner)

# What it does

**signAR** begins with a live camera view prompting the user to point their device at the hand of the signer they'd like to translate. From there, the app can detect finger lettering for most letters in the alphabet allowing signers to spell out what they wish to communicate. When a letter is detected, **signAR** automatically adds it to the in-progress phrase as well as displaying the sign & letter in augmented reality for confirmation.
*(We definitely wish we could have implemented gesture recognition as gestures are at the heart of understanding and using ASL; however, technical and time constraints only allowed for a spelling translator)*
After the user has received the signer's message, it gets saved to an archive of all the previous phrases they have saved from other interactions with signers. The user can even go back to any given phrase they translated and see the corresponding sign for each letter so that they can practice ASL on their own time!

# How we built it

We built this app in Swift 4 using such frameworks as ARKit, CoreML, and Vision (Apple's computer vision). Additionally, we used Microsoft's Custom Vision to build a model around training data that we compiled ourselves.

# Challenges we ran into

One of our biggest challenges was building a model that could effectively distinguish between each letter of the ASL alphabet (which proved very difficult seeing how similar many of the letters are. Six letters are a variation on a fist!). After much trial and error with datasets and images we found on the internet, we decided that it would be best if we made our own data set by taking over 6,000 of our hands under different lighting conditions signing almost every ASL letter. We took some ideas from the workshop on neural networks to further improve our model. Even after all this, our model still isn't as accurate as we would like it to be (The six variations on a fist are truly challenging for our algorithm).

# Accomplishments that we're proud of

We're very proud of the fact that we can detect some letters at all as when we initially started experimenting with CoreML and Microsoft's Custom Vision, the potential for our idea to work seemed very bleek. Though after much trial and error, we were able to build **signAR**, which was a big achievement for the two of us particularly because we are both new to Machine Learning and Augmented Reality.

# What we learned

Coming out of this hackathon, we are leaving with a much stronger understanding of how a neural network functions as well as the reasons why you can't just give an algorithm a bunch of arbitrary pictures and expect it to immediately know ASL!

# What's next for signAR?

The most pertinent feature that we want to implement is gesture recognition as you cannot do ASL without gestures. Additionally, we think that there is a lot of potential for our project to be able to translate between different kinds of sign languages (like translating from American Sign Language to Spanish Sign Language for example).
## Inspiration

Video game background music and a need to have hands off ambient entertainment.

## What it does

Will play ambient instrumental music depending on your location and weather. Meant to be a easy simple to use app to set the mood of your real life environment so you can enjoy it better.

## How I built it

React & Dark Sky API. Spotify Integration Backend almost complete.

## Challenges I ran into

## Accomplishments that I'm proud of

## What I learned

## What's next for aumbiance

Spotify Integration, more customizability
## Inspiration

In school, we were given the offer to take a dual enrollment class called Sign Language. A whole class for the subject can be quite time consuming for most children including adults. If people are interested in learning ASL, they either watch Youtube videos which are not interactive or spend HUNDREDS of dollars in classes (<https://deafchildren.org> requiring $70-100). Our product provides a cost-effective, time-efficient, and fun experience when learning the new unique language.

## What it does

Of course you have to first learn the ASL alphabets. A, B, C, D ... Z. Each alphabet has a unique hand gesture. You also have the option to learn phrases like "Yes", "No", "Bored", etc. The app makes sure you have done the alphabet correctly by displaying a circular progress view on how long you have to hold the gesture. We provide many images to make the learning experience accessible. After learning all the alphabets and practicing a few words, time for GAME time :). Test your ability to show a gesture and see how long you can go until you give up. The gamified experience leads to more learning and engaging for children.

## How we built it

The product was built using the language Swift. The hand-tracking was done using CoreML Components. We used hand landmarks and found distances between all points of the hand. Comparing the distances it SHOULD be and what it is as a specific time frame helps us figure out whether the hand pose is occurring. For the UI we planned it out using Figma and later wrote the code in Swift. We used the SwiftUI components to save time. For data storing we used UIData which syncs across devices with the same iCloud account.

## Challenges we ran into

There are 26 alphabets. That's a lot of arrays, comparing statements, and repetitive work. Testing would sometimes become difficult because the iPhone would eventually become hot and get temperature notifications. We only had one phone to test, so phone testing was frequently used for hand landmarks mostly. The project was extremely lengthy and putting so much content in one 36 hours is difficult, so we had to risk sleep. A cockroach in the room.

## Accomplishments that we're proud of

The hand landmark detection for an alphabet actually works much better than expected. Moving your hand super fast does not glitch the system. A fully functional vision app with clean UI makes the experience fun and open for all people.

## What we learned

Quantity < Quality. We created more than 6 functioning pages with different level of UI quality. It's very noticeable which views were created quickly because of the time crunch. Instead of having so many pages, decreasing the number of pages and maybe adding more content into each View would make the app appear flawless. Comparing arrays of the goal array and current time-frame array is TEDIOUS. So much time is wasted from testing. We could not figure out action classifier in Swift as there was no basic open-source code. Explaining problems to Chat GPT becomes difficult because the LLM never seems to understand basic tasks, but perfectly performs in complex tasks. Stack Overflow will still be around (for now) if we face problems.

## What's next for Hands-On

The app fits well on my iPhone 11, but on an iPad? I do not think so. The next step to take the project further is to scale UI, so it works for iPads an iPhones of any size. Once we fix that problem, we could release the app to the App Store. Since we do not use any API, we would have no expenses related to hosting the API. Making the app public could help people of all ages learn a new language in an interactive manner.