## Inspiration

The ability to easily communicate with others is something that most of take for granted in our everyday life. However, for the millions of hearing impaired and deaf people all around the world, communicating their wants and needs is a battle they have to go through every day. The desire to make the world a more accessible place by bringing ASL to the general public in a fun and engaging manner was the motivation behind our app.

## What it does

Our app is essentially an education platform for ASL that is designed to also be fun and engaging. We provide lessons for basic ASL such as the alphabet, with plans to introduce more lessons in the future. What differentiates our app and makes it engaging, is that users can practice their ASL skills right in the app, with any new letter or word they learn, the app uses their webcam along with AI to instantly tell users when they are making the correct sign. The app also has a skills game that puts what they learnt to the test, in a time trial, that allows users to earn score for every signed letter/word. There is also a leaderboard so that users can compete globally and with friends.

## How we built it

Our app is a React app that we built with different libraries such as MUI, React Icons, Router, React-Webcam, and most importantly Fingerpose along with TensorflowJS for all our AI capabilities to recognize sign language gestures in the browser.

## Challenges we ran into

Our main struggle within this app was implementing Tensorflowjs as none of us have experience with this library prior to this event. Recognizing gestures in the browser in real time initially came with a lot of lag that led to a bad user experience, and so it took a lot of configuring and debugging in order to get a much more seamless experience.

## Accomplishments that we're proud of

As a team we were initially building another application with a similar theme that involved hardware components and we had to pivot quite late due to some unforseen complications, and so we're proud of being able to turn around with such a short amount of time and make a good product that we would be proud to show anyone. We're also proud of building a project that also has a real world usage to it that we all feel strongly about and that we think really does require a solution for.

## What we learned

Through this experience we all learned more about React as a framework, in addition to real time AI with Tensorflowjs.

## What's next for Battle Sign Language

Battle Sign Language has many more features that we would look to provide in the future, we currently have limited lessons, and our gestures are limited to the alphabet, so in the future we would increase our app to include more complex ASL such as words or sentences. We would also look forward to adding multiplayer games so that people can have fun learning and competing with friends simultaneously.
## 💭 Motivation

Medical images play a crucial role in training artificial intelligence (AI) algorithms for various diagnostic tasks. However, limited access to large-scale datasets poses a challenge in developing accurate and robust models. Recent advances in latent diffusion models have demonstrated the generation of novel, realistic images using text prompts. Generating high-fidelity medical images could potentially solve the medical data gap. We present an approach for generating synthetic chest X-rays (CXRs) with domain adaptation of Stable Diffusion (SD), a state-of-the-art latent diffusion model for text-to-image generation. We also present a method for removal of text and artifacts from the resulting synthetic images, and evaluated images with two expert radiologists.

## 🔨 Methods

To generate novel CXRs, we fine-tuned the SD v2-1-base model for 10 epochs on an Intel® Max Series GPU, using low-rank adaptation. We used a dataset of 32,532 CXRs paired with clinical reports, a proprietary dataset obtained in a research partnership with Segmed. We then performed GPU inference and generated 1,000 synthetic CXRs using classifier-free guidance of 4, and 75 inference steps.

We implemented a text removal step on images, to eliminate textual artifacts on the generated images. This pipeline leverages text segmentation using Meta’s Segment Anything Model, and inpainting with Stability.AI’s Stable Diffusion 2. The inference is run on an Intel® Max Series GPU and can be executed through a Gradio web application.

## 📈 Evaluation and Results

1. The synthetic images showed quantitative concordance with real CXRs. The Fréchet Inception Distance between the synthetic and real dataset was 68.4, calculated using InceptionV3 pretrained on ImageNet.
2. A random sample of 50 generated CXRs and 50 real CXRs was shuffled, given to a board-certified radiologist with >10 years of clinical experience, and labeled as either "synthetic" or "real". The reader had an accuracy of 0.81, sensitivity 0.86, and specificity 0.76. **Notably, Reader 1 misidentified 7 out of 50 synthetic images as real images.**
3. A random sample of 100 generated CXRs was examined for overall quality (anatomy, style, artifacts) by the same radiologist expert. Some feedback, both positive and negative, is included below in reference to the synthetic X-rays:

>
> A few [synthetic images] stand out due to their sharpness and defined details, making them more closely resemble real X-rays.
>
>
> The rib cage and heart have a "box-y" shape. However, the lung parenchyma looks good.
>
>
> Looks good, except for the ECG marker with a weird position (also the only one).
>
>
> Costo-phrenic angles too deep, esp on the left.
>
>
>

**In total, there were 36 positively scoring samples that were deemed by the radiologist to be comparable to real X-rays, in both anatomy and appearance.** Of the remainder, many were labeled as moderate quality, with the only identified downside being the presence of text artifacts. In the majority of these cases, the AI-enabled text removal is able to rectify this, creating a clean synthetic CXR.

## 🏆 Conclusion

Our pipeline is able to generate synthetic CXRs that resemble real CXRs, with further improvement in performance to be expected as we scale up the training time and data. The proposed method holds promise for augmenting limited medical image datasets, facilitating the development of robust ML algorithms in chest radiography. The ability to generate realistic synthetic CXRs has significant implications for the field of medical imaging. By augmenting limited datasets, our approach can improve training ML algorithms for diagnostic tasks, holding tremendous potential for advancing healthcare outcomes.
## Inspiration

In school, we were given the offer to take a dual enrollment class called Sign Language. A whole class for the subject can be quite time consuming for most children including adults. If people are interested in learning ASL, they either watch Youtube videos which are not interactive or spend HUNDREDS of dollars in classes (<https://deafchildren.org> requiring $70-100). Our product provides a cost-effective, time-efficient, and fun experience when learning the new unique language.

## What it does

Of course you have to first learn the ASL alphabets. A, B, C, D ... Z. Each alphabet has a unique hand gesture. You also have the option to learn phrases like "Yes", "No", "Bored", etc. The app makes sure you have done the alphabet correctly by displaying a circular progress view on how long you have to hold the gesture. We provide many images to make the learning experience accessible. After learning all the alphabets and practicing a few words, time for GAME time :). Test your ability to show a gesture and see how long you can go until you give up. The gamified experience leads to more learning and engaging for children.

## How we built it

The product was built using the language Swift. The hand-tracking was done using CoreML Components. We used hand landmarks and found distances between all points of the hand. Comparing the distances it SHOULD be and what it is as a specific time frame helps us figure out whether the hand pose is occurring. For the UI we planned it out using Figma and later wrote the code in Swift. We used the SwiftUI components to save time. For data storing we used UIData which syncs across devices with the same iCloud account.

## Challenges we ran into

There are 26 alphabets. That's a lot of arrays, comparing statements, and repetitive work. Testing would sometimes become difficult because the iPhone would eventually become hot and get temperature notifications. We only had one phone to test, so phone testing was frequently used for hand landmarks mostly. The project was extremely lengthy and putting so much content in one 36 hours is difficult, so we had to risk sleep. A cockroach in the room.

## Accomplishments that we're proud of

The hand landmark detection for an alphabet actually works much better than expected. Moving your hand super fast does not glitch the system. A fully functional vision app with clean UI makes the experience fun and open for all people.

## What we learned

Quantity < Quality. We created more than 6 functioning pages with different level of UI quality. It's very noticeable which views were created quickly because of the time crunch. Instead of having so many pages, decreasing the number of pages and maybe adding more content into each View would make the app appear flawless. Comparing arrays of the goal array and current time-frame array is TEDIOUS. So much time is wasted from testing. We could not figure out action classifier in Swift as there was no basic open-source code. Explaining problems to Chat GPT becomes difficult because the LLM never seems to understand basic tasks, but perfectly performs in complex tasks. Stack Overflow will still be around (for now) if we face problems.

## What's next for Hands-On

The app fits well on my iPhone 11, but on an iPad? I do not think so. The next step to take the project further is to scale UI, so it works for iPads an iPhones of any size. Once we fix that problem, we could release the app to the App Store. Since we do not use any API, we would have no expenses related to hosting the API. Making the app public could help people of all ages learn a new language in an interactive manner.