## Inspiration

Research from Press Ganey Associates found that the average ER wait time in 2009 was 6 hours, which is plenty of time for a patient's condition to worsen significantly before they are seen. Despite best efforts, it is difficult for staff to monitor all patients waiting be seen, which puts them at risk.

## Solution

We had the idea for a system that could visually monitor the vital signs of multiple people in the waiting room and alert nurses to any abnormalities. Currently, we have used a Kinect sensor to accurately monitor the breathing of multiple people simultaneously, and send an alert if the breathing rates are outside of normal ranges. In the future, this sensor could easily be extended to monitor temperature and heart rate, which can be done using a camera as shown in Gambi, Ennio et al.[1].

## Sources

1. <https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5579477/>
## Inspiration

After spending some time in the hospital last year, I quickly became aware of just how isolating and scary the experience can be. And after hearing one of my good friend’s stories of her own experience being hospitalized, I was sure that I could use our experiences to help other patients, and other people all around the world. My friend found herself in an Italian hospital, oceans away from home in a country whose language she didn't speak. This is an unfortunate reality that many face, though applications such as Google Translate can help to bridge the gaps in communication that can occur during these tense moments. However, without any Google Translate for sign language, as well as a 2022 *Reuters Health* article which reported that nearly one-third of hospitals in the US don't offer any sign language interpreters at all, I knew that this was likely an experience which was all too familiar for those who rely on signing to communicate. From there, my project began, and just as quickly as it did, it grew and became something that can apply to many situations outside of healthcare as well.

## How it's Built & What it Does

My project acts a sort of sign language translator in which the user can sign into a camera, have their actions and hand gestures analyzed, and then the text equivalent of their signs will be displayed on the screen. Currently, my application supports American Sign Language (ASL) to English text translations for static letter signs. I built my project by first creating a machine learning model to recognize American Sign Language (ASL) letters. I trained the model using the Sign Language MNIST dataset in Python and PyTorch. Once the model was trained, I converted it to ONNX format to make it compatible with JavaScript for web deployment. On the front end, I used HTML, CSS, and JavaScript to create a simple user interface. The web app captures hand gestures using a webcam, processes the images, and then translates the detected signs into text displayed on the screen.

## Challenges I Faced

I ran into many challenges while bringing my project ideas to life. The first major obstacle that put a pause on my project development was the time requirement for training my model using the Sign Language MNIST dataset. My old MacBook, with its slower CPU, significantly increased the training time, forcing me to stop the process after only 20 epochs. This incomplete training caused some performance issues, leading to a higher rate of errors than expected, as the model had not fully parsed and analyzed the dataset. Another challenge was transitioning from Python to JavaScript when integrating the front end of the application. Initially, I had trained and developed my model in PyTorch, which produced a .pt model. However, for web deployment, I had to convert this model to an ONNX format compatible with JavaScript. This process introduced additional complexity, as I faced issues with mismatched input shapes and difficulty loading the model correctly in the browser. The switch from Python to JavaScript for model inference posed significant challenges, requiring me to rethink how I handled model inputs and outputs, particularly with the ONNX runtime.

## What I Learned

Throughout the making of this project, I learned a lot about working with machine learning models and their deployment. Having never worked with any sort of machine learning before, one of the biggest lessons for me was learning how to bridge the gap between different technologies — from training a model in Python using PyTorch to converting it for use in a JavaScript-based web app. I also gained experience in handling the challenges that come with working across both back-end and front-end environments. Additionally, I improved my skills in debugging, especially when integrating complex machine learning models with front-end interfaces.

## What's Next for SimplySign

The future of SimplySign includes the addition of words, phrases, and full sentences, as well as other common sign languages such as British Sign Language (BSL), Spanish Sign Language (SSL), French Sign language (FSL), and more. Additionally, I would like to continue to work on my user interface and improve the overall design of the project, including adding subtitles to display the detected text as well as incorporating text-to-speech functionality. After that, utilizing an application such as Unity to create a mobile app which supports IOS and Android builds would help spread my project and put it in the hands of those who could benefit from it.
## Inspiration

In our current hospital space, 93% nurses surveyed in a study reported they thought hospitals were understaffed. That means without proper care and watch over specific patients with no means of accessing the helpline themselves (i.e those physically unable to move, unaccustomed to communication, or marginalized groups), critical signals that lead to life-or-death decisions may be missed. Observed by one of our own teammates, these patients’ only source of outlet may be vocal outbursts or facial expressions to convey their pain and need for help, which is not necessarily picked up by physical monitors.. Oftentimes, especially in absence of family members, their moans and cries echo unnoticed by nurses in understaffed hospitals – until it is too late

This is where ExpressCare comes in, a “call-button” technology that picks up these neglected cues to alert corresponding staff members, morphing hospitals into a more empathetic place.

## What it does

Let’s consider a low-income, non-native english speaker that is admitted to an overstaffed hospital without family members. The language barriers alone make a big threat to communication, which is then amplified if the patient is physically constrained to the bed. ExpressCare monitors patients in these states and picks up distress queues to alert nurses if an overall distress index surpasses our threshold. ExpressCare “call-button” device then displays an alert with pain level index and duration on the nurses’ side, informing about a call for help.

Our team considered varying needs in hospital spaces, including patients that need psychological support over immediate medical attention. Therefore, an elderly patient admitted for dementia that is exhibiting queues of confusion as the primary emotion may trigger the system as well. Our device monitors this and sends a milder message to corresponding staff members, including hospital volunteers of psychologists that may better address these circumstances.

Nurses are able to monitor their patients during normal circumstances as well, as ExpressCare offers a display of levels of discomfort throughout the day.

## How we built it

Express Care uses Hume API to configure facial patterns and vocal outbursts to pick up varying emotions. We used batch API and streaming API to process these configurations side-by-side, while isolating the specific negative emotions that resemble discomfort. A distress score is then calculated, which can include anger, pain, confusion, or others as underlying triggers.

## Challenges we ran into

Getting the API to process audio and video was the primary challenge. We tried using PyAudio to store audio files and processing them in time intervals alongside video frames, and to get the technology to monitor vocal outbursts and facial expressions at the same time. However, we were unable to make it run on our local devices due to a PyAudio installation error, so decided to pare down the features we wanted to include in this iteration.

## Accomplishments that we're proud of

This is an incredibly important need that we've identified in the healthcare space, and are proud of the contribution we've made to this area.

## What we learned

We learned a lot throughout this project that will be useful to us in the future--both technically and market knowledge of AI, and APIs as a product.

## What's next for ExpressCall

In the future, we would like to test this with real patients in a SF public hospital to measure the difference in length of distress period with and without ExpressCall, as well as to see if patients need anything else from this service.

Additionally, we would like to expand its use case to an iteration where we measure loneliness, sadness, and boredom for the long-term stay patients to see if they need psychological intervention.