## Inspiration

We've all been there - racing to find an empty conference room as your meeting is about to start, struggling to hear a teammate who's decided to work from a bustling coffee shop, or continuously muting your mic because of background noise.

As the four of us all conclude our internships this summer, we’ve all experienced these over and over. But what if there is a way for you to simply take meetings in the middle of the office…

## What it does

We like to introduce you to Unmute, your solution to clear and efficient virtual communication. Unmute transforms garbled audio into audible speech by analyzing your lip movements, all while providing real-time captions as a video overlay. This means your colleagues and friends can hear you loud and clear, even when you’re not. Say goodbye to the all too familiar "wait, I think you're muted".

## How we built it

Our team built this application by first designing it on Figma. We built a React-based frontend using TypeScript and Vite for optimal performance. The frontend captures video input from the user's webcam using the MediaRecorder API and sends it to our Flask backend as a WebM file. On the server side, we utilized FFmpeg for video processing, converting the WebM to MP4 for wider compatibility. We then employed Symphonic's API to transcribe visual cues.

## Challenges we ran into

Narrowing an idea was one of the biggest challenges. We had many ideas, including a lip-reading language course, but none of them had a solid use case. It was only after we started thinking about problems we encountered in our daily lives did we find our favorite project idea.

Additionally, there were many challenges on the technical side with using Flask and uploading and processing videos.

## Accomplishments that we're proud of

We are proud that we were able to make this project come to life.

## Next steps

Symphonic currently does not offer websocket functionality, so our vision of making this a real-time virtual meeting extension is not yet realizable. However, when this is possible, we are excited for the improvements this project will bring to meetings of all kinds.
## Inspiration

When attending crowded lectures or tutorials, it's fairly difficult to discern content from other ambient noise. What if a streamlined pipeline existed to isolate and amplify vocal audio while transcribing text from audio, and providing general context in the form of images? Is there any way to use this technology to minimize access to education? These were the questions we asked ourselves when deciding the scope of our project.

## The Stack

**Front-end :** react-native

**Back-end :** python, flask, sqlalchemy, sqlite

**AI + Pipelining Tech :** OpenAI, google-speech-recognition, scipy, asteroid, RNNs, NLP

## What it does

We built a mobile app which allows users to record video and applies an AI-powered audio processing pipeline.

**Primary use case:** Hard of hearing aid which:

1. Isolates + amplifies sound from a recorded video (pre-trained RNN model)
2. Transcribes text from isolated audio (google-speech-recognition)
3. Generates NLP context from transcription (NLP model)
4. Generates an associated image to topic being discussed (OpenAI API)

## How we built it

* Frameworked UI on Figma
* Started building UI using react-native
* Researched models for implementation
* Implemented neural networks and APIs
* Testing, Testing, Testing

## Challenges we ran into

Choosing the optimal model for each processing step required careful planning. Algorithim design was also important as responses had to be sent back to the mobile device as fast as possible to improve app usability.

## Accomplishments that we're proud of + What we learned

* Very high accuracy achieved for transcription, NLP context, and .wav isolation
* Efficient UI development
* Effective use of each tem member's strengths

## What's next for murmr

* Improve AI pipeline processing, modifying algorithms to decreasing computation time
* Include multi-processing to return content faster
* Integrate user-interviews to improve usability + generally focus more on usability
## Inspiration

We wanted to solve a problem that was real for us, something that we could get value out of. We decided upon Vocally as it solves an issue faced by a lot of people during job interviews, presentations, and other occasions which include speaking in a clear and concise manner. The problem was that it takes a long time to record yourself and re-listen to it just to spot any sentence fillers like "um" or "like". We would like to make it easier to display statistics of one's speech.

## What it does

The user clicks the record button and starts speaking. The application first converts speech-to-text using React's built-in speech recognition. After analyzing the results various text processing techniques (e.g. sentiment analysis), it displays feedback.

## How we built it

* First, we needed to see how keywords could be extracted from an audio recording in the back-end. We settled with React's speech-to-text feature.
* Next, we created API endpoints in Flask (a python web framework) for the React app to make requests from.
* Fuzzy string matching, grammatical, and sentiment analysis were used to process and return the stats to the user using data visualization.
* The last task was deployment to the pythonanywhere.com domain for demo testing purposes.

## Challenges I ran into

Using flask as an API was easy, but we initially tried to host it on GCP, which proved to be difficult as our firewall rules were not configured properly. We moved onto pythonanywhere.com for hosting. For the front-end, we first decided to take a look at the Flutter framework to be able to make the application mobile accessible but the framework was introduced in 2018, and there were a lot of configuration issues that needed to be resolved.

## Accomplishments that we are proud of

Getting the sound recorder to work on the front-end took longer than expected, but the end result was very satisfying. We're proud that we actually achieved creating an end-to-end solution.

## What I learned

Exploring different framework options like Flutter, in the beginning, was a journey for us. The API that was created needed to delve deeper into the python programming language. We learned about various syntactical and natural language processing techniques.

## What's next for Vocally

We may re-explore the concept of natural language processing, perhaps build our own algorithm from scratch and do more over a longer time period.