## Inspiration

Although they may be a quiet minority, people who are either born with-- or corrected out of-- speech sound impediments have stories of their own. Often spending much of their childhoods at speech therapists' offices due to these "disorders" ranging from lisps or stuttering to physical limitations of the vocal tract often, affected children find themselves confused and conscious about what makes them different from their peers . With SuperSpeech, kids in these tender, formative years have the opportunity to normalize their speech in an environment and pace curated for them to control and enjoy. In fact, through endlessly fun Marvel universe quotes and an engaging, interactive UI, the transition through child speech therapy can be removed almost entirely from the mysterious world of hospitals and doctors' visits to a universally familiar, lighthearted phone game.

## What it does

Super Speech provides loads of entertainments for kids to practice their sentence articulation by reading aloud different Marvel quotes, at which point we used the Houndify API for speech-to-text recognition and phonetic pattern matching. A user will choose their favorite Marvel character, and we apply the Dale-Chell readability algorithm to have the app choose more quotes based on suitable difficulty for them to try saying out loud again, and give them points accordingly. Each quote is presented in the form of a responsive flashcard, and a picture of the character who said the quote pops up if they can most closely verbalize the words on the flashcard.

## How we built it

We used the Houndify API and the Marvel API. The front end of the iOS app is implemented in Swift with a generous spoonful of CocaoPods, and the backend is in NodeJS. We transfer post and get requests from an online web server that we set up which uses the Google Cloud Platform. The backend communicates with the Marvel API to get information about various characters and sends it over to the app for display to the user.

The iOS app uses SwipeViews to create an aesthetically appealing flashcard-like interface.

## Challenges we ran into

We should have been more organized in our work process, seeing that we lost many hours trying to merge our branches to master at the same time (conflicts galore). We also were pretty new to iOS development, so getting all the APIs to work could become a struggle.

## Accomplishments that we're proud of

We are super proud of how much we learned in 48 hours! We used CocoaPods, different APIs, and Sprite-Kit for the first time to get the best approximation to our vision of the final product, an intuitive and kid-friendly interface with a bit of a wow-factor. We feel passionate about our project and the potential social impact it could have on kids across the country.

## What we learned

We were really put to the test about how iOS development works, juggling the tradeoffs inherent int he process. We learned how to integrate APIs and use CocoaPods, animate UI elements to our specifications, and get hacky with swiftyJSON, Alamofire, Houndify, Moya, CheerioJS and the Marvel API.

While coding our backend, we learned tons about web scraping and different Marvel APIs available.

And of course, a very memorable education in Git.

## What's next for Super Speech

We want to implement a "Skip" feature, wherein kids are allowed to skip up to 5 quotes if they find them too difficult (and thereby collect data on how we can improve upon the Dale-Chell algorithm).
We also want to add quotes from other Disney properties to appeal to a wider audience of kids. It would be cool to use machine learning to give each user a unique experience, most suited to their learning curve/style. We would also like to flesh out the statistics we provide about each of their games, implementing a scoreboard and graphically showing their progress through different levels, partly for the parents' own benefit and ability to learn about their child's growth. Also, an option for non-English languages!
## Inspiration

Nearly 10% of all children experience speech-related disorders. But speech therapy can cost more than $31,000 each year. What if there was a simple, accessible, and affordable alternative to practicing speech/pronunciation?

## What it does

SpeechMe provides an intuitive way to practice speech without spending $100 ~ $150 each year on speech therapists. It is an iOS app that allows the user to input a word to practice along with a recording of their pronunciation using our user-friendly interface, then gives feedback on the input audio using our backend algorithm.

## How we built it

Our program architecture consists of three main components:

1. Swift Frontend
2. Python Backend API
3. Python Backend Analyzer

The usual flow of a use case is as follows:

The Swift frontend prompts the user for an input word that the user wants to practice, then allows them to record an audio file. The frontend takes these inputs and sends them to our backend API in a POST request as multi-part form data.

Our Flask API receives this data and sends the audio file to AssemblyAI’s audio-to-text API. Once it receives the text version of the input audio, our backend analyzer takes in the two strings (user input word and text-converted version of the input audio) and runs the Jaro-Winkler algorithm. This algorithm returns a similarity score between 0 and 1, which we scale into an integer from 0 ~ 100 to be then returned to the frontend.

Lastly, the frontend takes the return value from the POST request and displays it to the user as the score.

## Challenges we ran into

Throughout this hackathon, we ran into numerous challenges. With all of us being first-time hackers and not having taken any web-dev courses, our frontend team had never touched iOS development before, and our backend team had no experience in APIs at all. Nevertheless, we decided to take on the challenge!

One of our largest challenges was figuring out how to combine the SwiftUI frontend and the Python backend. Especially with the data transfer our program required from the frontend to the backend, then back to the frontend, we figured we needed to build our own API. Even so, we struggled even more with constructing POST requests to our API. Initially, we attempted to encode the audio file into a string and send JSON data with the input text and the input audio (as a binary string). However, the encoded audio file turned out to be too long, so we had to try to use multi-part form data instead. There was no easy way, however, to construct our data in the correct format for multi-part form data, and unfortunately, we were not able to successfully implement this POST request. Nevertheless, assuming successful integration, our program works as one would expect, and we were able to reach a point much further than we had imagined!

## Accomplishments that we're proud of

We are very proud of our beautiful and clean user interface. It is both intuitive and simple to use, so it can be used both by children and adults. Especially given that this was our frontend team’s first time ever touching iOS development and SwiftUI, we are very proud of what we’ve accomplished!

We are also proud of creating our very own API for the first time, integrated smoothly with an external API (by AssemblyAI) and a backend that perfectly runs our algorithm on the string returned from the AssemblyAI API. The moment 200 OK showed on our screen was definitely one of our most memorable moments of this hackathon.

## What we learned

We learned the most we could have ever learned in the span of two days. First, we learned how to use Swift, build an API, use an external API, and the integration process of frontend and backend. On the other hand, we also learned that when we aim high for a project, we must also be ready to pivot or take a slightly different approach at any point throughout the development process. We remained stuck on trying to get our ideal architecture to work, and it perhaps would have helped to think of other ways to make our product come alive.

## What's next for SpeechMe

Our future plans are as follows!

1. **Successful backend/frontend integration**: We would love to see our app successfully working with our backend! It would be cool to finally figure out how to organize multi-part form data to send as a POST request.
2. **Personalization**: We would love to implement user personalization! We can have a login feature that saves starred words so that users can go back to saved words later on to practice them more.
3. **Text to Speech**: A text-to-speech feature that plays back what the word should sound like would be a great addition, too.
## Inspiration

There are millions of people around the world who have a physical or learning disability which makes creating visual presentations extremely difficult. They may be visually impaired, suffer from ADHD or have disabilities like Parkinsons. For these people, being unable to create presentations isn’t just a hassle. It’s a barrier to learning, a reason for feeling left out, or a career disadvantage in the workplace. That’s why we created **Pitch.ai.**

## What it does

Pitch.ai is a web app which creates visual presentations for you as you present. Once you open the web app, just start talking! Pitch.ai will listen to what you say and in real-time and generate a slide deck based on the content of your speech, just as if you had a slideshow prepared in advance.

## How we built it

We used a **React** client combined with a **Flask** server to make our API calls. To continuously listen for audio to convert to text, we used a react library called “react-speech-recognition”. Then, we designed an algorithm to detect pauses in the speech in order to separate sentences, which would be sent to the Flask server.

The Flask server would then use multithreading in order to make several API calls simultaneously. Firstly, the **Monkeylearn** API is used to find the most relevant keyword in the sentence. Then, the keyword is sent to **SerpAPI** in order to find an image to add to the presentation. At the same time, an API call is sent to OpenAPI’s GPT-3 in order to generate a caption to put on the slide. The caption, keyword and image of a single slide deck are all combined into an object to be sent back to the client.

## Challenges we ran into

* Learning how to make dynamic websites
* Optimizing audio processing time
* Increasing efficiency of server

## Accomplishments that we're proud of

* Made an aesthetic user interface
* Distributing work efficiently
* Good organization and integration of many APIs

## What we learned

* Multithreading
* How to use continuous audio input
* How to use React hooks, Animations, Figma

## What's next for Pitch.ai

* Faster and more accurate picture, keyword and caption generation
* "Presentation mode”
* Integrate a database to save your generated presentation
* Customizable templates for slide structure, color, etc.
* Build our own web scraping API to find images