## Inspiration

Public speaking is an incredibly important skill that many seek but few master. This is in part due to the high level of individualized attention and feedback needed to improve when practicing. Therefore, we want to solve this with AI! We have created a VR application that allows you to get constructive feedback as you present, debate, or perform by analyzing your arguments and speaking patterns.
While this was our starting motivation for ArticuLab, we quickly noticed the expansive applications and social impact opportunities for it. ArticuLab could be used by people suffering from social anxiety to help improve their confidence in speaking in front of crowds and responding to contrasting opinions. It could also be used by people trying to become more fluent in a language, since it corrects pronunciation and word choice.

## What it does

ArticuLab uses AI in a VR environment to recommend changes to your pace, argument structure, clarity, and boy language when speaking. It holds the key to individualized public speaking practice. In ArticuLab you also have the opportunity to debate directly against AI, who'll point out all the flaws in your arguments and make counterarguments so you can make your defense rock-solid.

## How we built it

For our prototype, we used Meta's Wit.AI natural language processing software for speech recognition, built a VR environment on Unity, and used OpenAI's powerful ChatGPT to base our feedback system on argument construction and presenting ability. Embedding this into an integrated VR App results in a seamless, consumer-ready experience.

## Challenges we ran into

The biggest challenge we ran into is using the VR headset microphone as input for the speech recognition software, and then directly inputting that to our AI system. What made this so difficult was adapting the formatting from each API onto the next. Within the same thread, we ran into an issue where the microphone input would only last for a few seconds, limiting the dialogue between the user and the AI in a debate. These issues were also difficult to test because of the loud environment we were working in.
Additionally, we had to create a VR environment from scratch, since there were no free assets to fit our needs.

## Accomplishments that we're proud of

We're especially proud of accomplishing such an ambitious project with a team that is majority beginners! Treehacks is three of our integrants' first hackathon, so everyone had to step up and do more work or learn more new skills to implement in our project.

## What we learned

We learned a lot about speech to text software, designing an environment and programming in Unity, adapting the powerful ChatGPT to our needs, and integrating a full-stack VR application.

## What's next for ArticuLab

Naturally, there would be lots more polishing of the cosmetics and user interface of the program, which are currently restricted by financial resources and the time available. Among these, would be making the environment a higher definition with better quality assets, crowd responses, ChatGPT responses with ChatGPT plus, etc.
ArticuLab could be useful both academically and professionally in a variety of fields, education, project pitches like Treehacks, company meetings, event organizers… the list goes on! We would also seek to expand the project to alternate versions adapted for the comfort of the users, for example, a simplified iOS version could be used by public speakers to keep notes on their speech and let them know if they're speaking too fast, too slow, or articulating correctly live! Similarly, such a feature would be integrated into the VR version, so a presenter could have notes on their podium and media to present behind them (powerpoint, video, etc.), simulating an even more realistic presenting experience.
Another idea is adding a multiplayer version that would exponentially expand the uses for ArticuLab. Our program could allow debate teams to practice live in front of a mix of AI and real crowds, similarly, ArticuLab could host online live debates between public figures and politicians in the VR environment.
## Inspiration

Partially inspired by the Smart Cities track, we wanted our app to have the direct utility of ordering food, while still being fun to interact with. We aimed to combine convenience with entertainment, making the experience more enjoyable than your typical drive-through order.

## What it does

You interact using only your voice. The app automatically detects when you start and stop talking, uses AI to transcribe what you say, figures out the food items (with modifications) you want to order, and adds them to your current order. It even handles details like size and flavor preferences. The AI then generates text-to-speech audio, which is played back to confirm your order in a humorous, engaging way. There is absolutely zero set-up or management necessary, as the program will completely ignore all background noises and conversation. Even then, it will still take your order with staggering precision.

## How we built it

The frontend of the app is built with React and TypeScript, while the backend uses Flask and Python. We containerized the app using Docker and deployed it using Defang. The design of the menu is also done in Canva with a dash of Harvard colors.

## Challenges we ran into

One major challenge was getting the different parts of the app—frontend, backend, and AI—to communicate effectively. From media file conversions to AI prompt engineering, we worked through each of the problems together. We struggled particularly with maintaining smooth communication once the app was deployed. Additionally, fine-tuning the AI to accurately extract order information from voice inputs while keeping the interaction natural was a big hurdle.

## Accomplishments that we're proud of

We're proud of building a fully functioning product that successfully integrates all the features we envisioned. We also managed to deploy the app, which was a huge achievement given the complexity of the project. Completing our initial feature set within the hackathon timeframe was a key success for us. Trying to work with Python data type was difficult to manage, and we were proud to navigate around that. We are also extremely proud to meet a bunch of new people and tackle new challenges that we were not previously comfortable with.

## What we learned

We honed our skills in React, TypeScript, Flask, and Python, especially in how to make these technologies work together. We also learned how to containerize and deploy applications using Docker and Docker Compose, as well as how to use Defang for cloud deployment.

## What's next for Harvard Burger

Moving forward, we want to add a business-facing interface, where restaurant staff would be able to view and fulfill customer orders. There will also be individual kiosk devices to handle order inputs. These features would allow *Harvard Burger* to move from a demo to a fully functional app that restaurants could actually use. Lastly, we can sell the product by designing marketing strategies for fast food chains.
## Motivation:

Difficult conversations can be tough for several reasons, including fear of conflict or rejection, lack of confidence, emotional investment, power dynamics, cultural norms, and lack of skills. These conversations often involve discussing sensitive or personal topics, which can be emotionally challenging for both parties. Some examples include talking to new people, saying No to friends, talking about your feelings on your first date or negotiating your salary with a HR. Most often we are not prepared for these scenarios, but practicing these scenarios before we actually face them can reduce social anxiety and build confidence in a lot of people. Research also suggests that around 63% people find it hard to start a conversation around sensitive topic and many people also suffer with social anxiety and isolation due to lack of these communication skills .

## Our Solution

We want to build an AI-based application that acts as a speech buddy and enables users to navigate through these difficult conversations. Our tool is powered with Language, Speech and vision technologies to closely resemble the real world setting to practice a difficult conversation.

**Core Features include:**

* **Scenario-specific conversations**: We fine-tuned the GPT models through prompt-engineering to adapt to the customized scenarios that the users want to practice. For instance, if the user wants to improve on negotiation skills, the bot would act like a HR and if the practice scenario is a first date, then the bot would talk like your romantic interest on a dinner date with you.
* **Speech and Text Chatting**: Our AI tool can converse with the users through both text and speech. This provides the accessibility to users to talk through multiple modalities.
* **Humanized AI-Avatar**: We have an avatar integrated into our application that can basically take the image of the person you want to have a conversation with and is animated in accordance to the speech generated by the AI chatbot. Overall, the AI Avatar can serve as a proxy for the real human to practice real-life conversation.
* **Personalized Feedback**: Along with the generic speech use-case, our intelligent AI tool will also provide the relevant feedback based on the textual content. The feedback tried to quantify metrics like confidence and also suggestions on the phrases that could be potentially replaced/ avoided.

## How did we build it

We used a sophisticated tech stack to build this product and handle different use-cases. We developed a web-based application with front-end built on React JS and backend database with Convex. Our backend Convex server is also connected with the Flask which interacts with the OpenAI API to get the responses from the GPT, Azure model for the Text to Speech service and a Wav2Lip model hosted on Modal to support the AI avatar animation.

## Challenges Faced:

We faced technical challenges as we were trying to get the credits to host the models on different services like OpenAI and Modal. Our major difficulty was also to come up with an end-to-end pipeline with minimal latency in the conversation and have a seamless user experience. With all of us being very new to web development, we also found creating a complex chat application with language, video and audio to be a test to our technical abilities.

## What's Next?

This tool has a lot of potential to disrupt the education and healthcare industry. We use the cutting edge technology of LLMs and Deep Learning models to provide a seamless and human-like conversation experience to the users. We believe that this could be a great user product that can be a personal coach and buddy to have conversations and learn from them. We want to work further on this to develop more features, conduct user research and release a completely working app to users to test different hypotheses. We believe that the app can work well as a consumer product that could be released as a freemium version.