## Inspiration

Videos are made to help people learn another language by injecting the learning into a popular foreign show.

## What it does

Simply provide a video link in a foreign language. LinguaMedia will parse the content, extract meaningful words, generate subtitles and word bubbles, and help you practice newly learned vocabulary in a post-learning session and quiz section!

As the user watches the video, LinguaMedia will parse it in real-time, generating subtitles and popping up keyword chat along the way. Once the video finishes, a review section will display detailed information about each vocabulary word, along with pronunciation audio. This will be followed by a quiz section to reinforce learning.

## How we built it

The app front-end backend is built with Relex.dev
The backend logic/task is modulized and chained with fetch.ai and uagent framework.

* Video transcription is captured using Python script.
* Transcripts are processed using the GROQ model to extract the meaningful vocabulary.
* Translation, Description, Pronunciation text, and Quiz content are generated with the Google Gemini model.
* Pronunciation Audio is generated with Cartesia API

## Challenges we ran into

new framework and technology and documentations to read and learn

## Accomplishments that we're proud of

It works functionally, and the app has components.

## What we learned

Modulize a process to break down big problems.

## What's next for LinguaMedia

Fully link front-end and backend logics for full experience.
## Inspiration

It is known that the best way to learn a language is to immerse yourself in the media of that language. However, watching media is traditionally a passive process, which has been indicated in many studies to have limited effectiveness in knowledge retention. Our goal was to innovate a solution that transforms the act of video watching into an active, engaging process, ultimately enhancing language learning and expediting the acquisition of new languages for our users.

## What it does

linquiztics analyzes the video you are watching and creates a list of questions using AI that evaluates your understanding of the content of the video and provides detailed feedback on how the user can improve their understanding of the content. Not only does it have the functionality of playing audio for learning the pronunciation, it also can interpret the words that the user is saying for input.

## How we built it

Leveraging Taipy for both the frontend and backend development, we seamlessly integrated various APIs to enhance our project's functionality. By tapping into YouTube's API, we efficiently retrieved video transcripts, which were then processed through OpenAI's GPT to dynamically generate quiz questions. Introducing the Whisper API allowed us to incorporate voice input, enabling users to engage in vocal practice during the quiz. Furthermore, we employed gTTS to implement an auditory component, integrating sound playback for an immersive pronunciation learning experience.

## Challenges we ran into

One of our most significant hurdles in the development process was related to Chrome extensions. We encountered compatibility issues when attempting to integrate them with the YouTube and GPT APIs. Faced with this roadblock, we decided to pivot towards using Python, a language we were more familiar with and confident about its seamless integration with APIs. However, the transition came with its own set of challenges, particularly in learning Taipy, a language we chose for its suitability to our project. Adapting to Taipy's unique syntax posed a learning curve, demanding additional effort to grasp the intricacies of its Markdown style syntax. Overcoming these challenges became an essential part of our journey, ultimately contributing to the growth and expertise of our team.

## Accomplishments that we're proud of

We are especially proud of learning to use and implement Taipy into a product that works seamlessly. This success is a testament to our commitment to acquiring new technical skills and adapting to innovative tools. Connecting various APIs showcased the effectiveness of our teamwork and technical expertise. Beyond enhancing our technical capabilities, this experience fostered a stronger sense of camaraderie within our team as we collaborated toward a shared objective.

## What we learned

Throughout the hackathon, our team immersed ourselves in learning new skills and technologies. We not only became adept at version control using Git, but also connecting APIs became a breeze, allowing us to seamlessly integrate a variety of functionalities. Gaining proficiency in Taipy brought a distinctive flair to our toolkit, providing a versatile and powerful language for our project. This learning journey not only expanded our technical know-how but also armed us with the expertise to approach challenges creatively, nurturing a collaborative spirit within the team.

## What's next for linquiztics

Our vision for linquiztics involves transforming it into a Chrome extension, seamlessly integrating it into the Chrome browser for enhanced accessibility. To elevate the user experience, we're introducing live transcription functionality, highlighting problematic words in real-time when watching videos. This feature not only aids in identifying and addressing language comprehension challenges but also reinforces learning through visual cues. Additionally, we're incorporating a pronunciation coach equipped with a language phonetic guide, utilizing the International Phonetic Alphabet (IPA). This guide will offer users precise assistance in mastering pronunciation, adding an invaluable dimension to language learning.
## Inspiration

Often times we’ve seen people post live streams on social media, use video calling for a more personal touch to a conversation, and engage in activities which require extensive expression of emotion through language. However, with over 430 million people around the world having hearing disabilities and less than a fourth of people fluent in sign language, our team was sought to bridge this significant, yet ironically, unattended problem.

## What it does

ConnexionAI.tech is a web application that converts video input of sign language to three most popular languages English, Spanish, and French. The application also has in built capability of converting speech to text and also translate various languages to English - Text . In the future, ConnexionAI can be deployed in video chats and live streams to caption people with speaking disabilities to caption sign language and it could be used as a live ASL to text convertor.

## How I built it

The machine learning model was trained on google cloud’s autoML platform, the web framework we’re using is flask, the website was built in HTML5, CSS, JS, and the project integration was in python.

## Challenges I ran into

Our biggest challenge was to find a reliable dataset and deciding the hyper parameters for training the model under such a time constraint. The other challenge we ran into was integrating the video format in the web browser and taking inputs to backend and pushing outputs to the browser again.

## Accomplishments that I'm proud of

We are proud of having the opportunity to solve a problem that affects the lives of millions of people in a fun learning environment!

## What I learned

All of us worked with something that we had worked with before. Using google cloud for the first time was fascinating and the idea of connecting millions of dots really inspired us to pull through a working model.

## What's next for Connexion

Future possibilities include implementing connexion to livestreams on Facebook and videos on YouTube to caption sign language for all audiences.