## Discord contact

* Nand Vinchhi - vinchhi
* Advay Choudhury - pdy.gn
* Rushil Madhu - rushil695

## Inspiration

Over the past few years, social media platforms like Tiktok and Instagram reels have used short-form content to take over the entertainment industry. Though this hyper-optimized form of content is engaging, it has led to students having shortened attention spans and a higher tendency to multi-task. Through Relearn, we aim to modernize studying to match these changing mental abilities.

## What it does

#### Scrolling interface for short videos

* Intuitive tiktok-like interface for scrolling educational short videos
* Complete with engaging music, bold subtitles, and background videos
* Functionality to like videos, indicating that the user has understood that particular concept

#### Content generation AI

* Uses lecture videos from Khanacademy + GPT-4.5 analysis to generate bite-sized educational clips
* Custom video processing for adding elements such as music and subtitles

#### Recommendation algorithm

* Ensures that the user progressively covers all content in a given topic
* Occasional meme videos for a quick study break!

#### Edison AI

* An LLM-based bot account that answers student questions in the comments section
* Understands the context from the current video to provide precise and useful answers

## Additional details for sponsor prizes

#### Best Use of AI in Education

Relearn explores the confluence of cutting-edge AI tech and useful education solutions -

* Our content generation pipeline utilizes LLMs (GPT-4.5) to understand context in Khanacademy lecture videos to create short and engaging clips
* Our AI account, Edison, utilizes LLMs to provide precise and useful answers to questions in the comments of a specific video

#### Best Accessibility Hack sponsored by Fidelity

Our platform makes high-quality content available over the internet. We enable anyone with a mobile phone to improve their study consistency and quality, which would help bridge economic and social gaps in the long term.

#### Most Creative Use of Redis Cloud

We used Redis for the following pivotal aspects of our app -

* High speed caching and access of dynamic data such as likes, comments, and progress
* Caching the latest video that a given user interacted with in, and feeding that into our recommendation algorithm (speeds up the algorithm by a huge margin)
* See this file for our Redis functions - <https://github.com/NandVinchhi/relearn-hackathon/blob/main/backend/dbfunctions.py>

## How we built it

#### Front-end mobile application

* Built with Swift and Swift UI
* Used multiple packages for specific functionalities such as Google Auth, video playback etc.

#### Content generation pipeline

* YT Transcript API + PyTube for scraping Khanacademy videos
* Open AI to retrieve context and divide long lecture videos into bite-sized clips
* MoviePy for automatic video editing - subtitles, music, and background video

#### Back-end

* Python + FastAPI for the server
* Supabase database + cloud buckets for storing short videos + curriculum
* Redis database for storing dynamic like, comment, and progress data
* Server deployment on AWS EC2

#### Recommendation algorithm

* Caching latest 'liked' short videos for each topic and unit in Redis
* Super fast progress retrieval from in order to compute the next video on scroll

#### Edison AI

* Used GPT 4.5 + short video transcripts for context-specific answers
* AI chat endpoints deployed along with server on AWS

## Challenges we ran into

* LLM hallucinations
* Bugs with automated video processing
* Getting the video scrolling interface to work smoothly without any sort of flickering or unwanted behavior
* Getting the whole stack working together and deployed before submission deadline

## Accomplishments that we're proud of

* Implementing a custom video scrolling interface for our short videos
* Automatically generating high-quality short videos
* Designing a clean user interface using Figma before building it out
* Getting back-end integrated with google auth + multiple databases + APIs and deploying to cloud
* All of this in < 36 hours!

## What we learned

* Using Redis to enhance a more conventional tech stack
* Deploying FastAPI to AWS EC2 using Gunicorn workers
* Building complex automated video editing functionality using MoviePy
* Utilizing the newly released GPT-4.5 API to perform context retrieval and create high-quality videos.
* Building custom UI Components in Swift and working with Google Auth in an iOS application

## What's next for Relearn

* Expanding curriculum to cover topics for all ages
* Multi-language support to increase accessibility
* Periodic flash cards and quizzes based on progress
* Opportunity for creators to upload their own content directly to Relearn
## Inspiration

In a world full of so much info but limited time as we are so busy and occupied all the time, we wanted to create a tool that helps students make the most of their learning—quickly and effectively. We imagined a platform that could turn complex material into digestible, engaging content tailored for a fast-paced generation.

## What it does

Lemme Learn More (LLM) transforms study materials into bite-sized TikTok-style trendy and attractive videos or reels, flashcards, and podcasts. Whether it's preparing for exams or trying to stay informed on the go, LLM breaks down information into formats that match how today's students consume content. If you're an avid listener of podcasts during commute to work - this is the best platform for you.

## How we built it

We built LLM using a combination of AI-powered tools like OpenAI for summaries, Google TTS for podcasts, and pypdf2 to pull data from PDFs. The backend runs on Flask, while the frontend is React.js, making the platform both interactive and scalable. Also used fetch.ai ai agents to deploy on test net - blockchain.

## Challenges we ran into

Due to a highly-limited time, we ran into a deployment challenge where we faced some difficulty setting up on Heroku cloud service platform. It was an internal level issue wherein we were supposed to change config files - I personally spent 5 hours on that - my team spent some time as well. Could not figure it out by the time hackathon ended : we decided not to deploy today.

In brainrot generator module, audio timing could not match with captions. This is something for future scope.

One of the other biggest challenges was integrating sponsor Fetch.ai agentverse AI Agents which we did locally and proud of it!

## Accomplishments that we're proud of

Biggest accomplishment would be that we were able to run and integrate all of our 3 modules and a working front-end too!!

## What we learned

We learned that we cannot know everything - and cannot fix every bug in a limited time frame. It is okay to fail and it is more than okay to accept it and move on - work on the next thing in the project.

## What's next for Lemme Learn More (LLM)

Coming next:

1. realistic podcast with next gen TTS technology
2. shorts/reels videos adjusted to the trends of today
3. Mobile app if MVP flies well!
**Finding a problem**
Education policy and infrastructure tend to neglect students with accessibility issues. They are oftentimes left on the backburner while funding and resources go into research and strengthening the existing curriculum. Thousands of college students struggle with taking notes in class due to various learning disabilities that make it difficult to process information quickly or write down information in real time.

Over the past decade, Offices of Accessible Education (OAE) have been trying to help support these students by hiring student note-takers and increasing ASL translators in classes, but OAE is constrained by limited funding and low interest from students to become notetakers.

This problem has been particularly relevant for our TreeHacks group. In the past year, we have become notetakers for our friends because there are not enough OAE notetakers in class. Being note writers gave us insight into what notes are valuable for those who are incredibly bright and capable but struggle to write. This manual process where we take notes for our friends has helped us become closer as friends, but it also reveals a systemic issue of accessible notes for all.

Coming into this weekend, we knew note taking was an especially interesting space. GPT3 had also been on our mind as we had recently heard from our neurodivergent friends about how it helped them think about concepts from different perspectives and break down complicated topics.

**Failure and revision**
Our initial idea was to turn videos into transcripts and feed these transcripts into GPT-3 to create the lecture notes. This idea did not work out because we quickly learned the transcript for a 60-90 minute video was too large to feed into GPT-3.

Instead, we decided to incorporate slide data to segment the video and use slide changes to organize the notes into distinct topics. Our overall idea had three parts: extract timestamps the transcript should be split at by detecting slide changes in the video, transcribe the text for each video segment, and pass in each segment of text into a gpt3 model, fine-tuned with prompt engineering and examples of good notes.

We ran into challenges every step of the way as we worked with new technologies and dealt with the beast of multi-gigabyte video files. Our main challenge was identifying slide transitions in a video so we could segment the video based on these slide transitions (which signified shifts in topics). We initially started with heuristics-based approaches to identify pixel shifts. We did this by iterating through frames using OpenCV and computing metrics such as the logarithmic sum of the bitwise XORs between images. This approach resulted in several false positives because the compressed video quality was not high enough to distinguish shifts in a few words on the slide. Instead, we trained a neural network using PyTorch on both pairs of frames across slide boundaries and pairs from within the same slide. Our neural net was able to segment videos based on individual slides, giving structure and organization to an unwieldy video file. The final result of this preprocessing step is an array of timestamps where slides change.

Next, this array was used to segment the audio input, which we did using Google Cloud’s Speech to Text API. This was initially challenging as we did not have experience with cloud-based services like Google Cloud and struggled to set up the various authentication tokens and permissions. We also ran into the issue of the videos taking a very long time, which we fixed by splitting the video into smaller clips and then implementing multithreading approaches to run the speech to text processes in parallel.

**New discoveries**
Our greatest discoveries lay in the fine-tuning of our multimodal model. We implemented a variety of prompt engineering techniques to coax our generative language model into producing the type of notes we wanted from it. In order to overcome the limited context size of the GPT-3 model we utilized, we iteratively fed chunks of the video transcript into the OpenAI API at once. We also employed both positive and negative prompt training to incentivize our model to produce output similar to our desired notes in the output latent space. We were careful to manage the external context provided to the model to allow it to focus on the right topics while avoiding extraneous tangents that would be incorrect. Finally, we sternly warned the model to follow our instructions, which did wonders for its obedience.

These challenges and solutions seem seamless, but our team was on the brink of not finishing many times throughout Saturday. The worst was around 10 PM. I distinctly remember my eyes slowly closing, a series of crumpled papers scattered nearby the trash can. Each of us was drowning in new frameworks and technologies. We began to question, how could a group of students, barely out of intro-level computer science, think to improve education.

The rest of the hour went in a haze until we rallied around a text from a friend who sent us some amazing CS notes we had written for them. Their heartfelt words of encouragement about how our notes had helped them get through the quarter gave us the energy to persevere and finish this project.

**Learning about ourselves**
We found ourselves, after a good amount of pizza and a bit of caffeine, diving back into documentation for react, google text to speech, and docker. For hours, our eyes grew heavy, but their luster never faded. More troubles arose. There were problems implementing a payment system and never-ending CSS challenges. Ultimately, our love of exploring technologies we were unfamiliar with helped fuel our inner passion.

We knew we wanted to integrate Checkbook.io’s unique payments tool, and though we found their API well architectured, we struggled to connect to it from our edge-compute centric application. Checkbook’s documentation was incredibly helpful, however, and we were able to adapt the code that they had written for a NodeJS server-side backend into our browser runtime to avoid needing to spin up an entirely separate finance service. We are thankful to Checkbook.io for the support their team gave us during the event!

Finally, at 7 AM, we connected the backend of our website with the fine-tuned gpt3 model. I clicked on CS106B and was greeted with an array of lectures to choose from. After choosing last week’s lecture, a clean set of notes were exported in LaTeX, perfect for me to refer to when working on the PSET later today!

We jumped off of the couches we had been sitting on for the last twelve hours and cheered. A phrase bounced inside my mouth like a rubber ball, “I did it!”

**Product features**
Real time video to notes upload
Multithreaded video upload framework
Database of lecture notes for popular classes
Neural network to organize video into slide segments
Multithreaded video to transcript pipeline