## Inspiration

Who doesn't love studying with their friends; getting work done with countless hours of fun conversations. But when you have a looming midterm the next morning, maybe you shouldn't study with your friends?

That was until StudySync, a collaborative studying platform that optimizes your ability to study with your friends. By tracking attention and note taking using deep learning, users get assigned points for focusing with a leaderboard of everyone's score at the study session, allowing you to stay motivated to study and encourage your friends to not fall behind. With AI detection when users are going on their phone or about to sleep, StudySync uses AI voices to wake users up and yells at users to put their phone away.

StudySync understands the importance of active recall, and by inputting your notes/course text, StudySync will periodically give a quizzes based on content the group just finished studying. Our smart quiz system pairs you with peers to master difficult topics and lock in. With public and private study rooms for introverts and extroverts alike, our app creates a vibrant, supportive environment for learning—anywhere, anytime for a diverse group of people, thus, promoting diversity in educational attainment. However, StudySync knows the importance of breaks as well, and gives the group periodic breaks.

## What it does

StudySync isn't just another study app—it’s your new go-to for leveling up your learning game with a dose of fun. Imagine hoping into a collaborative study session with friends, where your attention is tracked by your camera using OpenCV and MediaPipe. Stay locked in and score points, but if you start scrolling through your phone, brace yourself—your points take a hit 😲. And for an extra twist, you'll get a dramatic cutscene with a voiceover (thanks, ElevenLabs!) calling you out, keeping things lighthearted but on track. The app can even tell when you're just plain exhausted, suggesting a break or nap when you need it most, with built-in Pomodoro timers for structured rest. When it's time to test your knowledge, StudySync serves up periodic quizzes based on the material you’ve uploaded, and the app toggles between quizzes and time intervals as you complete sections. Active recall keeps the pressure on, tossing in questions from earlier sections, while the app keeps track of your mistakes, pairing you with peers who can help fill in the gaps. It’s not just studying—it’s an interactive, engaging learning experience designed to keep you sharp and connected.

You're also able to create public/private study rooms – helpful for people who are introverted and want a study partner, and implement a system where you can input how many days you have before a test/quiz—it'll tailor your study plan to help you lock in on the topics that matter the most!

## How we built it

We built the backend with Node.js, Express.js, and used WebSockets for real-time features like room creation, joining rooms, sending messages and updating points. We also used HTTP (REST API) to handle regular POST requests with the Express framework. These requests return JSON data about rooms and their existence or provide a basic welcome message for users interacting with the backend directly, and is hosted on Vercel. We used other features to make the computer vision task work, such as OpenCV, and MediaPipe for facial detection and a DNN using TensorFlow, and for object detection we used a CNN dataset trained on 9,900 images. We also used ElevenLabs' Text-to-Speech API for a much more expressive voice.

The frontend is primarily built using JavaScript/TypeScript with React and Tailwind CSS. Socket.IO is used for real-time communication between the frontend and backend. For webcam access we used WebRTC, then we used a Fetch API for sending captured frames to the backend for attention analysis. We also created a leaderboard system to add in our gamification effect.

Not only that, but to keep track of user and session data, we implemented OAuth authentication using Clerk!

## Challenges we ran into

Firstly, we didn't have full team to begin with, as well as all our members having midterms during the time of the hackathon, in particular, one member in our team has her midterm on the second day of the hackathon, which was a difficult challenge as we navigate through how to make the best use of our time. Throughout this weekend, it has been essential to balance working on our project with our mental health, rest, and last-minute study. 2 of our members were also from Waterloo, so commuting back and forth was a difficult challenge for us. As we work in a team of three, we had to decide on a project idea, scope, features and get working on it immediately due to the limited time given the challenges above. Maximizing our productivity was difficult when some tasks depended on others.

We encountered challenges with merging Git commits, often overwriting each other's code and introducing bugs. To overcome this, we had to learn new technologies, techniques, and adapt quickly, and learned new technologies to make our ideas happen such as navigating around the computer vision aspect. We initially started with a variety of features we wanted to work with, and ran into a challenge where we had to decide which to pursue and how to pursue each individual features.

To navigate these challenges, we read documentations, used Stack Overflow, and spoke with sponsors and mentors. With some turnarounds, we solved them and executed out the features (mostly) according to plan!

## Accomplishments that we're proud of

We’re proud of the significant progress we’ve made in such a short time on a project we initially thought was overly ambitious. We've successfully reached the majority of our key milestones and have also picked up and learned different tools along the way, such as navigating around the computer vision aspect.

## What we learned

Tackling computer vision and integrating tools like OpenCV and MediaPipe introduced us to a host of new technical challenges. With that said, we learned to quickly adapt, troubleshoot, and leverage documentation and mentorship. Tackling CV and integrating different features with tools like ElevenLabs introduced us to a host of new technical challenges, but we learned to quickly adapt and troubleshoot.

## What's next for StudySync

StudySync is just getting started. Our next steps are focused on enhancing engagement and personalization. We’re working on new ways to customize study plans based on individual needs, such as tailoring quizzes and attention tracking based on how close exams are, indicated by users. We’ll also be expanding our social features, making it easier for introverted users to join public or private study rooms. Integrating active recall, cumulative quizzes, and collaborative teaching moments will ensure deeper learning and concept mastery. Plus, refining attention tracking is another major next step, with more advanced metrics to help users manage fatigue and focus. We hope to incorporate more features like uploadable materials, interactive cut scenes, and real-time study performance insights. Stay tuned for the next wave of immersive, collaborative, effective study experiences!
## Inspiration

Being second years, we all related to finding it difficult connecting with people and making support/study groups, especially due to the transition onto online learning. StudyBuddy is a way to connect you with people taking the same courses as you, in hope of forming these friend/study groups.

## What it does

Study buddy is a clever online application that enables students to learn collectively in an effective and efficient manner. Study buddy keeps records of these courses a user takes in their current study period, it searches through its database to find students taking the same course and groups nearby matches to a group study session. Collaborative education has been proven to improve student comprehension and overall content because students can turn lecture notes into their own words.

## Challenges we ran into

We didn't start working until Saturday night and ran into a time crunch. It was especially difficult getting the routes for various pages to synchronize with one another and ensure that all the pages were running smoothly. However, after persevering quite a bit we were able to make a breakthrough and figure out what we needed to do. Another issue we ran into near the submission period was trying to deploy the app on an online platform.

## What we learned

Time management is key. We definitely learned a lot about effective communication skills and that taking a proactive approach to discussions can not only boost morale for a team but also help build stronger bonds between team members. Several of us were exposed to new technologies and learned how to use them in a short period of time which is one of our key takeaways from this event.

## What's next for StudyBuddy

We may refine the app more and possibly launch it on the app/play store. Furthermore, we plan to implement more immersive features to reduce the number of third parties we would rely on by building out an in-house chat system and connect feature. Study buddy has great potential to become an MVP in a student's resource arsenal. We would like to conduct more research on the target market we aim to reach and how much of that market we will be able to capture. Finally, we would love to implement more management software to ensure that our databases are updated and managed regularly.
### FOR EVALUATORS/SPONSORS:

Scroll down for a handy guide to navigating our repository and our project's assets.

## 💥 How it all started

Every student learns differently. Traditional educational approaches, while effective for some, often fail to cater to the diverse needs and paces of individual learners. Even traditional AI learning is often one-sided lecturing that fails to properly engage the student in actual learning. This realization sparked a quest for more personalized and adaptive teaching strategies.

The capability of AI to analyze and interpret human facial expressions in real-time presented a unique opportunity to improve on today’s sparse learning environment. We envisioned a system that could monitor students' engagement levels, emotional responses, and cognitive states during lessons, allowing for dynamically changing teaching methods.

By leveraging Hume.AI’s expression detection, we aimed to create a feedback loop where we could have an AI educator receive immediate and actionable information about their students' learning experiences. This creates a seamless experience where the user does not even need to inform the AI about whether the teaching approach is right for them—they simply can focus on learning, while letting us do the rest.

The problems encountered by the rigid, traditional teaching methods:

* Lack of Personalization: One-size-fits-all approach fails to address individual learning styles and paces, disproportionately affecting low-income and neurodivergent learners.
* Student Disengagement: Inability to maintain student interest and motivation due to uniform delivery methods.
* Limited Feedback: Current education practices struggle to gauge real-time understanding and emotional responses of all students.
* Underutilized Potential: Advanced students may not be sufficiently challenged, leading to boredom and disengagement.
* Teacher Workload: High demand on teachers to adapt lessons manually for diverse classrooms without sufficient support.
* Resistance to Change: Institutional inertia and reluctance to adopt new teaching technologies and methodologies.
* Rigid Schedules: Students must choose between real-time teaching with instant feedback (that they must attend at certain times) or asynchronous learning that does not guarantee instant feedback (as the instructor is not always online)

## 📖 What it does

To address these challenges, we developed **lock in**, an app designed to allow students of any age or level to learn the way that is best for them.

Here’s how **lock in** leverages advanced technologies to serve its mission:

* Personalized Learning Assistants: By leveraging Hume’s Empathic Voice Interface, we are able to provide students the ability to learn from and converse with an interactive AI assistant that leverage RLHF (reinforcement learning through human feedback) techniques to adjust its teaching methods based on the students facial and vocal expressions.
* Inclusive Assistant for Neurodivergent Learners: **lock in** was designed with neurodivergent learners in mind (if not at the forefront). Those with special needs in education (especially in developing parts of the world) often don’t have proper infrastructure for their learning. We especially aim to target those with ADHD, with our AI constantly watching and innovating on its education in order to keep those with shorter attention spans engaged in learning.
* Intelligent Curriculum Generation: By integrating Mathpix and Claude 3.5 through Amazon Bedrock, **lock in** is able to generate a detailed curriculum to be taught by the AI learning assistant. In doing so, students can easily supplement their in-class learning with a curriculum that is based on their current learning.
* Active Learning Encouragement: We designed our model to not simply lecture the student, but involve the student in the learning process by presenting them examples to grapple with and inviting them to connect concepts. Passive consumption only makes students feel like they’re learning, but results in less learning overall¹. We focus on inviting the student to actively grapple with material.
**lock in** is more than just an education app; it's a movement towards personalized, efficient learning. Through the use of advanced technology such as React, FastAPI, AWS, Hume, and Claude, we aim to create a world in which learning is fun for everyone.

## 🔧 How we built it

To address the challenges of creating personalized education with our app, **lock in**, we've woven together a fabric of cutting-edge software technologies, each selected for its ability to enhance the app's functionality and user experience. Here's how each technology contributes to **lock in**:

### Technologies:

* React: Powers the user interface, offering a smooth and pleasing experience
* FastAPI: Provides the backbone for the backend, handling requests efficiently and ensuring scalability to support a growing user base.
* AWS Bedrock (Claude 3.5 Sonnet): Used to give more context about syllabi, especially when they contain images and diagrams that cannot be processed by Mathpix. This allows **lock in** to more easily understand what it’s looking at. Claude also creates comprehensive lesson plans and lesson topics, which are then handed to EVI for specifics.
* Hume’s EVI (powered by Claude 3.5 Sonnet): Provides speech to speech voice chatting capabilities through EVI (Empathic Voice Interface), taking into account users’ voice tones and emotions. This allows EVI to learn how to better teach on the fly and change teaching styles if it detects boredom or other negative emotions. This switching of styles can be especially useful for neurodivergent students who may have trouble staying on target. EVI handles the specifics of teaching each lesson, handed to it by Claude on Bedrock.
* Hume’s Expression Measurement API: Provides the ability for EVI to read facial expressions and emotions, giving even more emotional context. EVI will respond to speech in a more nuanced way, with even more information than it usually has by itself.
* Mathpix: Extracting the text from syllabi, allowing it to be passed into Claude to generate a curriculum. This allows the user to easily and accurately import pre-existing lesson plans.

Learning with AI today is currently extremely hard and time consuming. But, with the integration of these emotionally aware technologies into **lock in**, we can create a more proactive and sensitive learning environment, suitable for those young and old, as well as those neurotypical and neurodivergent.

Our AI assistant takes more control of the curriculum, rather than taking a backseat and requiring the student to come up with prompts. React and FastAPI ensure a robust and efficient app infrastructure, while Mathpix and the multimodal Claude 3.5 on AWS Bedrock give **lock in** the ability to understand images and create curriculums Lastly, Hume’s EVI (powered by Claude) and Expression Measurement API do the emotional heavy lifting. This makes **lock in** not just a lifeless learning tool, but a supportive and initiative-taking academic companion. This synergy of technologies makes **lock in** a powerful platform for individuals of all learning styles and needs, taking in human feedback and approaching change in a more human way.

## 🚩 Challenges we ran into

Information Integration: **lock in** records and processes data from multiple sources, including, but not limited to, user speech, syllabi, and user-drawn visual resources. It was a huge challenge to manage our data, creatively integrate them into different services, and process them to quickly provide Samantha’s seamless, and human-like responses.

Prompt Engineering: Our control over model input and output is fairly limited, meaning that we have to get creative with things like config settings and context. Most of our ability to specify behavior lies in our system prompt, however. This creates a nearly bottomless creative task of prompting our teaching assistant to educate our users as best as possible. This is, of course, made even more complicated by the unique personalities and learning styles of different people. We strive to capture all learners, but there is no “one-size-fits-all” prompt.

## 🏆 Accomplishments that we're proud of

Fully implementing all the major features we planned to when we first started brainstorming
Integrating both vocal conversations/vocal expression (EVI) and facial expression (Expression Measurement API) into one seamless product, which has not yet been fully explored

We improved on Learning with AI in the following ways:

1. Typical AI chatbot learning experiences revolves around students needing to take the initiative to split a potentially complex topic up into parts themselves and tracking progress on them. Our lesson-based learning helps students who prefer learning "on-rails", giving a more structured learning experience that is less daunting to those who are unfamiliar with the topics they are asking about. This makes it easier to stay on task, track progress, and understand how the current lesson fits into the topic as a whole.
2. AI chatbots revolve around inputs and outputs being types. This results in stilted, slow experiences that don't quite feel "real" to a user. **lock in** features speech input and output, giving a more quick, conversational experience. This is also a more natural way of communicating, allowing users to properly immerse themselves in learning as if they were talking to a tutor or teacher.
3. Most LLMs require a good bit of calibration and user input in order to receive the desired result. Most casual users are either not aware or not skilled at this, and thus they are not able to extract full potential from their learning. Samantha continuously tracks emotions through speech prosody and facial expressions, and will offer a change of gears to the student if she detects something isn't quite working.

## 📝 What we learned

Nicholas: I learned a lot about the strengths and limits of modern multi-modal LLMs like Claude 3.5. After experimenting with different prompts and tasks, I feel more confident on how I would use them in the future and I am excited to see how new models push the boundaries.

Leon: Participating in this AI hackathon showed me the unlimited possibilities of AI, and how it can improve so much in the world. Working on this project had a lot of various struggles, but ultimately, building together with my team was an amazing experience that allowed me to learn so much.

Ryan: This AI Hackathon was my first ever hackathon experience, so getting a hang of the process and conventions was a very novel experience. I got a far better sense of how applications are actually tied together.

Alex: I had tons of fun figuring out how to integrate all of these completely different technologies into one seamless application. It was a huge challenge to manage our data and figure out if and how things should connect to each other. This is the most data heavy project I have ever worked on, and I had a blast doing so!

## ✈️ What's next for **lock in**

Just like how a great trip has a great itinerary, we envision **lock in**’s future plans in phases.

### Phase 1: Solidifying Our Roots

Phase 1 involves solidifying the use of the whiteboard, as well as having Samantha remember specific learning characteristics of students. We also want to improve our UI to create a more streamlined and user-friendly experience.

### Phase 2: Branching Out

Phase 2 involves improving Samantha’s specific subject capabilities, especially her skills in mathematics. We may experiment with other image analysis strategies, using the multimodal Claude for understanding diagrams and Mathpix’s OCR capabilities for reading text. We also want to give her image generation capabilities to supplement explanations with visual examples and close the loop on image to image communication.

### Phase 3: Take Off

Phase 3 involves partnering with established online education companies like Brilliant.org or Curiosity, improving their services and reaching a larger community of those who love to learn.

## 📋 Evaluator's Guide to **lock in**

Intended for judges, however the viewing public is welcome to take a look.
Hey! We wanted to make this guide in order to help provide you further information on our implementations of certain programs and provide a more in-depth look to cater to both the viewing audience and evaluators like yourself.

### SPONSOR SERVICES WE HAVE USED THIS HACKATHON

* Hume
* AWS Bedrock (Claude 3.5)
* Mathpix

### REFERENCE:

1: Deslauriers L, McCarty LS, Miller K, Callaghan K, Kestin G. Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proc Natl Acad Sci U S A. 2019 Sep 24;116(39):19251-19257. doi: 10.1073/pnas.1821936116. Epub 2019 Sep 4. PMID: 31484770; PMCID: PMC6765278.