## Inspiration

Frustrated by the steep learning curve of LaTeX—a formatting language used by over 90% of students in scientific fields like math or physics—we, as engineering students, found ourselves spending more time converting handwritten notes into LaTeX than solving the actual problems.

## What it does

Upload notes or problem sets as images, and Elevate converts them into LaTeX documents while annotating errors with explanations. This process not only simplifies formatting but also enhances learning by helping you identify and correct mistakes for improved academic performance.

## How we built it

Convex helped us link our React frontend to the backend smoothly, making it quicker to change things and catch mistakes thanks to its type-safe coding. We organized our data using Convex's NoSQL database, adding structure and making searches faster with indexing. For security, we hooked up authentication easily with Convex's Clerk Adaptor. Since we needed to store images and LaTeX files, Convex was handy there too. Plus, we used Convex’s vector database to smarten up our chatbot, making everything work together nicely and more efficiently.

**OCR Preprocessing (Image2Latex)**:
We converted images to grayscale via the following formula:

![Grayscale Conversion Formula](https://i.ibb.co/KWnLMwQ/Screenshot-2024-02-18-at-8-19-38-AM.png)

We applied Gaussian Blur, which smooths the image by averaging the pixels based on their spatial closeness and intensity similarity, helping to reduce high-frequency noise for next steps. We utilized the OpenCV Library to compute a (5, 5) kernel with standard deviation 0, via the following formula for a two-dimensional Gaussian:

![](https://i.ibb.co/Nj50g0L/Screenshot-2024-02-18-at-8-29-08-AM.png)

Finally, we employed a custom sharpening kernel,

![](https://i.ibb.co/7vdwTBp/Screenshot-2024-02-18-at-8-24-13-AM.png)

which enhances edges by increasing the contrast between adjacent pixels. The convolution operation of the image with the kernel, I \* K, is given by

![](https://i.ibb.co/88qvbSq/Screenshot-2024-02-18-at-8-24-36-AM.png)

After the preprocessing stage, when then feed our resulting image into OpenAI’s Computer Vision API, to generate text describing what is in the image. We then feed this text into together.AI’s Mixtral-8x7B LLM, which is able to generate appropriate latex code for the text. Finally, once again using together.AI’s LLM and extensive prompt engineering and fine-tuning, we were able to highlight errors and generate corrections.

![](https://i.ibb.co/b30xYJK/DEMO.png)

Original Image

![](https://i.ibb.co/rGdvkPP/BBB.jpg)

Image After Gaussian Blur (Reduced Noise)

![](https://i.ibb.co/RySLp00/CCC.jpg)

Image After Sharpening Kernel Convolution

![](https://i.ibb.co/1G1m7YJ/Screenshot-2024-02-18-at-8-46-05-AM.png)

Latex Code Generated by Together.AI

![](https://i.ibb.co/311Z2Lm/Screenshot-2024-02-18-at-8-46-20-AM.png)

Latex Code Corrected by Together.AI

**Database Vector Search**:

Using Intersystem’s Langchain-IRIS vector search technology, we were able to:

* Generate embedded vectors for latex notes
* Use cosine similarity to query, for a certain problem, the most relevant notes that contain information to answer this problem
* By representing each note as an edge and drawing an edge between nodes with high similarity (low vector degree distance), create a connection graph and mind map of all notes in the database
* For example, querying the question “How can I find a stable matching?” in a database of one of our team member's discrete math course notes results in the following responses from Intersystem’s API:

![https://ibb.co/Vg0Fq0F](https://i.ibb.co/ZBRQMRQ/Screenshot-2024-02-18-at-8-58-17-AM.png)

Leveraging Together.AI’s Mixtral LLM, we also created an interactive ChatBot that is able to summarize ideas in the notes and answer questions from users about the notes.

![https://ibb.co/xXwVj7B](https://i.ibb.co/rFqn6wz/Screenshot-2024-02-18-at-9-14-22-AM.png)

## Challenges we ran into

It took a while to figure out how to render LaTeX text in a browser, as no existing packages met our needs for clarity and precision in displaying complex formulas.

Creating LLM prompts for annotating student solutions with feedback proved difficult. We had to not only ensure that the feedback was accurate but also that it was formatted and presented properly.

The technical complexity of implementing the vector graph for note comparison presented a hard challenge. Perfecting an algorithm that seemlessly integrate Intersystem's Langchain vector embeddings into a graph structure, and generating appropriate adjacency lists required deep dives into ML, optimization, and Data Structures. Additionally, rendering the graph on our live website presented an additional challenge.

## Accomplishments that we're proud of

We successfully built a tool that we will use for the rest of our lives and are optimistic enough about the quality/market potential to go to market after the Hackathon.

## What we learned

We learned the crucial importance of having a diverse team with varied skill sets, enabling us to effectively divide the workload across design, backend development, and machine learning tasks.

## What's next for Elevate

Get 1k users by posting on social media. Use feedback from users to iterate product and add features they would find helpful.
## Inspiration

Coming from South Texas, two of the team members saw ESL (English as a Second Language) students being denied of a proper education. Our team created a tool to break down language barriers that traditionally perpetuate socioeconomic cycles of poverty by providing detailed explanations about word problems using ChatGPT. Traditionally, people from this group would not have access to tutoring or 1-on-1 support, and this website is meant to rectify this glaring issue.

## What it does

The website takes in a photo as input, and it uses optical character recognition to get the text from the problem. Then, it uses ChatGPT to generate a step-by-step explanation for each problem, and this output is tailored to the grade level and language of the student, enabling students from various backgrounds to get assistance they are often denied of.

## How we built it

We coded the backend in Python with two parts: OCR and ChatGPT API implementation. Moreover, we considered the multiple parameters, such as grade and language, that we could implement in our code and eventually query ChatGPT with to make the result as helpful as possible. On the other side of the stack, we coded it in React with TypeScript to be as simple and intuitive as possible. It has two sections that clearly show what it is outputting and what ChatGPT has generated to assist the student.

## Challenges we ran into

During the development of our product we often ran into struggles with deciding the optimal way to apply different APIs and learning how to implement them, many of which we ended up not using or changing our applications for, such as the IBM API. Through this process, we had to change our high level plan for the backend functions and consequently reimplement our frontend user interface to fit the operations. This also provided a compounding challenge of having to reestablish and discuss new ideas while communicating as a team.

## Accomplishments that we're proud of

We are proud of the website layout. Personally the team is very fond of the color and the arrangement of the site’s elements. Another thing that we are proud of is just that we have something working, albeit jankily. This was our first hackathon, so we were proud to be able to contribute to the hackathon in some form.

## What we learned

One invaluable skill we developed through this project was learning more about the unique plethora of APIs available and how we can integrate and combine them to create new revolutionary products that can help people in everyday life. We not only developed our technical skills, including git familiarity and web development, but we also developed our ability to communicate our ideas as a team and gain the confidence and creativity to create and carry out an idea from thought to production.

## What's next for Homework Helper

As part of our mission to increase education accessibility and combat common socioeconomic barriers, we hope to use Homework Helper to not only translate and minimize the language barrier, but to also help those with visual and auditory disabilities. Some functions we hope to implement include having text-to-speech and speech-to-text features, and producing video solutions along with text answers.
## Inspiration

Virtually every classroom has a projector, whiteboard, and sticky notes. With OpenCV and Python being more accessible than ever, we wanted to create an augmented reality entertainment platform that any enthusiast could learn from and bring to their own place of learning. StickyAR is just that, with a super simple interface that can anyone can use to produce any tile-based Numpy game. Our first offering is *StickyJump* , a 2D platformer whose layout can be changed on the fly by placement of sticky notes. We want to demystify computer science in the classroom, and letting students come face to face with what's possible is a task we were happy to take on.

## What it does

StickyAR works by using OpenCV's Contour Recognition software to recognize the borders of a projector image and the position of human placed sticky notes. We then use a matrix transformation scheme to ensure that the positioning of the sticky notes align with the projector image so that our character can appear as if he is standing on top of the sticky notes. We then have code for a simple platformer that uses the sticky notes as the platforms our character runs, jumps, and interacts with!

## How we built it

We split our team of four into two sections, one half that works on developing the OpenCV/Data Transfer part of the project and the other half who work on the game side of the project. It was truly a team effort.

## Challenges we ran into

The biggest challenges we ran into were that a lot of our group members are not programmers by major. We also had a major disaster with Git that almost killed half of our project. Luckily we had some very gracious mentors come out and help us get things sorted out! We also first attempted to the game half of the project in unity which ended up being too much of a beast to handle.

## Accomplishments that we're proud of

That we got it done! It was pretty amazing to see the little square pop up on the screen for the first time on top of the spawning block. As we think more deeply about the project, we're also excited about how extensible the platform is for future games and types of computer vision features.

## What we learned

A whole ton about python, OpenCV, and how much we regret spending half our time working with Unity. Python's general inheritance structure came very much in handy, and its networking abilities were key for us when Unity was still on the table. Our decision to switch over completely to Python for both OpenCV and the game engine felt like a loss of a lot of our work at the time, but we're very happy with the end-product.

## What's next for StickyAR

StickyAR was designed to be as extensible as possible, so any future game that has colored tiles as elements can take advantage of the computer vision interface we produced. We've already thought through the next game we want to make - *StickyJam*. It will be a music creation app that sends a line across the screen and produces notes when it strikes the sticky notes, allowing the player to vary their rhythm by placement and color.