## Inspiration

Has your browser ever looked like this?

![](https://i.imgur.com/waCM1K0.png)
... or this?
![](https://i.imgur.com/WX2dTfz.png)

Ours have, *all* the time.

Regardless of who you are, you'll often find yourself working in a browser on not just one task but a variety of tasks. Whether its classes, projects, financials, research, personal hobbies -- there are many different, yet predictable, ways in which we open an endless amount of tabs for fear of forgetting a chunk of information that may someday be relevant.

Origin aims to revolutionize your personal browsing experience -- one workspace at a time.

## What it does

In a nutshell, Origin uses state-of-the-art **natural language processing** to identify personalized, smart **workspaces**. Each workspace is centered around a topic comprised of related tabs from your browsing history, and Origin provides your most recently visited tabs pertaining to that workspace and related future ones, a generated **textual summary** of those websites from all their text, and a **fine-tuned ChatBot** trained on data about that topic and ready to answer specific user questions with citations and maintaining history of a conversation. The ChatBot not only answers general factual questions (given its a foundation model), but also answers/recalls specific facts found in the URLs/files that the user visits (e.g. linking to a course syllabus).

Origin also provides a **semantic search** on resources, as well as monitors what URLs other people in an organization visit and recommend pertinent ones to the user via a **recommendation system**.

For example, a college student taking a History class and performing ML research on the side would have sets of tabs that would be related to both topics individually. Through its clustering algorithms, Origin would identify the workspaces of "European History" and "Computer Vision", with a dynamic view of pertinent URLs and widgets like semantic search and a chatbot. Upon continuing to browse in either workspace, the workspace itself is dynamically updated to reflect the most recently visited sites and data.

**Target Audience**: Students to significantly improve the education experience and industry workers to improve productivity.

## How we built it

![](https://i.imgur.com/HYsZ3un.jpg)

**Languages**: Python ∙ JavaScript ∙ HTML ∙ CSS

**Frameworks and Tools**: Firebase ∙ React.js ∙ Flask ∙ LangChain ∙ OpenAI ∙ HuggingFace

There are a couple of different key engineering modules that this project can be broken down into.

### 1(a). Ingesting Browser Information and Computing Embeddings

We begin by developing a Chrome Extension that automatically scrapes browsing data in a periodic manner (every 3 days) using the Chrome Developer API. From the information we glean, we extract titles of webpages. Then, the webpage titles are passed into a pre-trained Large Language Model (LLM) from Huggingface, from which latent embeddings are generated and persisted through a Firebase database.

### 1(b). Topical Clustering Algorithms and Automatic Cluster Name Inference

Given the URL embeddings, we run K-Means Clustering to identify key topical/activity-related clusters in browsing data and the associated URLs.

We automatically find a description for each cluster by prompt engineering an OpenAI LLM, specifically by providing it the titles of all webpages in the cluster and requesting it to output a simple title describing that cluster (e.g. "Algorithms Course" or "Machine Learning Research").

### 2. Web/Knowledge Scraping

After pulling the user's URLs from the database, we asynchronously scrape through the text on each webpage via Beautiful Soup. This text provides richer context for each page beyond the title and is temporarily cached for use in later algorithms.

### 3. Text Summarization

We split the incoming text of all the web pages using a CharacterTextSplitter to create smaller documents, and then attempt a summarization in a map reduce fashion over these smaller documents using a LangChain summarization chain that increases the ability to maintain broader context while parallelizing workload.

### 4. Fine Tuning a GPT-3 Based ChatBot

The infrastructure for this was built on a recently-made popular open-source Python package called **LangChain** (see <https://github.com/hwchase17/langchain>), a package with the intention of making it easier to build more powerful Language Models by connecting them to external knowledge sources.

We first deal with data ingestion and chunking, before embedding the vectors using OpenAI Embeddings and storing them in a vector store.

To provide the best chat bot possible, we keep track of a history of a user's conversation and inject it into the chatbot during each user interaction while simultaneously looking up relevant information that can be quickly queries from the vector store. The generated prompt is then put into an OpenAI LLM to interact with the user in a knowledge-aware context.

### 5. Collaborative Filtering-Based Recommendation

Provided that a user does not turn privacy settings on, our collaborative filtering-based recommendation system recommends URLs that other users in the organization have seen that are related to the user's current workspace.

### 6. Flask REST API

We expose all of our LLM capabilities, recommendation system, and other data queries for the frontend through a REST API served by Flask. This provides an easy interface between the external vendors (like LangChain, OpenAI, and HuggingFace), our Firebase database, the browser extension, and our React web app.

### 7. A Fantastic Frontend

Our frontend is built using the React.js framework. We use axios to interact with our backend server and display the relevant information for each workspace.

## Challenges we ran into

1. We had to deal with our K-Means Clustering algorithm outputting changing cluster means over time as new data is ingested, since the URLs that a user visits changes over time. We had to anchor previous data to the new clusters in a smart way and come up with a clever updating algorithm.
2. We had to employ caching of responses from the external LLMs (like OpenAI/LangChain) to operate under the rate limit. This was challenging, as it required revamping our database infrastructure for caching.
3. Enabling the Chrome extension to speak with our backend server was a challenge, as we had to periodically poll the user's browser history and deal with CORS (Cross-Origin Resource Sharing) errors.
4. We worked modularly which was great for parallelization/efficiency, but it slowed us down when integrating things together for e2e testing.

## Accomplishments that we're proud of

The scope of ways in which we were able to utilize Large Language Models to redefine the antiquated browsing experience and provide knowledge centralization.

This idea was a byproduct of our own experiences in college and high school -- we found ourselves spending significant amounts of time attempting to organize tab clutter systematically.

## What we learned

This project was an incredible learning experience for our team as we took on multiple technically complex challenges to reach our ending solution -- something we all thought that we had a potential to use ourselves.

## What's next for Origin

We believe Origin will become even more powerful at scale, since many users/organizations using the product would improve the ChatBot's ability to answer commonly asked questions, and the recommender system would perform better in aiding user's education or productivity experiences.
## Inspiration

Access to course resources is often fragmented and lacks personalization, making it difficult for students to optimize their learning experiences. When students use Large Language Models (LLMs) for academic insights, they often encounter limitations due to LLMs’ inability to interpret various data formats—like lecture audio or photos of notes. Additionally, context often gets lost between sessions, leading to fragmented study experiences.

We created OpenContext to offer a comprehensive solution that enables students to manage, customize, and retain context across multiple learning resources.

## What it does

OpenContext is an all-in-one platform designed to provide students with personalized, contextually-aware resources using LLMs. It aims to make course data open source by allowing students to share their course materials.

### Features:

* Upload and Process Documents: Users can upload or record audio, PDF, and image files related to their classes.
* Chat Assistant: Users can chat with an assistant which will have the context of all the uploaded documents, and will be able to refer to course materials on any given questions.
* Real-Time Audio Transcription: Record lectures directly in the browser, and the audio is transcribed and processed in real time.
* Document Merging and Quiz Generation: Users can combine documents from different formats and generate quizzes that mimic Quizlet-style flashcards.
* Progress Tracking: After completing quizzes, users receive a detailed summary of their performance.

**Full User Flow**: The user faces a landing page where they are prompted with three options: Create a new chat, Upload documents, Generate quizzes. As the user navigates to Upload documents, they have the option to record their current lecture real time from their browser. Or, they can upload documents ranging from audio, pdf and image. We are using Tesseract for Optical Character Recognition on image files, OpenAI’s SpeechtoText API on audio files, and our own PDF parser for other class documents. The user can also record the lecture in real time which will be transcribed and processed in real time. After the transcription of a lecture or another class document is finished, it is displayed to the user. They will be able to create a new chat and ask our AI assistant anything related to their course materials. The assistant will have the full context of uploaded documents and will be able to answer with references to those documents. The user will also have the option to generate a quiz based on the transcription of the lecture that just got recorded. They will also be able to merge multiple class documents and generate a custom quiz out of all. The quizzes will have the format of quizlet flashcards, where a question is asked, 4 answers are provided as options and after an option is chosen, the website will prompt with either the chosen answer is correct or incorrect. The score for each question is calculated and at the end of the quiz a summary of the performance is written for the student user to track their progress.

## How We Built It

* Frontend: Built with React for a responsive and dynamic user interface.
* Backend: Developed with FastAPI, handling various tasks from file processing to vector database interactions.
* AI Integration: Utilized OpenAI's Whisper for real-time speech-to-text transcription and embedding functionalities.
* OCR: Tesseract is used for Optical Character Recognition on uploaded images, allowing us to convert handwritten or printed text into machine-readable text.
Infrastructure: Hosted with Defang for production-grade API management, alongside Cloudflare for data operations and performance optimization.

### Tech Stack:

* Figma: We used Figma to design our warm color palette and simplistic theme for pages and logo.
* Python: Python scripts are used ubiquitously in OpenContext whether it’s for our deployment scripts for Defang or pipelines to our Vector Databases, our API servers, OpenAI wrappers or other miscellaneous tasks, we utilize Python.
* React: React framework is used to build the entire components, pages, and routes on the frontend.
* Tesseract OCR: For converting images to text.
* FastAPI: We have multiple FastAPI apps that we use for multiple purposes. Having endpoints to respond to different types of file requests from the user, making connections to our Vector DB, and other scripting tasks are all handled by the FastAPI endpoints that we have built.
* OpenAI API: We are using multiple services of OpenAI such as the Whisper for ASR or Text Embeddings for later function calling and vector storing.

### Sponsor Products:

* Defang: We used Defang in order to test and host both of our API systems in a production environment. Here is the production PDF API: <https://alpnix-pdf-to-searchable--8080.prod1b.defang.dev/docs>
* Terraform: We used Terraform, a main.tf script, to test and validate our configuration for our deployment services such as our API hosting with Defang and nginx settings.
* Midnight: For open-source data sharing, Midnight provides us the perfect tool to encrypt the shared information. We created our own wallet as the host server and each user is able to create their own private wallets to share files securely.
* Cloudflare: We are using multiple services of Cloudflare…
	+ Vectorize: In addition to using Pinecone, we have fully utilized Vectorize, Cloudflare’s Vector DB, at a high level.
	+ Cloudflare Registrar: We are using Cloudflare’s domain registration to buy our domain.
	+ Proxy Traffic: We are using Cloudflare’s proxy traffic service to handle requests in a secure and efficient manner.
	+ Site Analytics: Cloudflare’s data analytics tool to help us analyze the traffic as the site is launched.
* Databricks: We have fully utilized Databricks Starter Application to familiarize ourselves with efficient open source data sharing feature of our product. After running some tests, we also decided to integrate LangChain in the future to enhance the context-aware nature of our system.

## Challenges we ran into

One significant challenge was efficiently extracting text from images. This required converting images to PDFs, running OCR to overlay text onto the original document, and accurately placing the text for quiz generation. Ensuring real-time transcription accuracy and managing the processing load on our servers were also challenging.

## Accomplishments that we're proud of

* Tool Mastery: In a short time, we learned and successfully implemented production environment tools like NGINX, Pinecone, and Terraform.
* API Integration: Seamlessly integrated OpenAI’s Whisper and Tesseract for multi-format document processing, enhancing the utility of LLMs for students.
* Quiz Generation Pipeline: Developed an efficient pipeline for custom quiz generation from multiple class resources.

## What we learned

* Infrastructure Management: Gained experience using Defang, Terraform, and Midnight to host and manage a robust data application.
* Prompt Engineering: Through David Malan’s session, we enhanced our ability to prompt engineer, configuring ChatGPT’s API to fulfill specific roles and restrictions effectively.

## What's next for Open Context

We aim to develop a secure information-sharing system within the platform, enabling students to share their study materials safely and privately with their peers. Additionally, we plan to introduce collaborative study sessions where students can work together on quizzes and share real-time notes. This could involve shared document editing and group quiz sessions to enhance the sense of open source.
## Inspiration

There's a lot of negativity in news articles nowadays. "News fatigue" is a phenomenon where people become tired of the constant barrage of negative news. We thought of making a **positive only** news app. For some, a positive news app may be a welcome alternative that helps them stay informed without feeling emotionally drained.

## What it does

Proton News is a full-stack application, built completely by us. Here's what it does:

* Queries NewsAPI for articles based on a user's search category
* Uses Cohere to do a sentiment analysis of the articles, and filters out 'negative articles'
* Takes the positive articles, uses Cohere to summarize the articles and stores them on CockroachDB for easy access
* Retrieves the articles from Cockroach DB whenever a user runs the page

## How we built it

The application primarily has two components: a frontend, and a backend.

* The frontend is build on the Next.js framework, with server-side rendering for ultra-fast loads
* The runtime used is Bun, which is a Node.js alternative popular for its low latency and responsiveness
* The backend is primarily built in FastAPI on Python
* The database where the articles are stored is server less Cockroach DB
* Sentiment analysis and summarizing of articles is done using Cohere

## Challenges we ran into

The biggest challenge we had was working with ORM and relational database. We had little experience working with relational databases, so we spent large amount of time learning to query and insert data. The news API that we worked with also returned different types of data (e.g. HTML and string), making text parsing difficult.

## Accomplishments that we're proud of

The project was very ambitious, with plans of integrating a full-fledge SQL database such as CockroachDB, a front-end interface in React as well as a back-end in Python. We, therefore, couldn't believe that MVP was completed within just under 33 hours. It was our blood (nosebleed), sweat, midnight starvation and definitely the all-nighters that made the project much more special. Furthermore, each individual member developed much better skills in SQL database, full-stack developmennt and event graphic designer.

## What we learned

We learned how to work with relational databases (CockroachDB), developing frontend using Next.js, and using third-party services such as Cohere. We also learned to plan our projects thoroughly and research the feasibility of what we wanted to achieve.

## What's next for ProtonNews

ProtonNews still has a lot to offer. We're planning on integrating more of Cohere's features such as rerank and semantic search to allow a wider search range. The database (although functional) has fairly primitive functionalities. As such, better design schema could be contructed to further optimize the queries, allowing the response to be much faster. Finally, deployment is a must-have. The front-end will be quickly delivered with Nextjs, meanwhile, the back-end will be hosted on Google Cloud.