## Inspiration

While brainstorming for ideas, we were shocked to find quite a few projects similar to ones we've worked on before, and even some similar to ideas we had came up with during PennApps. We decided to make a website/search engine tool to help hackers like us realize whether our ideas are actually new, and prevent duplicate projects.

## What it does

"Been There Done That" is a minimalist and elegant website with just a search bar, but behind the scenes. it's a powerful data retrieval and analysis tool. Type in a couple sentences describing the core ideas of projects you're thinking about, and the website will look through hundreds of old hackathon projects submitted on Devpost while analyzing how similar their developer-written descriptions are to yours. Receive a nice overview of the key words of your descriptions, and which old projects are most similar to yours.

## How we built it

We first gathered hackathon project data by using Apify's web scraping API. We wrote logic that allows the scraper to crawl through many pages of a Devpost hackathon submissions page, and previous editions of the hackathon, all while opening each new project and storing the "What it does" section.

Then, we created a similarity checking process starting with twinword's text similarity API. It gave a decent metric for quickly finding which projects have at least some similarity to the user-supplied one. We improved this analysis by using twinword's topic tagging API in order to find the key words in the user's description, and then searching the hackathon database with it.

We designed the website using Materialize, and put the web app together using flask and python.

## Challenges we ran into

The most difficult part of developing this was the theorizing of how to improve text similarity analysis beyond the twinword API. We tried many different ideas, but most of them failed or weren't much of an improvement for too high a performance cost.

## Accomplishments that we're proud of

We built finished all parts of the product, and created a website that's really easy to use while having really useful info to offer. Our idea for using a keyword extractor for searching also improved accuracy above that of just twinword's text similarity API.

## What we learned

The people who research NLP have very difficult jobs. Even using an existing text analysis tool, the results weren't very good, and our many tries to improve the process either made it slower or less accurate. However, as a team, we learned the power of determination, from staying up so long and finally feeling the satisfaction of a (decently) working product with a beautiful look.

## What's next for Been There Done That

We hope to continue developing the tool to scan more hackathon submissions and become more accurate at detecting similar ideas.
## Inspiration

We realized that a lot of time is wasted on making mistakes that have been done before or failing to learn from others. We also were sometimes disappointed with the unoriginal content that was prominent during various hackathons.

## What it does

Our project scans and scrapes Devpost. It uses natural language processing techniques to find similarities between ideas and display similar projects based on an idea that the user inputs. Using our tool, the user can discover what techniques can be used for accomplishing their goal. They can build upon the work of others

## How we built it

## Challenges we ran into

## Accomplishments that we're proud of

## What we learned

## What's next for Devja Vu
## Inspiration

The inspiration for the project was our desire to make studying and learning more efficient and accessible for students and educators. Utilizing advancements in technology, like the increased availability and lower cost of text embeddings, to make the process of finding answers within educational materials more seamless and convenient.

## What it does

Wise Up is a website that takes many different types of file format, as well as plain text, and separates the information into "pages". Using text embeddings, it can then quickly search through all the pages in a text and figure out which ones are most likely to contain the answer to a question that the user sends. It can also recursively summarize the file at different levels of compression.

## How we built it

With blood, sweat and tears! We used many tools offered to us throughout the challenge to simplify our life. We used Javascript, HTML and CSS for the website, and used it to communicate to a Flask backend that can run our python scripts involving API calls and such. We have API calls to openAI text embeddings, to cohere's xlarge model, to GPT-3's API, OpenAI's Whisper Speech-to-Text model, and several modules for getting an mp4 from a youtube link, a text from a pdf, and so on.

## Challenges we ran into

We had problems getting the backend on Flask to run on a Ubuntu server, and later had to instead run it on a Windows machine. Moreover, getting the backend to communicate effectively with the frontend in real time was a real challenge. Extracting text and page data from files and links ended up taking more time than expected, and finally, since the latency of sending information back and forth from the front end to the backend would lead to a worse user experience, we attempted to implement some features of our semantic search algorithm in the frontend, which led to a lot of difficulties in transferring code from Python to Javascript.

## Accomplishments that we're proud of

Since OpenAI's text embeddings are very good and very new, and we use GPT-3.5 based on extracted information to formulate the answer, we believe we likely equal the state of the art in the task of quickly analyzing text and answer complex questions on it, and the ease of use for many different file formats makes us proud that this project and website can be useful for so many people so often. To understand a textbook and answer questions about its content, or to find specific information without knowing any relevant keywords, this product is simply incredibly good, and costs pennies to run. Moreover, we have added an identification system (users signing up with a username and password) to ensure that a specific account is capped at a certain usage of the API, which is at our own costs (pennies, but we wish to avoid it becoming many dollars without our awareness of it.

## What we learned

As time goes on, not only do LLMs get better, but new methods are developed to use them more efficiently and for greater results. Web development is quite unintuitive for beginners, especially when different programming languages need to interact. One tool that has saved us a few different times is using json for data transfer, and aws services to store Mbs of data very cheaply. Another thing we learned is that, unfortunately, as time goes on, LLMs get bigger and so sometimes much, much slower; api calls to GPT3 and to Whisper are often slow, taking minutes for 1000+ page textbooks.

## What's next for Wise Up

What's next for Wise Up is to make our product faster and more user-friendly. A feature we could add is to summarize text with a fine-tuned model rather than zero-shot learning with GPT-3. Additionally, a next step is to explore partnerships with educational institutions and companies to bring Wise Up to a wider audience and help even more students and educators in their learning journey, or attempt for the website to go viral on social media by advertising its usefulness. Moreover, adding a financial component to the account system could let our users cover the low costs of the APIs, aws and CPU running Whisper.