## Inspiration

We're 4 college freshmen that were expecting new experiences with interactive and engaging professors in college; however, COVID-19 threw a wrench in that (and a lot of other plans). As all of us are currently learning online through various video lecture platforms, we found out that these lectures sometimes move too fast or are just flat-out boring. Summaread is our solution to transform video lectures into an easy-to-digest format.

## What it does

"Summaread" automatically captures lecture content using an advanced AI NLP pipeline to automatically generate a condensed note outline. All one needs to do is provide a YouTube link to the lecture or a transcript and the corresponding outline will be rapidly generated for reading. Summaread currently generates outlines that are shortened to about 10% of the original transcript length. The outline can also be downloaded as a PDF for annotation purposes. In addition, our tool uses the Google cloud API to generate a list of Key Topics and links to Wikipedia to encourage further exploration of lecture content.

## How we built it

Our project is comprised of many interconnected components, which we detail below:

**Lecture Detection**

Our product is able to automatically detect when lecture slides change to improve the performance of the NLP model in summarizing results. This tool uses the Google Cloud Platform API to detect changes in lecture content and records timestamps accordingly.

**Text Summarization**

We use the Hugging Face summarization pipeline to automatically summarize groups of text that are between a certain number of words. This is repeated across every group of text previous generated from the Lecture Detection step.

**Post-Processing and Formatting**

Once the summarized content is generated, the text is processed into a set of coherent bullet points and split by sentences using Natural Language Processing techniques. The text is also formatted for easy reading by including “sub-bullet” points that give a further explanation into the main bullet point.

**Key Concept Suggestions**

To generate key concepts, we used the Google Cloud Platform API to scan over the condensed notes our model generates and provide wikipedia links accordingly. Some examples of Key Concepts for a COVID-19 related lecture would be medical institutions, famous researchers, and related diseases.

**Front-End**

The front end of our website was set-up with Flask and Bootstrap. This allowed us to quickly and easily integrate our Python scripts and NLP model.

## Challenges we ran into

1. Text summarization is extremely difficult -- while there are many powerful algorithms for turning articles into paragraph summaries, there is essentially nothing on shortening conversational sentences like those found in a lecture into bullet points.
2. Our NLP model is quite large, which made it difficult to host on cloud platforms

## Accomplishments that we're proud of

1) Making a multi-faceted application, with a variety of machine learning and non-machine learning techniques.

2) Working on an unsolved machine learning problem (lecture simplification)

3) Real-time text analysis to determine new elements

## What we learned

1) First time for multiple members using Flask and doing web development

2) First time using Google Cloud Platform API

3) Running deep learning models makes my laptop run very hot

## What's next for Summaread

1) Improve our summarization model through improving data pre-processing techniques and decreasing run time

2) Adding more functionality to generated outlines for better user experience

3) Allowing for users to set parameters regarding how much the lecture is condensed by
## Inspiration

The biggest irony today is despite the advent of the internet, students and adults are more oblivious than ever to world events, and one can easily understand why. Of course, Facebook, YouTube, and League will be more interesting than reading Huffington Post; coupled with the empirical decrease in the attention span of younger generations, humanity is headed towards disaster.

## What it does

Our project seeks to address this crisis by informing people in a novel and exciting way. We create a fully automated news extraction, summarization, and presentation pipeline that involves an AI-anime character news anchor. The primary goal of our project is to engage and educate an audience, especially that of younger students, with an original, entertaining venue for encountering reliable news that will not only foster intellectual curiosity but also motivate them to take into deeper consideration of relevant issues today, from political events to global warming.

The animation is basically a news anchor talking about several recent news, where related news is discussed in a short blurb.

## Demo Video Explanation

The demo video generally performs well, except for the first few seconds and the Putin/Taliban part. This is because the clusters are too small so many clusters get merged together as our kmeans has fixed number of clusters. A quick fix is to simply calculate the internal coherence of the cluster and filter based on that. more advanced methods can be based on those described in the Scatter Gather paper by Karger et al.

## How we built it

### News Summarization

For extraction and summarization, our first web scrapes news articles from trusted sources (CNN, New York Times, Huffington Post, Washington Post, etc…) to obtain the texts of recent news articles. Then it generates a compact summary of these texts using an in-house developed two-tier text summarization algorithm based on state-of-the-art natural language processing techniques. The algorithm first does an extractive summarization of individual articles. Next, it computes an overall 'topic feature' embedding. This embedding is used to cluster related news, and the final script is generated using these clusters and DL-based abstractive summarization.

### News Anchor Animation

Furthermore, using the google cloud text-to-speech API, we generate speech with our custom pitch and preferences and we then have code that generates a video using an image of any interesting, popular anime character. In order for the video to feel natural to the audience, we accounted for accurate lip and facial movement; there are calculations made using specific speech traits of the .wav file that produces realistic and not only educational but also humorous videos that will entertain the younger audience.

### Audience Engagement

Moreover, we wrote code using the Twitter API to automate the process of uploading videos to our Twitter account MinervaNews which is integrated within the project’s server that uploads a video initially when the server starts and automatically generates every 24 hours after a new video using the new articles from the sources.

## What's next for Minerva Daily News Reporter

Our project will have a lasting impact on the education of an audience ranging in all age groups. Anime is one great example of a venue that can broadcast news, and we selected anime characters as a humorous and eye-catching means to educate the younger audience. Our project and its customization allow for the possibility of new venues and greater exploration of making education more fun and accessible to a vast audience. We hope to take our project further and add more animations as well as more features.

## Challenges

Our compute platform, Satori has a unique architecture called IBM ppe64le that makes package and dependency management a nightmare.

## What we learned

8 hours in planning = 24 hours in real time.

## Github

<https://github.com/gtangg12/liszt>
## Inspiration

* Sardor is experienced in making generative AI chatbots for customer service.
* Helen is passionate about art, museums, and literature and making them approachable.
* Realization that large museums, as well as their websites, can be overwhelming for less experienced visitors.

## What it does

An AI that can have natural conversations with you through voice calls on anything you train it with, our example being:

* Your customized tour guide to the Harvard Art Museum that can:
	+ Talk about any object in the collection.
	+ Share any information about museum hours, admission, current exhibitions, events, etc. (that exist on the website).
	+ Provide interesting, intelligent, natural responses with extremely short reaction time.
	+ Give specific answers drawn from the website you train it with, instead of a generic one produced by LLMs.
* Other potential applications being:
	+ Online Student Support.
	+ Company Customer service.
	+ Plato (!) talking to you about different ideas that different characters hold on "love" in his work Symposium.

## How we built it

3 Big Steps.

* Draw from Deepgram to build an ASR application that you can voice call.
* A. Scrape the website of the Harvard Art Museum through its shared API for its object collections and all other information ("Visit," "Event," etc.) and convert them into a vector space.
* B. Retrieve the embeddings that are semantically similar to the question.
* C. Use OpenAI's GPT to put together the retrieved information from the vector into natural human-like language.
* Draw from Microsoft Azure to construct a Speech Generation Model that articulates the response to the user.

## Challenges we ran into

* The AI used to give a generic answer drawing from LLMs instead of information from the specific website.
* Hardcoding the scraper of the website was time-consuming and inefficient, and fortunately, we find access to the API of the museum objects on the website.
* How to shorten the reaction time caused by the turnaround time of the three AI models we use

## Accomplishments that we're proud of

* Extremely Short Reaction Time.
* Human-like answer and relatively accurate response.
* Bridging disciplines in the arts and tech.
* Clear impact on improving user experience in any enterprises we apply the model to

## What we learned

* Deepened our understanding of the speech recognition model, speech generation model, and large language models by constructing.
* Learned to deconstruct a problem into chunks and steps and combine complex language models into an application.
* Experimented with endless creative opportunities at the intersections of different disciplines.

## What's next for Artline

* Contact Harvard Art Museum and solicit support in turning this program into a free accessible product (Helen plans to have an internship at the museum).