## Team

Isaiah Ballah (full-stack), Will Hutchinson (full-stack), Evan Hutnik (back-end), Aras Siddiqui (designer).

## Inspiration

Bullying has always been a big issue in schools, and unfortunately this issue has carried over into today’s world of online learning. While teachers try their best, they can’t be everywhere and hear everything that goes on in their online classrooms. One of our group members has a younger sibling that has been directly affected by in-class cyberbullying which gave us the motivation to pursue this idea.

## What it does

Our project notifies educators of any hurtful or harassing messages sent during their online lectures. After each class finishes, teachers are sent an email detailing text and audio communications that were flagged by our software, which they are able to review, playback, and save in our user interface. They also have the ability to escalate messages to colleagues like vice principals if they feel disciplinary action is necessary.

## How we built it

We started off with brainstorming ideas and making a quick low-fed mockup. We discussed the key features we wanted it to have and made some wireframes as a proof of concept. This was based of different software's already used in education. This is so the teachers are familiar with the UI. Then we did our High Fidelity in Adobe XD.

Our software consists of two main parts, a backend and a user interface. Our backend performs sentiment analysis on messages from the teachers online classroom using machine learning algorithms. It then stores the messages that score above a threshold in a database for the teacher to review. Once processing of the messages has been completed, an email is sent to the instructor with a summary of the analysis as well as a link to view any flagged messages from that day’s lecture.

In the user interface the teacher has the ability to review, playback, and save flagged messages for each of their classes. The student’s home phone number and an option to send the message to a superior are readily available if they feel the need to take further action

## Challenges we ran into

While our group has a large amount of technical experience, the complexity of the software prevented us from implementing all of the functionality that we initially planned to with the time given. Some of our team members had never used Node before to create a backend, so there was a learning curve to get familiar with it.

## Accomplishments that we're proud of

When we started this project we set out to create something that could make an impact on the issue of cyberbullying in online classes. We truly feel like this software is an effective tool to curb this issue, and we are proud of the functionality implemented. Our backend seamlessly communicates with Google Cloud's Natural Language API to perform sentiment analysis and delivers the relevant results to the database for use by the user interface, and our frontend is both clearly laid out and functional.

The dashboard has a resources tab that displays cyberbullying resources for teachers from Telus Wise.
## Inspiration

In high school, a teacher of ours used to sit in the middle of the discussion to draw lines from one person to another on paper to identify the trends of the discussion. It's a very meaningful activity as we could see how balanced the discussion is, allowing us to highlight to people who had less chance to express their ideas. It could also be used by teacher in earlier education, to identify social challenges such as anxiety, speech disorder in children.

## What it does

The app initially is trained on a short audio from each of the member of the discussion. Using transfer learning, it will able to recognize the person talking. And, during discussion, very colorful and aesthetic lines will began drawing from person to another on REAL-TIME!

## How we built it

On the front-end, we used react and JavaScript to create a responsive and aesthetic website. Vanilla css (and a little bit of math, a.k.a, Bézier curve) to create beautiful animated lines, connecting different profiles.
On the back-end, python and tensorflow was used to train the AI model. First, the audios are pre-processed into smaller 1-second chunks of audio, before turning them into a spectrogram picture. With this, we performed transfer learning with VGG16, to extract features from the spectrograms. Then, the features are used to fit a SVM model, using scikit-learn. Subsequently, the back-end opens a web-socket with the front-end to receive stream of data, and return label of the person talking. This is also done with multi-threading to ensure all the data is being processed quickly.

## Challenges we ran into

As it's our first time with deep-learning or training an AI for that matter, it was very difficult to get started. Despite the copious amount of resources and projects done, it was hard to identify a suitable source. Different processing techniques were also needed to know, before the model could be trained. In additions, finding a platform (such as Google Colab) was also necessary to train the model in appropriate time. Finally, it was fairly hard to incorporate the project with the rest of the project. It needs to be able to process the data in real-time, while keeping the latency low.

Another major challenge that we ran into was connecting the back-end with the front-end. As we wanted it to be real-time, we had to be able to stream the raw data to the back-end. But, there were problems reconstructing the binary files into appropriate format, because we were unsure what format RecordRTC uses to record audio. There was also a problem of how much data or how frequent the data should be sent over due to our high latency of predicting (~500ms). It's a problem that we couldn't figure out in time

## Accomplishments that we're proud of

The process of training the model was really cool!!! We could never think of training a voice recognition model similar to how you would to to an image/face recognition. It was a very out-of-the-box method, that we stumbled up online. It really motivated us to get out there and see what else. We were also fairly surprised to get a proof-of-concept real-time processing with local audio input from microphone. We had to utilize threading to avoid overflowing the audio input buffer. And if you get to use threading, you know it's a cool project :D.

## What we learned

Looking back, the project was quite ambitious. BUT!! That's how we learned. We learned so much about training machine learning as well as different connection protocols over the internet. Threading was also a "wet" dream of ours, so it was really fun experimenting with the concept in python.

## What's next for Hello world

The app would be much better on mobile. So, there are plans to port the entire project to mobile (maybe learning React Native?). We're also planning on retrain the voice recognition model with different methods, and improve the accuracy as well as confidence level. Lastly, we're planning on deploying the app and sending it back to our high school teacher, who was the inspiration for this project, as well as teachers around the world for their classrooms.

## Sources

These two sources helped us tremendously in building the model:

<https://medium.com/@omkarade9578/speaker-recognition-using-transfer-learning-82e4f248ef09>

<https://towardsdatascience.com/automatic-speaker-recognition-using-transfer-learning-6fab63e34e74>
## Inspiration

The inspiration for our Auto-Teach project stemmed from the growing need to empower both educators and learners with a **self-directed and adaptive** learning environment. We were inspired by the potential to merge technology with education to create a platform that fosters **personalized learning experiences**, allowing students to actively **engage with the material while offering educators tools to efficiently evaluate and guide individual progress**.

## What it does

Auto-Teach is an innovative platform that facilitates **self-directed learning**. It allows instructors to **create problem sets and grading criteria** while enabling students to articulate their problem-solving methods and responses through text input or file uploads (future feature). The software leverages AI models to assesses student responses, offering **constructive feedback**, **pinpointing inaccuracies**, and **identifying areas for improvement**. It features automated grading capabilities that can evaluate a wide range of responses, from simple numerical answers to comprehensive essays, with precision.

## How we built it

Our deliverable for Auto-Teach is a full-stack web app. Our front-end uses **ReactJS** as our framework and manages data using **convex**. Moreover, it leverages editor components from **TinyMCE** to provide student with better experience to edit their inputs. We also created back-end APIs using "FastAPI" and "Together.ai APIs" in our way building the AI evaluation feature.

## Challenges we ran into

We were having troubles with incorporating Vectara's REST API and MindsDB into our project because we were not very familiar with the structure and implementation. We were able to figure out how to use it eventually but struggled with the time constraint. We also faced the challenge of generating the most effective prompt for chatbox so that it generates the best response for student submissions.

## Accomplishments that we're proud of

Despite the challenges, we're proud to have successfully developed a functional prototype of Auto-Teach. Achieving an effective system for automated assessment, providing personalized feedback, and ensuring a user-friendly interface were significant accomplishments. Another thing we are proud of is that we effectively incorporates many technologies like convex, tinyMCE etc into our project at the end.

## What we learned

We learned about how to work with backend APIs and also how to generate effective prompts for chatbox. We also got introduced to AI-incorporated databases such as MindsDB and was fascinated about what it can accomplish (such as generating predictions based on data present on a streaming basis and getting regular updates on information passed into the database).

## What's next for Auto-Teach

* Divide the program into **two mode**: **instructor** mode and **student** mode
* **Convert Handwritten** Answers into Text (OCR API)
* **Incorporate OpenAI** tools along with Together.ai when generating feedback
* **Build a database** storing all relevant information about each student (ex. grade, weakness, strength) and enabling automated AI workflow powered by MindsDB
* **Complete analysis** of student's performance on different type of questions, allows teachers to learn about student's weakness.
* **Fine-tuned grading model** using tools from Together.ai to calibrate the model to better provide feedback.
* **Notify** students instantly about their performance (could set up notifications using MindsDB and get notified every day about any poor performance)
* **Upgrade security** to protect against any illegal accesses