## Inspiration

Our idea was inspired by our group's shared interest in musical composition, as well as our interests in AI models and their capabilites. The concept that inspired our project was: "*What if life had a soundtrack?*"

## What it does

AutOST generates and produces a constant stream of original live music designed to automatically adjust to and accompany any real-life scenario.

## How we built it

We built our project in python, using the Mido library to send note signals directly to FL studio, allowing us to play constant audio without a need to export to a file. The whole program is linked up to a live video feed that uses Groq AI's computer vision api to determine the mood of an image and adjust the audio accordingly.

## Challenges we ran into

The main challenge we faced in this project is the struggle that came with making the generated music not only sound coherent and good, but also have the capability to adjust according to parameters. Turns out that generating music mathematically is more difficult than it seems.

## Accomplishments that we're proud of

We're proud of the fact that our program's music sounds somewhat decent, and also that we were able to brainstorm a concept that (to our knowlege) has not really seen much experimentation.

## What we learned

We learned that music generation is much harder than we initially thought, and that AIs aren't all that great at understanding human emotions.

## What's next for AutOST

If we continue work on this project post-hackathon, the next steps would be to expand its capabilities for recieving input, allowing it to do all sorts of amazing things such as creating a dynamic soundtrack for video games, or integrating with smart headphones to create tailored background music that would allow users to feel as though they are living inside a movie.
## Intro and Idea

For our team of First Year UofT Engineering Science students, this was our first Makeathon and first project as a team. We have varying ranges of experience with software and hardware within our team and decided to approach this competition as both a challenge, and a learning experience.

After a couple hours of brainstorming based on our collective interests, our team arrived on an idea we were all excited about: An interactive orchestra experience to allow players to more easily play together. Jazz ensembles are easily able to improvise together, because they usually play in the same keys. Classical musicians on the other hand, are often not able to predict key changes.

Our design provides a platform for conductors to change the orchestra in real-time according to their vision. By playing chords on a Midi keyboard, they can “play the orchestra” by transposing and transmitting the chords to the members of the orchestra through the use of wireless connectivity to individual displays powered by Raspberry Pi’s.

## Planning and Summary

As briefly identified in our introduction, our primary stakeholders for this project are:

Ourselves (a team of first year engineering students attempting their first Makeathon)
Orchestra performers and conductors
MakeUofT Organizers, Sponsors, and Judges

Based on this, we were able to develop some rough objectives to keep us on track for 24 hours:

To create a unique but achievable product
To improve our software and hardware integration skills
To incorporate sponsor innovations and technologies

To ensure we had something to show after 24 hours, we decided to aim for a minimum viable project (MVP) before adding any bells and whistles. We were able to reach our goals of Midi communication to the serving computer and having note name communication to the player displays. After we reached our MVP, we expanded on our design to have visualization of the notes on the staff, differing transposition options, and automated chord analysis. Finally, we implemented chord suggestion and prediction using Azure Machine Learning.

## Features:

Real time note communication between Midi and player displays
Visual display of notes on staff
Automated Chord Analysis
Chord suggestion and predictions using Azure Machine Learning
Multiple differently transposed sections available to accommodate a variety of instruments simultaneously

## Applications:

Large group improvisation and composition
Teaching and training
Creating new pieces using Azure Machine Learning

## The Process

## Raspberry Pi Setup and Enclosure:

To act as the receiving devices, we use four Raspberry Pi’s in our project. Each Pi is set up with Raspbian Stretch version 4.14.

A 7” touch display screen is attached to one of our Pi’s, and monitors to the remaining three. Initially, we were going to use small LCD graphic displays, but ruled these out due to size. A 7” touch display was offered to us to borrow, but to keep our cost down, we opted to use monitors for the remaining Pi’s. Ideally all the Pi’s would have 7” touch display screens, but we decided for a MVP prototype, one was sufficient.

Beyond our MVP prototype we added a push-button that allows the player to cycle through the available sections (more on sections to follow) on the Raspberry Pi. A simple python script was created to map the GPIO pin connected to the button to a keyboard stroke (the F5 refresh button for web pages). We left room in the box for more features to be added.

We initially hoped to modify the open source PIvena Raspberry Pi enclosure, but after beginning our modifications, we realized laser cutting was not being offered as a service at MakeUofT. Based on limited fabrication lab hours available to us, we opted to design our enclosure out of foamcore.

The display is set at an angle in order to allow notes to be read easily while playing an instrument. Hardware is located in a box behind the display, making it discrete but easily accessible.

## Network/IoT Setup:

The Network setup remained simple throughout the project. The basic concept was to have one computer that acted as a master, then any number of displays of all shapes and sizes that could receive instructions from the master and join in on the joys of music. To accomplish this, we used a Node.js framework and wrote predominantly JavaScript to manage the interactivity of different clients. As a result of our objectives, the final product that we have produced is capable of being run on any platform with a web browser, making it highly accessible and scalable. Furthermore, as the result works off of a wireless network, it is capable of accepting a high volume of hosts without added latency, as well as being very easy to connect to. The network setup has been geared towards the IoT model by connecting devices in a collaborative way to enable people to help and support each other in harmony.

Technically speaking, the socket works using Socket.io, integrating native HTTPS for requests to the Azure Cloud for learned suggestions of chords based on machine intelligence. The socketing breaks the network into a series of sections, which may all operate and play in different keys, requiring transposition for accessible harmony. The number of sections that can be created is theoretically undefined, though for our basic demonstration we make use of four different sections operating in different keys.They are all shown the classic chord data and have available to them the key in which the orchestra is playing giving the player further liberty for experimentation within the piece.

## Machine Learning:

The Azure Machine Learning framework was the center of a feature for the master controller of the project. Provided with historical data of chords played, it would recommend a good follow-up chord to harmonize. Our machine learning algorithm was fed by approximately 3 million data points from pop songs. Though in retrospect we should have trained it with music that finds more standard and less repetitive chords, the concept still worked well enough, though at our hand there was slight underfitting. The structure that worked well for our means was the feeding of 3 points of historical data predicting a fourth point of data to a fair degree of reason. This is a useful feature of our design for those who would use this concept to collaborate with others or create something new, as it would support them further in a desire for a good, harmonic sound.
## Inspiration

As music lovers, we found that certain songs trigger fond memories of the past. To capture that nostalgic essence, we developed a Retro AI BoomBox. This unique tool helps users rediscover tunes that transport them back in time, all while merging the charm of the olden years with modern technology.

## What it does

Our AI BoomBox is ultimately a music guru! BoomBot is equipped with tools to help users find new music through either a) a conversation with BoomBot with help from Cohere API, or b) accessing the user's Spotify data via Spotify API to curate a nostalgic playlist for the user based on their listening history.

## How we built it

We designed the user interface using Figma, heavily considering elements such as font, colour, and images to evoke that nostalgic feeling. We built the front-end using React.js and CSS, while our backend composed of MongoDB and Flask. The Cohere API was essential in creating BoomBot's chat feature, and the Spotify API gave us the necessary data to recommend users nostalgic songs based on their listening history.

## Challenges we ran into

Merge conflicts: Throughout our development process, we encountered challenges with merge conflicts, where multiple developers attempted to modify the same code simultaneously. Resolving these conflicts required careful coordination and communication among team members to ensure that all changes were integrated smoothly.

Problems with our original tech stack (Taipy): Initially, we faced issues with our chosen technology stack, Taipy. These challenges ranged from compatibility issues to limitations in functionality, as well as lack of documentation, prompting us to reevaluate our approach and make necessary adjustments to ensure the smooth progress of our project.

How to work and participate in a hackathon (this is our first in-person hackathon!): As first-time participants in an in-person hackathon, we encountered a learning curve in understanding how to effectively collaborate within the intense and time-constrained environment of a hackathon. Overcoming this challenge involved adapting quickly to the fast-paced nature of the event, prioritizing tasks, and leveraging each team member's strengths to maximize productivity and innovation.

The slowness of Azure and Cohere: Throughout the development process, we experienced delays due to the sluggish performance of Azure and Cohere, hindering our ability to efficiently execute certain tasks and slowing down our overall progress. Despite these challenges, we employed strategies such as optimizing our workflows and seeking alternative solutions to mitigate the impact of these performance issues and keep our project on track.

## Accomplishments that we're proud of

We finished our project with the goals we had in mind!: One of our proudest accomplishments is successfully completing our project while achieving the specific objectives we set out to accomplish. Despite the challenges and obstacles we faced along the way, our team remained focused and determined, ultimately delivering a product that aligns with our initial vision and goals.

Experimented and used up to 10 different technologies: Throughout the development process, we embraced a spirit of experimentation and innovation by incorporating and leveraging up to 10 different technologies. This breadth of exploration not only expanded our technical expertise but also allowed us to discover new tools and approaches that enhanced the functionality and robustness of our project.

Text to speech feature: One standout achievement of our project is the successful implementation of a text-to-speech feature. This functionality not only adds value to our product but also demonstrates our team's ability to integrate advanced capabilities into our project, enhancing its usability and accessibility for users.

Expanded our skills and built experiences for our resume: Engaging in this project provided us with invaluable opportunities to expand our skill sets and gain practical experience, enriching our resumes and bolstering our professional profiles. From honing technical proficiencies to refining collaboration and problem-solving skills, each team member has grown personally and professionally through their contributions to the project.

Developed the front-end close to its original design: We take pride in our meticulous attention to detail and dedication to delivering a front-end interface that closely resembles its original design. By prioritizing user experience and adhering to design principles, we ensured that our project not only functions seamlessly but also boasts an aesthetically pleasing and intuitive user interface.

## What we learned

Using Spotify API with Python: We gained proficiency in leveraging the Spotify API with Python, allowing us to access and manipulate Spotify's vast music database for our project. This experience deepened our understanding of API integration and expanded our capabilities in working with external data sources.

Flask: Through our project development, we familiarized ourselves with Flask, a lightweight and flexible Python web framework. Learning Flask enabled us to build web applications efficiently and effectively, providing us with a valuable toolset for future web development projects.

Setting up GitHub Actions with Azure: We acquired knowledge and skills in configuring GitHub Actions to automate our workflow processes, coupled with deploying our application on the Azure cloud platform. This integration streamlined our development pipeline and enhanced collaboration among team members, reinforcing the importance of automation in modern software development practices.

React: Our experience with React, a popular JavaScript library for building user interfaces, allowed us to create dynamic and interactive front-end components for our project. By mastering React, we gained insights into component-based architecture and learned best practices for building scalable and maintainable web applications.

Figma: We explored Figma, a collaborative interface design tool, to create and prototype the visual elements of our project. Working with Figma enhanced our design skills and facilitated seamless collaboration among team members, enabling us to iterate and refine our design concepts efficiently throughout the development process.

## What's next for BoomBot

Inclusion of users without Spotify accounts: We plan to expand the accessibility of BoomBot by accommodating users who do not have a Spotify account. This could involve integrating additional music streaming services or providing alternative methods for accessing and enjoying the features offered by BoomBot.

Responsiveness: Enhancing the responsiveness of BoomBot across various devices and screen sizes is a priority. By optimizing the user experience for desktops, tablets, and mobile devices, we aim to ensure that BoomBot remains functional and visually appealing regardless of the platform used.

Multi-language support: To cater to a diverse user base, we intend to implement multi-language support within BoomBot. This will involve translating the user interface and content into multiple languages, allowing users from different regions and linguistic backgrounds to interact with BoomBot in their preferred language.

Attention to small details like animations and transitions: We recognize the importance of refining the user experience by paying attention to small details such as animations and transitions. Adding subtle yet engaging animations and transitions throughout the interface will elevate the overall look and feel of BoomBot, contributing to a more immersive and enjoyable user experience.