## Bringing your music to life, not just to your ears but to your eyes 🎶

## Inspiration 🍐

Composing music through scribbling notes or drag-and-dropping from MuseScore couldn't be more tedious. As pianists ourselves, we know the struggle of trying to bring our impromptu improvisation sessions to life without forgetting what we just played or having to record ourselves and write out the notes one by one.

## What it does 🎹

Introducing PearPiano, a cute little pear that helps you pair the notes to your thoughts. As a musician's best friend, Pear guides pianists through an augmented simulation of a piano where played notes are directly translated into a recording and stored for future use. Pear can read both single notes and chords played on the virtual piano, allowing playback of your music with cascading tiles for full immersion. Seek musical guidance from Pear by asking, "What is the key signature of C-major?" or "Tell me the notes of the E-major diminished 7th chord." To fine tune your compositions, use "Edit mode," where musicians can rewind the clip and drag-and-drop notes for instant changes.

## How we built it 🔧

Using Unity Game Engine and the Oculus Quest, musicians can airplay their music on an augmented piano for real-time music composition. We used OpenAI's Whisper for voice dictation and C# for all game-development scripts. The AR environment is entirely designed and generated using the Unity UI Toolkit, allowing our engineers to realize an immersive yet functional musical corner.

## Challenges we ran into 🏁

* Calibrating and configuring hand tracking on the Oculus Quest
* Reducing positional offset when making contact with the virtual piano keys
* Building the piano in Unity: setting the pitch of the notes and being able to play multiple at once

## Accomplishments that we're proud of 🌟

* Bringing a scaled **AR piano** to life with close-to-perfect functionalities
* Working with OpenAI to synthesize text from speech to provide guidance for users
* Designing an interactive and aesthetic UI/UX with cascading tiles upon recording playback

## What we learned 📖

* Designing and implementing our character/piano/interface in 3D
* Emily had 5 cups of coffee in half a day and is somehow alive

## What's next for PearPiano 📈

* VR overlay feature to attach the augmented piano to a real one, enriching each practice or composition session
* A rhythm checker to support an aspiring pianist to stay on-beat and in-tune
* A smart chord suggester to streamline harmonization and enhance the composition process
* Depth detection for each note-press to provide feedback on the pianist's musical dynamics
* With the up-coming release of Apple Vision Pro and Meta Quest 3, full colour AR pass-through will be more accessible than ever — Pear piano will "pair" great with all those headsets!
## Inspiration

Our inspiration comes from the idea that the **Metaverse is inevitable** and will impact **every aspect** of society.

The Metaverse has recently gained lots of traction with **tech giants** like Google, Facebook, and Microsoft investing into it.

Furthermore, the pandemic has **shifted our real-world experiences to an online environment**. During lockdown, people were confined to their bedrooms, and we were inspired to find a way to basically have **access to an infinite space** while in a finite amount of space.

## What it does

* Our project utilizes **non-Euclidean geometry** to provide a new medium for exploring and consuming content
* Non-Euclidean geometry allows us to render rooms that would otherwise not be possible in the real world
* Dynamically generates personalized content, and supports **infinite content traversal** in a 3D context
* Users can use their space effectively (they're essentially "scrolling infinitely in 3D space")
* Offers new frontier for navigating online environments
	+ Has **applicability in endless fields** (business, gaming, VR "experiences")
	+ Changing the landscape of working from home
	+ Adaptable to a VR space

## How we built it

We built our project using Unity. Some assets were used from the Echo3D Api. We used C# to write the game. jsfxr was used for the game sound effects, and the Storyblocks library was used for the soundscape. On top of all that, this project would not have been possible without lots of moral support, timbits, and caffeine. 😊

## Challenges we ran into

* Summarizing the concept in a relatively simple way
* Figuring out why our Echo3D API calls were failing (it turned out that we had to edit some of the security settings)
* Implementing the game. Our "Killer Tetris" game went through a few iterations and getting the blocks to move and generate took some trouble. Cutting back on how many details we add into the game (however, it did give us lots of ideas for future game jams)
* Having a spinning arrow in our presentation
* Getting the phone gif to loop

## Accomplishments that we're proud of

* Having an awesome working demo 😎
* How swiftly our team organized ourselves and work efficiently to complete the project in the given time frame 🕙
* Utilizing each of our strengths in a collaborative way 💪
* Figuring out the game logic 🕹️
* Our cute game character, Al 🥺
* Cole and Natalie's first in-person hackathon 🥳

## What we learned

### Mathias

* Learning how to use the Echo3D API
* The value of teamwork and friendship 🤝
* Games working with grids

### Cole

* Using screen-to-gif
* Hacking google slides animations
* Dealing with unwieldly gifs
* Ways to cheat grids

### Natalie

* Learning how to use the Echo3D API
* Editing gifs in photoshop
* Hacking google slides animations
* Exposure to Unity is used to render 3D environments, how assets and textures are edited in Blender, what goes into sound design for video games

## What's next for genee

* Supporting shopping
	+ Trying on clothes on a 3D avatar of yourself
* Advertising rooms
	+ E.g. as your switching between rooms, there could be a "Lululemon room" in which there would be clothes you can try / general advertising for their products
* Custom-built rooms by users
* Application to education / labs
	+ Instead of doing chemistry labs in-class where accidents can occur and students can get injured, a lab could run in a virtual environment. This would have a much lower risk and cost.
	…the possibility are endless
## Inspiration

Our inspiration comes from the desire to democratize music creation. We saw a gap where many people, especially youth, had a passion for making music but lacked the resources, knowledge, or time to learn traditional instruments and complex DAW software. We wanted to create a solution that makes music creation accessible and enjoyable for everyone, allowing them to express their musical ideas quickly and easily.

## What it does

IHUM is an AI application that converts artists' vocals or hums into MIDI instruments. It simplifies the music creation process by allowing users to record their vocal melodies and transform them into instrumental tracks with preset instruments. This enables users to create complex musical pieces without needing to know how to play an instrument or use a DAW.

## How we built it

For the frontend, we used HTML, CSS and React JS to develop IHUM. Using React JS and its libraries such as Pitchy, we were able to process, change, and output the sound waves of audio inputs. For the backend, we used Auth0's API to create a login/signup system, which stores and verifies user emails and passwords.

## Challenges we ran into

One of the main challenges we faced was ensuring the AI's accuracy in interpreting vocal inputs and converting them into MIDI data that sounds natural and musical. Furthermore, the instruments that we converted had a ton of issues in how it sounds, especially regarding pitch, tone, etc. However, we were able to pull and troubleshoot through our way through most of them.

## Accomplishments that we're proud of

Through all the hours of hard work and effort, an accomplishment we are extremely proud of is the fact that our program is able to process the audio. By using Pitchy JS, we were able to change the audio to fit how we want it to sound. On top of this, we are also proud that we were able to implement a fully working login/signup system using Auth0's API and integrate it within the program.

## What we learned

As this was our first time working with audio in web development and many of our group's first hackathons, we faced many issues that we had to overcome and learn from. From processing to setting up APIs to modifying the sound waves, it definitely provided us valuable insight and expanded our skillsets.

## What's next for IHUM

Our future updates allow running multiple audio samples simultaneously and increasing our instrument libraries. By doing so, IHUM can essentially be a simple and easy-to-use DAW which permits the user to create entire beats out of their voice on our web application.