## Inspiration

Have you ever wondered or experienced a world without colour? What is life like being unable to see the surroundings, the people and do entertainment? In this case, hearing becomes an essential sense, which provides people with vision disabilities a unique way to experience the world. In addition, with the growing smartphone penetration rate and the forced stay-at-home caused by the pandemic, more people have gradually become isolated from society. As a team, we genuinely believe that social interaction is a crucial factor in achieving better health. Sonux is here to help build both online and offline connectivity, especially for those experiencing blindness and/or visual impairment. Through playing this game, players will not only gain a satisfying acoustic sensory experience but also encourage more people to stay away from the screen and appreciate the scenery of different communities, of diverse cultures and mother nature, leading to better health outcomes, mentally and physically.

## What it does

Sonux is a mobile application designed for anyone looking to relax through screen breaks while gaining a whole experience of auditory sensation. The application is user-friendly, in which people with blindness and/or visual impairment will achieve the same experience as other players. Once players launch the game, the phone’s screen will be in a darkened mode, and players can barely see any displays. The only way to navigate and select game options is through swiping, knocking, or simply waving their phones. As players explore around, Sonux will sense their surrounding environment and search for sounds. At distinct locations, Sonux is able to capture different soundtracks; it can be a lyric of a song, a piece of classical music, the sound of nature or a message from someone unknown. Players are, therefore, able to store all the sounds that they encountered during their journey. As users explore more places, travel to further locations, they will discover diverse sounds. The game enables players to expand their maps to other parts of the world and unlock more soundtracks. Furthermore, players are encouraged to leave warm and positive messages that other users will collect. With multiple sounds collected, users could always return to their sound archive, re-listen to all the sounds, and make various combinations of harmonies.

## How we built it

Unity Engine
Sonux is developed based on the Unity engine to enable the best 3-D acoustic experience. We started by creating a 3-D space with natural objects modelled inside. Each element has its unique sound and lighting effects, guiding the player's journey. Unlike traditional games where navigation is achieved through screen touches, Sonux utilizes hand gestures, device movement and haptic feedback. By creating a 3-D environment for players to explore, Doppler Effects is achieved virtually by varying sound channels and volume levels, maximizing Sonux's acoustic experience.

Natural Language Processing
We employed Unity's voice recognition API with a sentiment analysis model to analyze players' appropriateness of message input. After translating the player's voice message into texts, we process the text message using the textBlob API, a pre-trained NLP model, to determine a sentiment score of the message. To factor in the potential inconsistency of the model, messages indicating harmful contents will be filtered and further reviewed by our staff.

## Challenges we ran into

One of the biggest challenges we ran into was in our application of Unity. Unity is the most suitable real-time game development platform in our case to build the audio game for mobile devices. However, our group has encountered several debugging problems as we lack previous experience and skill set with this tool. We have gone through multiple tutorials, navigated the platform thoroughly, and self-taught the fundamental skills needed to develop Sonux. Nevertheless, when coding for the game to be able to listen and react to voice instructions, we failed the first few trials. It resulted from Mac laptops not supporting certain sound features. Thus, we switched to a PC laptop and adjusted our code to successfully make the voice recognition function. Recognizing this bug was one of the big moments for the project.

## Accomplishments that we're proud of

The biggest accomplishment we believe is that as a team, we have worked well together. With some members from a tech background and another from a media information major, we had very thoughtful and insightful discussions in the project brainstorming stage. In which all members have identified and agreed upon the need for mobile games to be more inclusive to the special player group and be more considerate towards players' health. Through working cohesively, we have turned our idea from sketch to an actual presentable model that is able to include all the features we have drafted.

## What we learned

We learned how to use Unity, how the platform works, and the foundation to make mobile games. Members with a tech background have gained a better understanding of the coding process for game development and how to navigate game engines. For the member who does not have previous tech/science knowledge, it was a fascinating experience to see how the games played throughout daily lives come into being. In addition, all members have gained well-rounded perspectives by seeing how different people think and approach problem-solving differently.

## What's next for SONUX

Sonux comes alive after evaluating both the practical components and considering the ethical and social responsibility that a mobile game company should take on to achieve long-term sustainability. Sonux hopes to raise more attention and care to the visual disability population while also creating a more healthy lifestyle for the ever-increasing mobile game players. With these goals in mind, Sonux will continue to build a more connected, long-lasting and inclusive player community.
## Inspiration

Open-world AR applications like Pokemon Go that bring AR into everyday life and the outdoors were major inspirations for this project. Additionally, in thinking about how to integrate smartphones with Spectacles, we found inspiration in video games like Phasmophobia's EMP sensors that react more strongly in the presence of ghosts or The Legend of Zelda: Skyward Sword that contained an in-game tracking functionality that pulsates more strongly when facing the direction of and walking closer to a target.

## What it does

This game integrates the Spectacles gear and smartphones together by allowing users to leverage the gyroscopic, haptic, and tactile functionalities of phones to control or receive input about their AR environment. In the game, users have to track down randomly placed treasure chests in their surrounding environment by using their phone as a sensor that begins vibrating when the user is facing a treasure and enters stronger modes of haptic feedback as users get closer to the treasure spots.

These chests come in three types: monetary, puzzle, and challenge. Monetary chests immediately give users in-game rewards. Puzzle chests engage users in a single-player mini-game that may require cognitive or physical activity. Finally, challenge chests similarly engage users in activities not necessarily games, and a stretch goal for multiplayers was that if multiple users were near a spot that another user found a treasure in, the other n users could challenge the treasure finder in a n vs. 1 duel, with the winner(s) taking the rewards.

## How we built it

Once we figured out our direction for the project, we built a user flow architecture in Figma to brainstorm the game design for our application ([link](https://www.figma.com/design/pyG5hlpYkWwVcyvIQJCnY3/Treasure-Hunt-UX-Architecture?node-id=0-1&node-type=canvas&t=clqInR0JpOM6tEnv-0)), and we also visualized how to implement the system for integrating phone haptic feedback with the spectacles depending on distance and directional conditions.

From there, we each took on specific aspects of the user flow architecture to primarily work on: (1) the treasure detection mechanism, (2) spawning the treasure once the user entered within a short distance from the target, and (3) the content of the treasure chests (i.e. rewards or activities). Nearly everything was done using in-house libraries, assets, and the GenAI suite within Snap's Lens Studio.

## Challenges we ran into

As we were working with Spectacles for the first time (compounded with internet problems), we initially encountered technical issues with setting up our development environment and linking the Spectacles for debugging. Due to limited documentation and forums since it is limited-access technology, we had to do a lot of trial-and-error and guessing to figure out how to get our code to work, but luckily, Snap's documentation provided templates to work off of and the Snap staff was able to provide technical assistance to guide us in the right direction. Additionally, given one Spectacle to work with, parallelizing our development work was quite challenging as we had to integrate everything onto one computer while dealing with merge conflicts between our code.

## Accomplishments that we're proud of

In a short span of time, we were able to successfully build a game that provides a unique immersive experience! We've come across and solved errors that didn't have solutions on the internet. For a couple of members of our team, this sparks a newfound interest in the AR space.

## What we learned

This was our first time working with Lens Studio and it's unanimously been a smooth and great software to work with.
For the experienced members on our team, it's been a rewarding experience to make an AR application using JS/TS instead of C# which is the standard language used in Unity.

## What's next for SnapChest

We're excited to push this app forward by adding more locations for treasures, implementing a point system, and also a voice agent integration that provides feedback based on where you're going so you won't get bored on your journey!
If Spectacles would be made available to the general public, a multiplayer functionality would definitely gain a lot of traction and we're looking forward to the future!
## Inspiration

As we sat down to brainstorm ideas for our next project, we were struck by a common thread that connected all of us. Each one of us had a family member who suffered from some form of visual impairment. It was a heart-wrenching reminder of the challenges that these individuals face on a daily basis. We shared stories of our loved ones struggling to read books, watch movies, or even navigate through everyday tasks. It was a deeply emotional conversation that left us feeling both empathetic and determined to make a difference. According to the World Health Organization, approximately 2.2 billion people worldwide have a vision impairment or blindness, with the majority of cases occurring in low and middle-income countries. The impact of visual impairment is far-reaching and significantly affects various daily activities such as reading, recognizing faces, navigating unfamiliar environments, and accessing information on digital platforms. This problem is valid, and it needs to be addressed to enhance the quality of life of those affected. We are passionate about developing a solution that will make a meaningful difference in the lives of those affected by visual impairment. Our project is inspired by personal experiences and fueled by a desire to make a real-world impact. We believe that everyone deserves equal access to information and the ability to participate fully in daily life. By addressing the challenges of visual impairment, we hope to create a more inclusive world for all.

## What it does

The product aims to bridge the gap for individuals with limited vision to experience the world around them. It helps individuals with visual impairments to perform various daily activities that are otherwise challenging, such as reading, recognizing faces, and navigating unfamiliar environments. It also assists in accessing information on digital platforms. The product can be particularly helpful for those who face barriers in accessing healthcare services due to their visual impairments. It can aid in reading prescription labels, understanding medical instructions, and navigating healthcare facilities, especially for older individuals who are aging.

## How we built it

In our project, we leverage cutting-edge computer vision techniques to interpret the surrounding environment of individuals with visual impairments. By utilizing advanced algorithms and neural networks, we process real-time visual data captured by a camera, enabling us to identify and analyze objects, obstacles, and spatial cues in the user's surroundings. We integrate state-of-the-art language models and natural language generation powered by Wisp AI software to bridge the gap between the interpreted world and the user. This allows us to generate detailed and contextually relevant descriptions of the environment in real time, providing visually impaired individuals with comprehensive auditory feedback about their surroundings. Additionally, our solution extends beyond descriptive capabilities to enhance accessibility in public transportation. By leveraging the interpreted environmental data, we develop guidance systems that assist users in navigating through streets and accessing transportation hubs safely and independently. For efficient and scalable deployment of our model, we utilize Intel's AI environment, leveraging its robust infrastructure and resources to host and optimize our machine learning algorithms. Our system architecture is implemented on a Raspberry Pi embedded platform, equipped with a high-resolution camera for real-time visualization and data capture. This combination of hardware and software components enables seamless integration and efficient visual information processing, empowering visually impaired individuals with enhanced mobility and independence in their daily lives.

## Challenges we ran into

As beginners in machine learning, we faced the tough challenge of setting up a machine learning model on a Raspberry Pi and connecting it to a camera, which was quite difficult to learn. Moreover, we had to figure out a way to train our model not only to understand text but also to recognize public transportation and calculate the distance to a bus entrance, which was quite a task. Adding our Intel-AI environment to the project made things even more complicated. Additionally, finding an affordable solution that could be easily accessible to people all around the world was a significant obstacle that we had to overcome.

## Accomplishments that we're proud of

Through this process of building a hardware product from scratch and learning how to use raspberry pi with computer vision, we not only gained technical knowledge but also learned how to work as a team. There were challenges and obstacles along the way, but we figured it out by collaborating, communicating, and leveraging each other's strengths. It was a great learning experience, and we are proud of what we have achieved together.

## What we learned

We learned about LLM, real-time text analysis, real-time text comprehension, and implementation of text-to-speech.

## What's next for True-Sight

With the growing potential of Artificial Intelligence, our idea of True-Sight is expanding to include not only text recognition but also the ability to detect surroundings, which could greatly benefit public transportation users who rely on finding stops and navigating their way onto the correct buses/trains. After further development, True-Sight could potentially allow users to locate their desired stop and use environment detection to guide them towards the door with specific step-by-step instructions. In addition, we aim to make True-Sight accessible to children who are visually impaired, so they can have an immersive learning experience. Adding sensors and custom software will also allow for a more personalized and relatable experience with the AI assistant.