**Inspiration**
The inspiration behind Block Touch comes from the desire to create an interactive and immersive experience for users by leveraging the power of Python computer vision. We aim to provide a unique and intuitive way for users to navigate and interact with a simulated world, using their hand movements to place blocks dynamically.

**What it does**
Block Touch utilizes Python computer vision to detect and interpret hand movements, allowing users to navigate within a simulated environment and place blocks in a virtual space. The application transforms real-world hand gestures into actions within the simulated world, offering a novel and engaging user experience.

**How we built it**
We built Block Touch by combining our expertise in Python programming and computer vision. The application uses computer vision algorithms to analyze and interpret the user's hand movements, translating them into commands that control the virtual world. We integrated libraries and frameworks to create a seamless and responsive interaction between the user and the simulated environment.

**Challenges we ran into**
While developing Block Touch, we encountered several challenges. Fine-tuning the computer vision algorithms to accurately recognize and interpret a variety of hand movements posed a significant challenge. Additionally, optimizing the application for real-time responsiveness and ensuring a smooth user experience posed technical hurdles that we had to overcome during the development process.

**Accomplishments that we're proud of**
We are proud to have successfully implemented a Python computer vision system that enables users to control and interact with a simulated world using their hand movements. Overcoming the challenges of accurately detecting and responding to various hand gestures represents a significant achievement for our team. The creation of an immersive and enjoyable user experience is a source of pride for us.

**What we learned**
During the development of Block Touch, we gained valuable insights into the complexities of integrating computer vision into interactive applications. We learned how to optimize algorithms for real-time performance, enhance gesture recognition accuracy, and create a seamless connection between the physical and virtual worlds.

**What's next for Block Touch**
In the future, we plan to expand the capabilities of Block Touch by incorporating more advanced features and functionalities. This includes refining the hand gesture recognition system, adding new interactions, and potentially integrating it with virtual reality (VR) environments. We aim to continue enhancing the user experience and exploring innovative ways to leverage computer vision for interactive applications.
## Inspiration

Open-world AR applications like Pokemon Go that bring AR into everyday life and the outdoors were major inspirations for this project. Additionally, in thinking about how to integrate smartphones with Spectacles, we found inspiration in video games like Phasmophobia's EMP sensors that react more strongly in the presence of ghosts or The Legend of Zelda: Skyward Sword that contained an in-game tracking functionality that pulsates more strongly when facing the direction of and walking closer to a target.

## What it does

This game integrates the Spectacles gear and smartphones together by allowing users to leverage the gyroscopic, haptic, and tactile functionalities of phones to control or receive input about their AR environment. In the game, users have to track down randomly placed treasure chests in their surrounding environment by using their phone as a sensor that begins vibrating when the user is facing a treasure and enters stronger modes of haptic feedback as users get closer to the treasure spots.

These chests come in three types: monetary, puzzle, and challenge. Monetary chests immediately give users in-game rewards. Puzzle chests engage users in a single-player mini-game that may require cognitive or physical activity. Finally, challenge chests similarly engage users in activities not necessarily games, and a stretch goal for multiplayers was that if multiple users were near a spot that another user found a treasure in, the other n users could challenge the treasure finder in a n vs. 1 duel, with the winner(s) taking the rewards.

## How we built it

Once we figured out our direction for the project, we built a user flow architecture in Figma to brainstorm the game design for our application ([link](https://www.figma.com/design/pyG5hlpYkWwVcyvIQJCnY3/Treasure-Hunt-UX-Architecture?node-id=0-1&node-type=canvas&t=clqInR0JpOM6tEnv-0)), and we also visualized how to implement the system for integrating phone haptic feedback with the spectacles depending on distance and directional conditions.

From there, we each took on specific aspects of the user flow architecture to primarily work on: (1) the treasure detection mechanism, (2) spawning the treasure once the user entered within a short distance from the target, and (3) the content of the treasure chests (i.e. rewards or activities). Nearly everything was done using in-house libraries, assets, and the GenAI suite within Snap's Lens Studio.

## Challenges we ran into

As we were working with Spectacles for the first time (compounded with internet problems), we initially encountered technical issues with setting up our development environment and linking the Spectacles for debugging. Due to limited documentation and forums since it is limited-access technology, we had to do a lot of trial-and-error and guessing to figure out how to get our code to work, but luckily, Snap's documentation provided templates to work off of and the Snap staff was able to provide technical assistance to guide us in the right direction. Additionally, given one Spectacle to work with, parallelizing our development work was quite challenging as we had to integrate everything onto one computer while dealing with merge conflicts between our code.

## Accomplishments that we're proud of

In a short span of time, we were able to successfully build a game that provides a unique immersive experience! We've come across and solved errors that didn't have solutions on the internet. For a couple of members of our team, this sparks a newfound interest in the AR space.

## What we learned

This was our first time working with Lens Studio and it's unanimously been a smooth and great software to work with.
For the experienced members on our team, it's been a rewarding experience to make an AR application using JS/TS instead of C# which is the standard language used in Unity.

## What's next for SnapChest

We're excited to push this app forward by adding more locations for treasures, implementing a point system, and also a voice agent integration that provides feedback based on where you're going so you won't get bored on your journey!
If Spectacles would be made available to the general public, a multiplayer functionality would definitely gain a lot of traction and we're looking forward to the future!
## Inspiration

Peripheral nerve compression syndromes such as carpal tunnel syndrome affect approximately 1 out of every 6 adults. They are commonly caused by repetitive stress and with the recent trend of working at home due to the pandemic it has become a mounting issue more individuals will need to address. There exist several different types of exercises to help prevent these syndromes, in fact studies show that 71.2% of patients who did not perform these exercises had to later undergo surgery due to their condition. It should also be noted that doing these exercises wrong could cause permanent injury to the hand as well.

## What it does

That is why we decided to create the “Helping Hand”, providing exercises for a user to perform and using a machine learning model to recognize each successful try. We implemented flex sensors and an IMU on a glove to track the movement and position of the user's hand. An interactive GUI was created in Python to prompt users to perform certain hand exercises. A real time classifier is then run once the user begins the gesture to identify whether they were able to successfully recreate it. Through the application, we can track the progression of the user's hand mobility and appropriately recommend exercises to target the areas where they are lacking most.

## How we built it

The flex sensors were mounted on the glove using custom-designed 3D printed holders. We used an Arduino Uno to collect all the information from the 5 flex sensors and the IMU. The Arduino Uno interfaced with our computer via a USB cable. We created a machine learning model with the use of TensorFlow and Python to classify hand gestures in real time. The user was able to interact with our program with a simple GUI made in Python.

## Challenges we ran into

Hooking up 5 flex sensors and an IMU to one power supply initially caused some power issues causing the IMU not to function/give inaccurate readings. We were able to rectify the problem and add pull-up resistors as necessary. There were also various issues with the data collection such as gyroscopic drift in the IMU readings. Another challenge was the need to effectively collect large datasets for the model which prompted us to create clever Python scripts to facilitate this process.

## Accomplishments that we're proud of

Accomplishments we are proud of include, designing and 3D printing custom holders for the flex sensors and integrating both the IMU and flex sensors to collect data simultaneously on the glove. It was also our first time collecting real datasets and using TensorFlow to train a machine learning classifier model.

## What we learned

We learned how to collect real-time data from sensors and create various scripts to process the data. We also learned how to set up a machine learning model including parsing the data, splitting data into training and testing sets, and validating the model.

## What's next for Helping Hand

There are many improvements for Helping Hand. We would like to make Helping Hand wireless, by using an Arduino Nano which has Bluetooth capabilities as well as compatibility with Tensorflow lite. This would mean that all the classification would happen right on the device! Also, by uploading the data from the glove to a central database, it can be easily shared with your doctor.

We would also like to create an app so that the user can conveniently perform these exercises anywhere, anytime.

Lastly, we would like to implement an accuracy score of each gesture rather than a binary pass/fail (i.e. display a reading of how well you are able to bend your fingers/rotate your wrist when performing a particular gesture). This would allow us to more appropriately identify the weaknesses within the hand.