## Inspiration

Our project, "**Jarvis**," was born out of a deep-seated desire to empower individuals with visual impairments by providing them with a groundbreaking tool for comprehending and navigating their surroundings. Our aspiration was to bridge the accessibility gap and ensure that blind individuals can fully grasp their environment. By providing the visually impaired community access to **auditory descriptions** of their surroundings, a **personal assistant**, and an understanding of **non-verbal cues**, we have built the world's most advanced tool for the visually impaired community.

## What it does

"**Jarvis**" is a revolutionary technology that boasts a multifaceted array of functionalities. It not only perceives and identifies elements in the blind person's surroundings but also offers **auditory descriptions**, effectively narrating the environmental aspects they encounter. We utilize a **speech-to-text** and **text-to-speech model** similar to **Siri** / **Alexa**, enabling ease of access. Moreover, our model possesses the remarkable capability to recognize and interpret the **facial expressions** of individuals who stand in close proximity to the blind person, providing them with invaluable social cues. Furthermore, users can ask questions that may require critical reasoning, such as what to order from a menu or navigating complex public-transport-maps. Our system is extended to the **Amazfit**, enabling users to get a description of their surroundings or identify the people around them with a single press.

## How we built it

The development of "**Jarvis**" was a meticulous and collaborative endeavor that involved a comprehensive array of cutting-edge technologies and methodologies. Our team harnessed state-of-the-art **machine learning frameworks** and sophisticated **computer vision techniques** to get analysis about the environment, like , **Hume**, **LlaVa**, **OpenCV**, a sophisticated computer vision techniques to get analysis about the environment, and used **next.js** to create our frontend which was established with the **ZeppOS** using **Amazfit smartwatch**.

## Challenges we ran into

Throughout the development process, we encountered a host of formidable challenges. These obstacles included the intricacies of training a model to recognize and interpret a diverse range of environmental elements and human expressions. We also had to grapple with the intricacies of optimizing the model for real-time usage on the **Zepp smartwatch** and get through the **vibrations** get enabled according to the **Hume** emotional analysis model, we faced issues while integrating **OCR (Optical Character Recognition)** capabilities with the **text-to speech** model. However, our team's relentless commitment and problem-solving skills enabled us to surmount these challenges.

## Accomplishments that we're proud of

Our proudest achievements in the course of this project encompass several remarkable milestones. These include the successful development of "**Jarvis**" a model that can audibly describe complex environments to blind individuals, thus enhancing their **situational awareness**. Furthermore, our model's ability to discern and interpret **human facial expressions** stands as a noteworthy accomplishment.

## What we learned

# Hume

**Hume** is instrumental for our project's **emotion-analysis**. This information is then translated into **audio descriptions** and the **vibrations** onto **Amazfit smartwatch**, providing users with valuable insights about their surroundings. By capturing facial expressions and analyzing them, our system can provide feedback on the **emotions** displayed by individuals in the user's vicinity. This feature is particularly beneficial in social interactions, as it aids users in understanding **non-verbal cues**.

# Zepp

Our project involved a deep dive into the capabilities of **ZeppOS**, and we successfully integrated the **Amazfit smartwatch** into our web application. This integration is not just a technical achievement; it has far-reaching implications for the visually impaired. With this technology, we've created a user-friendly application that provides an in-depth understanding of the user's surroundings, significantly enhancing their daily experiences. By using the **vibrations**, the visually impaired are notified of their actions. Furthermore, the intensity of the vibration is proportional to the intensity of the emotion measured through **Hume**.

# Ziiliz

We used **Zilliz** to host **Milvus** online, and stored a dataset of images and their vector embeddings. Each image was classified as a person; hence, we were able to build an **identity-classification** tool using **Zilliz's** reverse-image-search tool. We further set a minimum threshold below which people's identities were not recognized, i.e. their data was not in **Zilliz**. We estimate the accuracy of this model to be around **95%**.

# Github

We acquired a comprehensive understanding of the capabilities of version control using **Git** and established an organization. Within this organization, we allocated specific tasks labeled as "**TODO**" to each team member. **Git** was effectively employed to facilitate team discussions, workflows, and identify issues within each other's contributions.

The overall development of "**Jarvis**" has been a rich learning experience for our team. We have acquired a deep understanding of cutting-edge **machine learning**, **computer vision**, and **speech synthesis** techniques. Moreover, we have gained invaluable insights into the complexities of real-world application, particularly when adapting technology for wearable devices. This project has not only broadened our technical knowledge but has also instilled in us a profound sense of empathy and a commitment to enhancing the lives of visually impaired individuals.

## What's next for Jarvis

The future holds exciting prospects for "**Jarvis.**" We envision continuous development and refinement of our model, with a focus on expanding its capabilities to provide even more comprehensive **environmental descriptions**. In the pipeline are plans to extend its compatibility to a wider range of **wearable devices**, ensuring its accessibility to a broader audience. Additionally, we are exploring opportunities for collaboration with organizations dedicated to the betterment of **accessibility technology**. The journey ahead involves further advancements in **assistive technology** and greater empowerment for individuals with visual impairments.
## Inspiration

The idea was to help people who are blind, to be able to discretely gather context during social interactions and general day-to-day activities

## What it does

The glasses take a picture and analyze them using Microsoft, Google, and IBM Watson's Vision Recognition APIs and try to understand what is happening. They then form a sentence and let the user know. There's also a neural network at play that discerns between the two dens and can tell who is in the frame

## How I built it

We took a RPi Camera and increased the length of the cable. We then made a hole in our lens of the glasses and fit it in there. We added a touch sensor to discretely control the camera as well.

## Challenges I ran into

The biggest challenge we ran into was Natural Language Processing, as in trying to parse together a human-sounding sentence that describes the scene.

## What I learned

I learnt a lot about the different vision APIs out there and creating/trainingyour own Neural Network.

## What's next for Let Me See

We want to further improve our analysis and reduce our analyzing time.
## Inspiration

As a team, we've been increasingly concerned about the *data privacy* of users on the internet in 2022. There’s no doubt that the era of vast media consumption has resulted in a monopoly of large tech firms who hold a strong grasp over each and every single user input. When every action you make results in your data being taken and monetized to personalize ads, it’s clear to see the lack of security and protection this can create for unsuspecting everyday users.

That’s why we wanted to create an all-in-one platform to **decentralize the world of advertising** and truly give back **digital data ownership** to users. Moreover, we wanted to increase the transparency of what happens with this data, should the user opt-in to provide this info to advertisers. That’s where our project, **Sonr rADar**, comes in.

## What it does

As the name suggests—Sonr rADar, integrated into the **Sonr** blockchain ecosystem, is a mobile application which aims to decentralize the advertising industry and offer a robust system for users to feel more empowered about their digital footprint. In addition to the security advantages and more meaningful advertisements, users can also monetarily benefit from their interactions with advertisers. We incentivize users by rewarding them with **cryptocurrency tokens** (such as **SNR**) for their time spent on ads, creating a **win/win situation**. Not only that, but it can send advertisers anonymous info and analytics about how successful their ads are, helping them to improve further.

Upon opening the app, users are met with a clean dashboard UI displaying their current token balance, as well as transfer history. In the top right corner, there exists a dropdown menu where the user can choose from various options. This includes the opportunity to enter and update their personal information that they would like to share with advertisers (for ad personalization), an ads dashboard where they can view all relevant ads, and a permissions section for them to opt in/out of where and how their data is being shared (e.g. third parties). Their information is then carefully stored using **data schemas, buckets, and objects**, which directly utilize Sonr’s proprietary **blockchain** technology to offer a *fully transparent* data management solution.

## How we built it

We built this application using the Flutter SDK (more specifically, **motor\_flutter**), programming languages such as Dart & C++, and a suite of tools associated with Sonr; including the Speedway CLI! We also utilized a **testing dataset of 500** user profiles (names, emails, phone numbers, locations) for the **schema architecture**.

## Challenges we ran into

Since our project leverages tools from an early-stage startup, some documentation was quite hard to find, so there were many instances when we were stuck during the backend implementation. Luckily, we were able to visit the Sonr sponsor booth and get valuable assistance with any troubleshooting—as they were very helpful. It was also largely our first time working with Flutter, Go, and a lot of the other components of blockchain and Sonr, so naturally there was a lot to process and overcome.

## Accomplishments that we're proud of

We're proud to have created a proof-of-concept prototype for our ambitious idea; working together as a team to overcome the many obstacles we faced along the way! We hope that this project will create a lasting impact in the advertising space.

## What we learned

How to work together as a team and delegate tasks, balance our time amongst in-person activities + working on our project, and create something exciting in only 36 hours! By working with feedback and workshops offered by the sponsor CEO and Chief of Staff, we got an insightful look into the eventful startup experience and what that journey may look like. It's hard to think of a better way to dive into the exciting opportunities of Blockchain!

## What's next for Sonr rADar

The state of our project is only the beginning; we plan to further build out the individual pages on the application and eventually deploy it to the App Store! We also hope to receive guidance and mentorship from the Sonr team, so that we can utilize their products in the most effective way possible. As we gain more traction, more advertisers and users will join—making our app even better. It's a positive feedback loop that can one day allow us to even bring the decentralized advertisement platform into AI!

## Team Member Badge IDs

**Krish:** TSUNAMI TRAIT CUBE SCHEMA
**Mihir:** STADIUM MOBILE LAP BUFFER
**Amire:** YAM MANOR LIABILITY BATH
**Robert:** CRACKERS TRICK GOOD SKEAN