## Inspiration

People who have mobility issues have a hard time finding friends or loved ones if they are in a crowded space. The number of adults unable (or very difficult) to walk a quarter mile: 17.2 million which is 7.3% of the population.

## What it does

Spot-E is a Remote Control car that can detect and recognize faces. Imagine if you have a loved one who has alzheimer's and visual impairment and you can’t find them in a large building. You can send Spot-E around to look for them. Once the user is found by the car, they will drop a GPS signal for where the car is.

The finder will send a message to the user such as, Wait here John, I will come get you in 5 minutes.

On the other hand, if the finder has a disability such as the inability to walk or a visual impairment, it can tell the car to look for the person.

## How I built it

Raspberry pi, android studio, python, java, javascript php, azure

## Challenges I ran into

Everything was a challenge because none of us had worked with cross platform communication channels, android and designing a hardware system

## Accomplishments that I'm proud of

That this actually works! We came up with better and better design for the hardware.

## What I learned

Maintain code integrity across the channels, some channel designs are better than others

## What's next for Spotty

Incorporating a drone into the device
It will be fully autonomous in the future
## Inspiration

Our team focuses on real-world problems. One of our own classmates is blind, and we've witnessed firsthand the difficulties he encounters during lectures, particularly when it comes to accessing the information presented on the board.

It's a powerful reminder that innovation isn't just about creating flashy technology; it's about making a tangible impact on people's lives. "Hawkeye" isn't a theoretical concept; it's a practical solution born from a genuine need.

## What it does

"Hawkeye" utilizes Adhawk MindLink to provide essential visual information to the blind and visually impaired. Our application offers a wide range of functions:

* **Text Recognition**: "Hawkeye" can read aloud whiteboard text, screens, and all text that our users would not otherwise see.
* **Object Identification**: The application identifies text and objects in the user's environment, providing information about their size, shape, and position.
* **Answering Questions** Hawkeye takes the place of Google for the visually impaired, using pure voice commands to search

## How we built it

We built "Hawkeye" by combining state-of-the-art computer vision and natural language processing algorithms with the Adhawk MindLink hardware. The development process involved several key steps:

1. **Data Collection**: We used open-source AI models to recognize and describe text and object elements accurately.
2. **Input System**: We developed a user-friendly voice input system that can be picked up by anyone.
3. **Testing and Feedback**: Extensive testing and consultation with the AdHawk team was conducted to fine-tune the application's performance and usability.

## Challenges we ran into

Building "Hawkeye" presented several challenges:

* **Real-time Processing**: We knew that real-time processing of so much data on a wearable device was possible, but did not know how much latency there would be. Fortunately, with many optimizations, we were able to get the processing to acceptable speeds.
* **Model Accuracy**: Ensuring high accuracy in text and object recognition, as well as facial recognition, required continuous refinement of our AI models.
* **Hardware Compatibility**: Adapting our software to work effectively with Adhawk MindLink's hardware posed compatibility challenges that we had to overcome.

## Accomplishments that we're proud of

We're immensely proud of what "Hawkeye" represents and the impact it can have on the lives of blind and visually impaired individuals. Our accomplishments include:

* **Empowerment**: Providing a tool that enhances the independence and quality of life for visually impaired individuals. To no longer rely upon transcribers and assistants is something that real
* **Inclusivity**: Breaking down barriers to education and employment, making these opportunities more accessible.
* **Innovation**: Combining cutting-edge technology and AI to create a groundbreaking solution for a pressing societal issue.
* **User-Centric Design**: Prioritizing user feedback and needs throughout the development process to create a genuinely user-friendly application.

## What we learned

Throughout the development of "Hawkeye," we learned valuable lessons about the power of technology to transform lives. Key takeaways include:

* **Empathy**: Understanding the daily challenges faced by visually impaired individuals deepened our empathy and commitment to creating inclusive technology.
* **Technical Skills**: We honed our skills in computer vision, natural language processing, and hardware-software integration.
* **Ethical Considerations**: We gained insights into the ethical implications of AI technology, especially in areas like facial recognition.
* **Collaboration**: Effective teamwork and collaboration were instrumental in overcoming challenges and achieving our goals.

## What's next for Hawkeye

The journey for "Hawkeye" doesn't end here. In the future, we plan to:

* **Expand Functionality**: We aim to enhance "Hawkeye" by adding new features and capabilities, such as enhanced indoor navigation and support for more languages.
* **Accessibility**: We will continue to improve the user experience, ensuring that "Hawkeye" is accessible to as many people as possible.
* **Partnerships**: Collaborate with organizations and institutions to integrate "Hawkeye" into educational and workplace environments.
* **Advocacy**: Raise awareness about the importance of inclusive technology and advocate for its widespread adoption.
* **Community Engagement**: Foster a supportive user community for sharing experiences, ideas, and feedback to further improve "Hawkeye."

With "Hawkeye," our vision is to create a more inclusive and accessible world, where visual impairment is no longer a barrier to achieving one's dreams and aspirations. Together, we can make this vision a reality.
## Inspiration

Our project, "**Jarvis**," was born out of a deep-seated desire to empower individuals with visual impairments by providing them with a groundbreaking tool for comprehending and navigating their surroundings. Our aspiration was to bridge the accessibility gap and ensure that blind individuals can fully grasp their environment. By providing the visually impaired community access to **auditory descriptions** of their surroundings, a **personal assistant**, and an understanding of **non-verbal cues**, we have built the world's most advanced tool for the visually impaired community.

## What it does

"**Jarvis**" is a revolutionary technology that boasts a multifaceted array of functionalities. It not only perceives and identifies elements in the blind person's surroundings but also offers **auditory descriptions**, effectively narrating the environmental aspects they encounter. We utilize a **speech-to-text** and **text-to-speech model** similar to **Siri** / **Alexa**, enabling ease of access. Moreover, our model possesses the remarkable capability to recognize and interpret the **facial expressions** of individuals who stand in close proximity to the blind person, providing them with invaluable social cues. Furthermore, users can ask questions that may require critical reasoning, such as what to order from a menu or navigating complex public-transport-maps. Our system is extended to the **Amazfit**, enabling users to get a description of their surroundings or identify the people around them with a single press.

## How we built it

The development of "**Jarvis**" was a meticulous and collaborative endeavor that involved a comprehensive array of cutting-edge technologies and methodologies. Our team harnessed state-of-the-art **machine learning frameworks** and sophisticated **computer vision techniques** to get analysis about the environment, like , **Hume**, **LlaVa**, **OpenCV**, a sophisticated computer vision techniques to get analysis about the environment, and used **next.js** to create our frontend which was established with the **ZeppOS** using **Amazfit smartwatch**.

## Challenges we ran into

Throughout the development process, we encountered a host of formidable challenges. These obstacles included the intricacies of training a model to recognize and interpret a diverse range of environmental elements and human expressions. We also had to grapple with the intricacies of optimizing the model for real-time usage on the **Zepp smartwatch** and get through the **vibrations** get enabled according to the **Hume** emotional analysis model, we faced issues while integrating **OCR (Optical Character Recognition)** capabilities with the **text-to speech** model. However, our team's relentless commitment and problem-solving skills enabled us to surmount these challenges.

## Accomplishments that we're proud of

Our proudest achievements in the course of this project encompass several remarkable milestones. These include the successful development of "**Jarvis**" a model that can audibly describe complex environments to blind individuals, thus enhancing their **situational awareness**. Furthermore, our model's ability to discern and interpret **human facial expressions** stands as a noteworthy accomplishment.

## What we learned

# Hume

**Hume** is instrumental for our project's **emotion-analysis**. This information is then translated into **audio descriptions** and the **vibrations** onto **Amazfit smartwatch**, providing users with valuable insights about their surroundings. By capturing facial expressions and analyzing them, our system can provide feedback on the **emotions** displayed by individuals in the user's vicinity. This feature is particularly beneficial in social interactions, as it aids users in understanding **non-verbal cues**.

# Zepp

Our project involved a deep dive into the capabilities of **ZeppOS**, and we successfully integrated the **Amazfit smartwatch** into our web application. This integration is not just a technical achievement; it has far-reaching implications for the visually impaired. With this technology, we've created a user-friendly application that provides an in-depth understanding of the user's surroundings, significantly enhancing their daily experiences. By using the **vibrations**, the visually impaired are notified of their actions. Furthermore, the intensity of the vibration is proportional to the intensity of the emotion measured through **Hume**.

# Ziiliz

We used **Zilliz** to host **Milvus** online, and stored a dataset of images and their vector embeddings. Each image was classified as a person; hence, we were able to build an **identity-classification** tool using **Zilliz's** reverse-image-search tool. We further set a minimum threshold below which people's identities were not recognized, i.e. their data was not in **Zilliz**. We estimate the accuracy of this model to be around **95%**.

# Github

We acquired a comprehensive understanding of the capabilities of version control using **Git** and established an organization. Within this organization, we allocated specific tasks labeled as "**TODO**" to each team member. **Git** was effectively employed to facilitate team discussions, workflows, and identify issues within each other's contributions.

The overall development of "**Jarvis**" has been a rich learning experience for our team. We have acquired a deep understanding of cutting-edge **machine learning**, **computer vision**, and **speech synthesis** techniques. Moreover, we have gained invaluable insights into the complexities of real-world application, particularly when adapting technology for wearable devices. This project has not only broadened our technical knowledge but has also instilled in us a profound sense of empathy and a commitment to enhancing the lives of visually impaired individuals.

## What's next for Jarvis

The future holds exciting prospects for "**Jarvis.**" We envision continuous development and refinement of our model, with a focus on expanding its capabilities to provide even more comprehensive **environmental descriptions**. In the pipeline are plans to extend its compatibility to a wider range of **wearable devices**, ensuring its accessibility to a broader audience. Additionally, we are exploring opportunities for collaboration with organizations dedicated to the betterment of **accessibility technology**. The journey ahead involves further advancements in **assistive technology** and greater empowerment for individuals with visual impairments.