## Inspiration

It takes over 9 months to train a guide dog. The issue is: dogs can't talk (sorry). What if your favorite furry friend could not only guide you, but also talk to you — and understand exactly where you need to go? This was the founding idea behind Rex. We were inspired to create a fantastically helpful pup, who could lead you anywhere you wanted to go. By combining "classic" robotics with cutting edge AI, we were able to bring this dream to reality. We hope, with this, to facilitate the next generation of accessibility support at malls, hospitals, and even hackathons!

## What it does

Rex is a wayfinding bot with agentic functionality. They listen to your request — whatever it may be — and interprets that to be a location in the building you're in. For example, if you told it "I'm feeling hungry", it would lead you to the RBC Oasis tent for some snacks! Along the way, it will look out for obstacles in your path and avoid them if necessary. It focuses highly on accessibility.

## How we built it

**On the software side**, we made extensive use of VoiceFlow and Mappedin. This provided a significant challenge since these services could not efficiently be run on a Raspberry Pi, so we created a mobile-friendly web application that takes care of human-computer interaction (text to speech, speech to text) as well as making API requests.

We used a custom knowledge base on VoiceFlow to customize the agent to be very specific to the different rooms in E7 and point of interests at Hack the North. We used VoiceFlow's Supabase integration to log a request in the database to begin a trip in the database (that the Raspberry Pi picks up on). Through the app, we made calls to VoiceFlow's API to progress the user's conversation with the agent.

For Mappedin, we rendered the map on a React Typescript application and made use of the Wayfinding endpoint to retrieve thorough directions. Not only are these directions displayed on the app, but they are sent to the Supabase database which the Raspberry Pi listens to for entries to trigger certain movements.

Supabase was used to connect the Raspberry Pi, VoiceFlow Agent, and Mappedin wayfinding. We chose Supabase due to its realtime subscription abilities and also its compatibility with all three ends of Rex.

**On the firmware side**, our hardware system involved multiple sensors and actuators such as a Gyroscope, accelerometer, ultra-sonic sensor, and 4 DC motors. Challenged with not having access to a GPS or motor encoders, we used the accelerometer to determine our current location with respect to the origin when navigating through some area, and used the Gyroscope to feed data into our PID controller to ensure we can travel straight and make accurate turns. We also used an ultra-sonic sensor for obstacle avoidance. Once we have received the directions to the requested destination from the database, we use these sensors and actuators to ensure you are able to get there safely.

**On the mechanical side**, the dog-shaped cover, axles, and ears for this project were 3D printed. The cover was designed to resemble a Dachshund, serving as a protective casing for the wheels, MCU, and circuit components. Two axles were fabricated to hold the assembly together, with the lower axle specifically supporting the main microcontroller. One of the main challenges we encountered was the placement of the battery pack inside the dog’s head, which affected the balance. To counteract this, we added another axle vertically, connecting the base to the top of the robot dog to maintain stability.

## Challenges we ran into

We ran into many significant challenges primarily in the realm of interfacing between all the different services and especially the fact that we had to package everything into a Raspberry Pi with real-world movement. We had to leverage a cloud database to communicate with all the different end devices. Moreover, not having a GPS receiver made us look at other solutions to maintain accuracy during movement. Lastly, some APIs didn't have support in certain languages which forced us to be creative with our implementations.

## Accomplishments that we're proud of

We are proud of the way we were able to interconnect all these systems and put it into a compelling form factor. We truly learned a lot about various areas of engineering.

## What's next for Rex

* Using GPS for more accurate positioning
* Capability to traverse different elevations like stairs
* Increasing the capabilities of the agentic system
* Giving the user the ability to upload designs of their own areas directly within the application
* Making Rex look even cooler!
## Inspiration

In a world in which we all have the ability to put on a VR headset and see places we've never seen, search for questions in the back of our mind on Google and see knowledge we have never seen before, and send and receive photos we've never seen before, we wanted to provide a way for the visually impaired to also see as they have never seen before. We take for granted our ability to move around freely in the world. This inspired us to enable others more freedom to do the same. We called it "GuideCam" because like a guide dog, or application is meant to be a companion and a guide to the visually impaired.

## What it does

Guide cam provides an easy to use interface for the visually impaired to ask questions, either through a braille keyboard on their iPhone, or through speaking out loud into a microphone. They can ask questions like "Is there a bottle in front of me?", "How far away is it?", and "Notify me if there is a bottle in front of me" and our application will talk back to them and answer their questions, or notify them when certain objects appear in front of them.

## How we built it

We have python scripts running that continuously take webcam pictures from a laptop every 2 seconds and put this into a bucket. Upon user input like "Is there a bottle in front of me?" either from Braille keyboard input on the iPhone, or through speech (which is processed into text using Google's Speech API), we take the last picture uploaded to the bucket use Google's Vision API to determine if there is a bottle in the picture. Distance calculation is done using the following formula: distance = ( (known width of standard object) x (focal length of camera) ) / (width in pixels of object in picture).

## Challenges we ran into

Trying to find a way to get the iPhone and a separate laptop to communicate was difficult, as well as getting all the separate parts of this working together. We also had to change our ideas on what this app should do many times based on constraints.

## Accomplishments that we're proud of

We are proud that we were able to learn to use Google's ML APIs, and that we were able to get both keyboard Braille and voice input from the user working, as well as both providing image detection AND image distance (for our demo object). We are also proud that we were able to come up with an idea that can help people, and that we were able to work on a project that is important to us because we know that it will help people.

## What we learned

We learned to use Google's ML APIs, how to create iPhone applications, how to get an iPhone and laptop to communicate information, and how to collaborate on a big project and split up the work.

## What's next for GuideCam

We intend to improve the Braille keyboard to include a backspace, as well as making it so there is simultaneous pressing of keys to record 1 letter.
## Inspiration

Recent mass shooting events are indicative of a rising, unfortunate trend in the United States. During a shooting, someone may be killed every 3 seconds on average, while it takes authorities an average of 10 minutes to arrive on a crime scene after a distress call. In addition, cameras and live closed circuit video monitoring are almost ubiquitous now, but are almost always used for post-crime analysis. Why not use them immediately? With the power of Google Cloud and other tools, we can use camera feed to immediately detect weapons real-time, identify a threat, send authorities a pinpointed location, and track the suspect - all in one fell swoop.

## What it does

At its core, our intelligent surveillance system takes in a live video feed and constantly watches for any sign of a gun or weapon. Once detected, the system immediately bounds the weapon, identifies the potential suspect with the weapon, and sends the authorities a snapshot of the scene and precise location information. In parallel, the suspect is matched against a database for any additional information that could be provided to the authorities.

## How we built it

The core of our project is distributed across the Google Cloud framework and AWS Rekognition. A camera (most commonly a CCTV) presents a live feed to a model, which is constantly looking for anything that looks like a gun using GCP's Vision API. Once detected, we bound the gun and nearby people and identify the shooter through a distance calculation. The backend captures all of this information and sends this to check against a cloud-hosted database of people. Then, our frontend pulls from the identified suspect in the database and presents all necessary information to authorities in a concise dashboard which employs the Maps API. As soon as a gun is drawn, the authorities see the location on a map, the gun holder's current scene, and if available, his background and physical characteristics. Then, AWS Rekognition uses face matching to run the threat against a database to present more detail.

## Challenges we ran into

There are some careful nuances to the idea that we had to account for in our project. For one, few models are pre-trained on weapons, so we experimented with training our own model in addition to using the Vision API. Additionally, identifying the weapon holder is a difficult task - sometimes the gun is not necessarily closest to the person holding it. This is offset by the fact that we send a scene snapshot to the authorities, and most gun attacks happen from a distance. Testing is also difficult, considering we do not have access to guns to hold in front of a camera.

## Accomplishments that we're proud of

A clever geometry-based algorithm to predict the person holding the gun. Minimized latency when running several processes at once. Clean integration with a database integrating in real-time.

## What we learned

It's easy to say we're shooting for MVP, but we need to be careful about managing expectations for what features should be part of the MVP and what features are extraneous.

## What's next for HawkCC

As with all machine learning based products, we would train a fresh model on our specific use case. Given the raw amount of CCTV footage out there, this is not a difficult task, but simply a time-consuming one. This would improve accuracy in 2 main respects - cleaner identification of weapons from a slightly top-down view, and better tracking of individuals within the frame. SMS alert integration is another feature that we could easily plug into the surveillance system as well, and further compound the reaction improvement time.