## Inspiration

News frequently reports on vehicle crashes and accidents, with one statistic highlighting the prevalence of heavy truck accidents caused by driver fatigue. Truck drivers endure long hours on the road, delivering shipments nationwide, contributing to the tiredness that can lead to accidents. According to the National Transportation Safety Board, nearly 40% of heavy truck accidents originate from fatigue. In response, we pushed to develop a system capable of monitoring both facial expressions and heartbeats to detect early signs of fatigue among drivers.

## What it does

Our web app boasts two features aimed at improving driver safety: one harnesses computer vision technology to track the driver's face, effectively detecting signs of drowsiness, while the other streams the driver's heartbeat in real-time, providing an additional layer of drowsiness detection. Accessible through our web app is a dedicated page for viewing the webcam feed, which ideally can be monitored via personal devices like smartphones. Should the webcam detect the driver falling asleep, it triggers an alert with flashing lights and a sound to awaken the driver. Additionally, our dashboard feature enables managers to monitor their drivers and their respective drowsiness levels. We've incorporated a graphing feature within the dashboard that dynamically turns red when a selected driver's drowsiness level drops below the acceptable threshold, providing a clear visual indication of potential fatigue.

## How we built it

By combining Reflex and TerraAPI, as well as a companion mobile app in Swift, we were able to create a solution all within our ecosystem. The TerraAPI provided the crucial heartrate data in real time, which we livestreamed through a webhook that our Reflex website could read. The Reflex website also contains a manager-style dashboard for viewing several truckers and collect their unique data all at the same time. As a demo for future mobile usage, we also included a facial recognition and landmarking model to detect drowsiness and alert the user if they are falling asleep. The Swift app also provided additional information such as the heartrate in real time and establishing the connection to the webhook from the wearable device.

## Challenges we ran into

In order to construct the complex data flow of our project, we had to learn several new technologies along the way. It started with developing on a new wearable device with limited documentation and support only through a Swift iOS app, which none of us had experience with. With Reflex, we also encountered some bugs, which all had workarounds, and the difficulties that come with developing any website.

## Accomplishments that we're proud of

We're proud of being able to integrate such complex technologies and orchestrate them in a seamless way. At times, we were afraid that our product wouldn't come together since all the components depended on each other and we needed to complete all of them. However, our team made everything work in the end.

## What we learned

Many of the technologies we worked with during TreeHacks were new and had a large learning curve in order to build our end goal. Along this journey, our team picked up valuable skills in Swift, Python, computer vision, web development, and how to work on 2 hours of sleep.

## What's next for TruckrZzz

We hope to broaden our target audience and not only apply these technologies for truck drivers, but also every day drivers that might need some extra assistance staying awake on the road.
## Inspiration

Over the past year, we've seen numerous events close down major highways, arterial roads, and also shut down rail lines. Weather-related events such as blowing snow, layers of ice, unprecedented amounts of snow all contributed to natural causes, but there were also a significant number of disturbances due to human factors: trucker convoys, parades, demonstrations, etc.

![ontario highway closure](https://beta.ctvnews.ca/dam/ctvnews/images/2022/12/23/1_6207252.png?1671818880604)

Throughout all these events, Ontarions still continued to rely on our transportation and delivery backbone to continue getting groceries, goods, and other items. Throughout all of these major events, we noticed that traditional routing applications such as google maps, waze, waymo, etc. all failed to quickly recognize the road closures and re-route accordingly. In fact - they often re-routed hoardes of people onto the same detours, leading to long stretches of bumper to bumper traffic.

We wanted to take the opportunity to provide a better system to aid the drivers and companies working around the clock to keep our infrastructure alive, by giving them a system that pre-emptively provides alternate routes, avoids traffic, & hard-to-drive narrow roads (especially common on highway bypass detours) all while saving fuel and time for the company and the driver.

## What it does

In short, Courier Pigeon operates as a fleet management service that synergizes company vehicles with personal and third party delivery platforms, assigning drivers routes scaled to the capabilities of the vehicle an driver.

It allows drivers to minimize delivery times by providing constant updates to the route, accounting for extra package pickups, slowdowns ahead, or inclement weather areas (such as steep, icy roads!). It also automates the assignment of routes, as well as providing real-time live updates for package delivery status, vehicle status (such as fuel/energy remaining, driving habits, or general GPS updates!)

## How we built it

Courier Pigeon's core logic is built using Python3, integrated with HTML and JS. This core logic uses Solace's PubSub+ Network as it's primary communication pipeline, allowing independent application instances to connect and communicate with each other over the cloud.

The "supervisor" acts as the main decision and logic hub for Courier Pigeon, being subscribed to any event and handling tasks such as: updating package status to the user! All PubSub events followed a topic format that we decided on: `pigeon/<type>/<Van #>/<Package #>`

A brief look at the full topic hierarchy is shown here:
![topic hierarchy](https://media.discordapp.net/attachments/1072894591415636089/1074336066850533378/image.png)

For example, we can have a route that dynamically adjusts itself to incorporate multiple pickups! These changes would be reported by a request event over Solace PubSub, which then tells the supervisor to resequencing & re-optimize the remaining route! This route is then broadcast out in a format similar to this:
![pubsub reroute broadcasts](https://media.discordapp.net/attachments/1072894591415636089/1074208056453378108/image.png?width=1920&height=299)

## Challenges we ran into

Over the past 36 hours, we ran into a few challenges, some major, some minor.

To start, the hardest part was our own skillset. We didn't really have prior frontend experience, nor did we have large amounts of python experience - this continued to prove to be a challenge throughout the entire weekend, but we didn't let that stop us and we still tried our best!

We also ran into constant issues with network port connections. When learning how to use Solace's pubsub+ event broker, we spent nearly half a day googling and trying to see what we were doing wrong - when the issue turned out to be that the connection port had been blocked on eduroam and we needed to switch to either a mobile hotspot, or later on, guOttawa.

Once our Solace issues were resolved, we continued to develop, but we hadn't come up with a thorough system architecture - meaning that we were often faced with the challenge of actually figuring out how certain components would interface with each other - leading to some re-writes, and then a few more re-writes. A particular bugger for this was the difference between latitude, longitude and longitude, latitude between ISO standard and certain APIs.

Lastly, we had the fun experience of realizing that we wrote a large package to support a particular API that was only in beta, meaning that we had to re-write a large portion of package support to use an older, publicly available and free version of that API. We also later realized that the public & free version of the API did not do what we needed to do, so we had to come up with a fairly last-minute solution to ensure full functionality.

## Accomplishments that we're proud of

Although we may not have much at the end, we're really proud that we were able to give android app development a shot, as well as web dev, which is not something any of our team had done before.

## What we learned

Android Studio makes our laptops overheat.
Also, PubSub+, Mapbox are easier to get started on than we thought! PubSub+ especially was very cool and made development a lot easier.

## What's next for Courier Pigeon

What's next? You've seen our proof of concept, next time it'll be a full app and website, maybe using some distributed ledger so that we can have a decentralized supervisor network! (no need to run hosted servers? - instead, allow mapbox and solace to handle the routing and optimization part, we can even run our supervisor in the cloud with Solace tools!
## Inspiration

While searching for ideas, our team came across an ECG dataset that was classified and labeled into categories for normal and abnormal patterns. While examining the abnormal patterns, it was observed that most seizure patterns had a small window that transitioned from a typical pattern to a seizure pattern around a 15-second window. Most of the accidents and damage in seizures are caused by falling, lack of help, or getting caught in disadvantaged situations -driving, cooking, etc.- detecting this short period in real-time and predicting a seizure with machine learning to warn the user seemed as somewhat of a viable solution. After this initial ideation, sharing patient data such as allergies, medicinal history, emergency contacts, previous seizures, and essential information at the moment of attack for the emergency workers was thought out in the case of unresponsiveness from the user to the app's notification.

## What does it do?

The system contains the following:
Three main agents.
A smartwatch with an accurate ECG sensor.
A machine learning algorithm on the cloud.
An integrated notification app on mobile phones to retrieve patient information during attacks.
The workflow includes a constant data transfer between the smartwatch and the machine learning algorithm to detect anomalies. If an attack is predicted, a notification prompts the user to check if this is a false positive prediction. Nothing is triggered if the user confirms nothing is wrong and dismisses the warning. The seizure protocol starts if the notification stays unattended or is answered as positive.

Seizure Protocol Includes:
-The user is warned by the prediction and should have found a safe space/position/situation to handle the seizure
-Alarms from both synced devices, mobile, and smartwatch
-Display of the FHIR patient history on the synced device, allergies, medicinal data, and fundamental id info for emergency healthcare workers
-Contacting emergency numbers recorded for the individual

With the help of the app, we prevent further damage by accidents by predicting the seizure. After the episode, we help the emergency workers have a smoother experience assisting the patient.

## Building the System

We attempted to use Zepp's smartwatch development environment to create a smartwatch app to track and communicate with the cloud (Though there have been problems with the ECG sensors, which will be mentioned in the challenges section.) For the machine learning algorithm, we used an LSTM model (Long-Short Term Memory Networks) to slice up the continuously fed data and classify between "normal" and "abnormal" states after training it on both Mit-Bih Epileptic Seizure Recognition datasets we have found. If the "abnormal" form has been observed for more than the threshold, we have classified it as a "seizure predictor." When the state changed to the seizure protocol, we had two ways of information transfer, one is to the smartwatch as an alarm/notification, and the other one to the synced mobile app to display information. For the mobile app, we have created a React Native app for the users to create profiles and transfer/display health information via the InterSystem's FHIR.js package. While in the "listening" state, the app waits for the seizure notification. When it receives it, it fetches and displays health information/history/emergency contacts and anything that can be useful to the emergency healthcare worker on the lock screen without unlocking the phone. Thus, providing a safer and smoother experience for the patient and the healthcare workers.

## Challenges

There have been several challenges and problems that we have encountered in this project. Some of them stayed unresolved, and some of them received quick fixes.

The first problem was using the ECG function of the Zepp watches. Because the watch ECG function was disabled in the U.S. due to a legal issue with the FDA, we could not pull up live data in the Hackathon. We resolved this issue by finding a premade ECG dataset and doing the train-test-validation on this premade dataset for the sake of providing a somewhat performing model for the Hackathon.

The second problem we encountered was that we could only measure our accuracy with our relatively mid-sized dataset. In the future, testing it with various datasets, trying sequential algorithms, and optimizing layers and performance would be advised. In the current state, without a live information feed and a comprehensive dataset, it is hard to be sure about the issues of overfitting/underfitting the dataset.

## Accomplishments

We could create a viable machine learning algorithm to predict seizures in a concise time frame, which took a lot of effort, research, and trials, especially in the beginning since we switched from plain RNN to LSTM due to the short-time frame problem. However, our algorithm works with a plausible accuracy (Keeping in mind that we cannot check for overfitting/underfitting without a diverse dataset). Another achievement we are proud of is that we attempted to build a project with many branches, like ReactApp, Zepp Integration, and Machine Learning in Python, which forced us to experience a product-development process in a super-dense mode.

But most importantly, attending the Hackathon and meeting with amazing people that both organized, supported, and competed in it was an achievement to appreciate!

## Points to Take Home

The most discussed point we learned was that integrating many APIs is a rather daunting process in terms of developing something within 24 hours. It was much harder to adapt and link these different technologies together, even though we had anticipated it before attempting it. The second point we learned was that we needed to be careful about our resources during the challenges. Especially our assumption about the live-data feed from the watch made us stumble in the development process a bit. However, these problems make Hackathons a learning experience, so it's all good!

## Future for PulseBud

The plans might include sharpening the ML with a variety of dense and large-scale datasets and optimizing the prediction methodology to reach the lowest latency with the highest accuracy. We might also try to run it on the watch itself if it can get a robust state like that. Also, setting personalized thresholds for each user would be much more efficient in terms of notification frequency if the person is an outlier. Also, handling the live data feed to the algorithm should be the priority. If these can be done to the full extent, this application can be a very comfortable quality of life change for many people who experience or might experience seizures.