Memento is an all in one IoT application that provides therapy and care for dementia patients. It runs on two separate platforms that communicate their data with each other. The first part of our solution is a mobile Android Application. In this application, the user can enter Admin mode and create their settings for voice control and right and wrong answers. Then the user or their family members can create stimuli that will then be used in their game. The game then will show the user pictures of people, places, or memories, and ask them what the picture is of. The user can then use voice commands to tell the application what is in that picture. If the user gets the answer right, he scores a point and continue playing the game. This data is then sent to the second part of this solution. But, there is an additional part of the mobile application.

## Location Services for Tracking the Patient

One important concern for dementia patients is that they begin to wander and don't remember where they were going or how they got there, and most importantly, they don't remember how to get home. So in our mobile application, we have incorporated location services that keep track of the user in real time. The user can go and check how far he has strayed from home by simply pressing a single button, and this will send data to our web application as well. In addition, with the integration of the Uber API, the user can simply press a single button to call for an uber to bring them home using the location they are currently at.

## Web Application

The second portion of this hack, is the web application hosted by Microsoft Azure. The web application allows the user to log into their dashboard and access information about themselves in the case that they begin to forget things from their history. We focused on the personal aspect of this mental illness, such that the patient could trigger their memories or answer the questions the ask over and over again by viewing this web page. So, once they log in, they will have access to the most basic of information which includes their name, birthday, phone number, address, and a link to 911 in case of an emergency. They will see links to "Family", "Friends", "Memories", or "Home" with corresponding pictures. If the user clicks on the **Family** or **Friends** tab, they will be brought to their contacts page, where they have access to a database of family and friends' contact information along with corresponding pictures of the contact. This page shows the contact's picture, their relation to the user, their phone number, address, and email. This way, the user can contact them however they'd like.

If the user were to click on the **Home** button, they would be brought to a map page. Using the Google Maps API, we integrated a map that provides them with interactive access to past events in their life as well as their current location. They can scan over the map to see where they were born and on what date. Other examples of events that could potentially go on the map include the date of marriage or when their first child was born.

If the user were to click on the **Charts** tab, they would be brought to a page full of data visualizations for various things regarding Dementia in general as well as their own personal data. The first graph they would see is how well they are performing in the mobile application on a day to day basis. Then, they'd have access to a pie chart providing awareness on the most common forms of Dementia, and lastly, they could see how much time they are spending within their home by gathering data from the location tracking aspect of the mobile application. This data can be used to see how long they have wandered for to see how far away they might have gone, or track daily habits on how much time they typically spend in their homes on a day to day basis. These data visualizations were made using Morris and Flot.

Finally, if the user were to click on the **Memories** tab, they would be brought to an interactive time line that is easily customizable. They can add new events or delete old ones. This time line will show them the date, time, place, information, and a picture of the event taking place at the current time. And the user can swipe through events to recall what he or she has done throughout their life as well as view pictures of the event. This is really important to the user because this will provide them answers to a lot of questions, such as who was at which event and what exactly took place or when the event took place and what the importance was. It provides the user not only with information about the date, but with information on when it happened and pictures for further memory recollection.

## Purpose

As some of our team member have witness how hard it is for Dementia patients first hand, we thought this series of applications would come in great use not only for the patient themselves, but for their families and care takers. We put everything a patient would need in one easy to access place. They can view their memories, call for an Uber if they get lost, play memory games and track their progress, they have access to a database of contacts, and they have location services so their caretaker can see how far from home they are.
## Inspiration

**According to the Alzheimer’s Association, 6.7 million Americans age 65 and older are living with Alzheimer's in 2023.** Alzheimer’s is a gradually progressive brain disorder involving memory loss. It is the most common form of dementia; people forget who their loved ones are and cannot carry out daily tasks anymore. Alzheimer’s and dementia is not only a personal health crisis but impacts family, friends, and caregivers. It is important to focus on the prevention and slow progression of symptoms related to memory loss. **How can we use the Memory Palace – a psychology-based technique where people can associate mnemonic images in their mind to places they know – to help prevent and ease the lives of those with Alzheimer’s and dementia?**

There is a lot of psychology research focused on memory and learning that can be used to guide technical applications and enhance memory performance. Our team is very interested in memory and learning mechanisms, which have inspired our idea to use scientific background to improve memory.

## What it does

**Memory Playground is a web application that helps boost memory recall and retention through the Memory Palace technique, especially for senior citizens and those with Alzheimer’s and dementia.** The application allows users to pick a setting/environment and list out words that are related. Then, we create broad yet distinct categories for the words. From here, we have users practice classifying objects, allowing them to create visual mappings of images physically and mentally. This allows them to strengthen memory connections and enhance memory performance. We also give them other words that fall into those categories to expand on the established mental connections.

Memory Playground also uses an integration of zero-knowledge proofs. It stores uploaded data securely on servers and allows users to anonymously interact with the application without revealing personally identifiable information, which is ideal for those concerned about privacy.

## How we built it

We used the OpenAI API to prompt their GPT-4 model for category groupings and new objects. Then we used Together.AI's stable diffusion model for image generation. From there we connected the Python components to the web side using Fetch API. We built the web application with React, HTML, CSS, and JS. We used Flask to integrate the Python backend with the frontend.

## Challenges we ran into

1) Working with multiple servers
2) Integrating Flask with React
3) Learning to use multiple APIs to integrate various services
4) Implementing drag and drop functionality using use states
5) Limited credit and GPU usage
6) Utilizing multimodal machine learning models

## Accomplishments that we're proud of

We are proud of how we efficiently and quickly were able to understand and implement new technologies and concepts. We were new to Flask and a lot of the recent AI technology. We are also proud of how we worked as a team, ideation stages to product creation. Additionally, we are proud of how we were able to integrate multiple varying technologies.

## What we learned

We developed a lot of technical skills involving using APIs, prompt engineering, model optimization, Flask, managing multiple server applications, and web development. We also learned a lot about the practical applications of AI in healthcare towards a potential treatment of Alzheimer's and dementia. It is critical for us as a society to consider how we can *prevent*, not just treat, such diseases.

We also learned a lot about the engineering design process. Throughout the hackathon we went through research, ideation, designing, prototyping, and building phases. We gained a lot of skills through this process that helped us as we adapted to new technologies and grew our knowledge base.

## What's next for Memory Playground

1) Scaling: We are looking to make our application public and available to the community. This would involve cloud data storage (ex. Firestore) and increased efficiency to manage larger requests.
2) Speech-to-text recognition: We would like to implement a speech to text recognition model so that we can utilize verbal connections and improve accessibility.
## Inspiration

2 days before flying to Hack the North, Darryl forgot his keys and spent the better part of an afternoon retracing his steps to find it- But what if there was a personal assistant that remembered everything for you? Memories should be made easier with the technologies we have today.

## What it does

A camera records you as you go about your day to day life, storing "comic book strip" panels containing images and context of what you're doing as you go about your life. When you want to remember something you can ask out loud, and it'll use Open AI's API to search through its "memories" to bring up the location, time, and your action when you lost it. This can help with knowing where you placed your keys, if you locked your door/garage, and other day to day life.

## How we built it

The React-based UI interface records using your webcam, screenshotting every second, and stopping at the 9 second mark before creating a 3x3 comic image. This was done because having static images would not give enough context for certain scenarios, and we wanted to reduce the rate of API requests per image. After generating this image, it sends this to OpenAI's turbo vision model, which then gives contextualized info about the image. This info is then posted sent to our Express.JS service hosted on Vercel, which in turn parses this data and sends it to Cloud Firestore (stored in a Firebase database). To re-access this data, the browser's built in speech recognition is utilized by us along with the SpeechSynthesis API in order to communicate back and forth with the user. The user speaks, the dialogue is converted into text and processed by Open AI, which then classifies it as either a search for an action, or an object find. It then searches through the database and speaks out loud, giving information with a naturalized response.

## Challenges we ran into

We originally planned on using a VR headset, webcam, NEST camera, or anything external with a camera, which we could attach to our bodies somehow. Unfortunately the hardware lottery didn't go our way; to combat this, we decided to make use of MacOS's continuity feature, using our iPhone camera connected to our macbook as our primary input.

## Accomplishments that we're proud of

As a two person team, we're proud of how well we were able to work together and silo our tasks so they didn't interfere with each other. Also, this was Michelle's first time working with Express.JS and Firebase, so we're proud of how fast we were able to learn!

## What we learned

We learned about OpenAI's turbo vision API capabilities, how to work together as a team, how to sleep effectively on a couch and with very little sleep.

## What's next for ReCall: Memories done for you!

We originally had a vision for people with amnesia and memory loss problems, where there would be a catalogue for the people that they've met in the past to help them as they recover. We didn't have too much context on these health problems however, and limited scope, so in the future we would like to implement a face recognition feature to help people remember their friends and family.