## What Inspired us

Due to COVID, many students like us have become accustomed to work on their schoolwork, projects and even hackathons remotely. This led students to use online resources at their disposal in order to facilitate their workload at home. One of the tools most used is “ctrl+f” which enables the user to quickly locate any text within a document. Thus, we came to a realisation that no such accurate method exists for images. This led to the birth of our project for this hackathon titled “PictoDocReader”.

## What you learned

We learned how to implement Dash in order to create a seamless user-interface for Python. We further learnt several 2D and 3D pattern matching algorithms such as, Knuth-Morris-Pratt, Bird Baker, Karp and Rabin and Aho-Corasick. However, only implemented the ones that led to the fastest and most accurate execution of the code.
Furthermore, we learnt how to convert PDFs to images (.png). This led to us learning about the colour profiles of images and how to manipulate the RGB values of any image using the numpy library along with matplotlib. We also learnt how to implement Threading in Python in order to run tasks simultaneously. We also learnt how to use Google Cloud services in order to use Google Cloud Storage to enable users to store their images and documents on the cloud.

## How you built your project

The only dependencies we required to create the project were PIL, matplotlib, numpy, dash and Google Cloud.

**PIL** - Used for converting a PDF file to a list of .png files and manipulating the colour profiles of an image.

**matplotlib** - To plot and convert an image to its corresponding matrix of RGB values.

**numpy** - Used for data manipulation on RGB matrices.

**dash** - Used to create an easy to use and seamless user-interface

**Google Cloud** - Used to enable users to store their images and documents on the cloud.

All the algorithms and techniques to parse and validate pixels were all programmed by the team members. Allowing us to cover any scenario due to complete independence from any libraries.

## Challenges we faced

The first challenge we faced was the inconsistency between the different RGB matrices for different documents. While some matrices contained RGB values, others were of the form RGBA. Therefore, this led to inconsistent results when we were traversing the matrices. The problem was solved using the slicing function from the numpy library in order to make every matrix uniform in size.

Trying to research best time complexity for 2d and 3d pattern matching algorithms. Most algorithms were designed for square images and square shaped documents. While we were working with any sized images and documents. Thus, we had to experiment and alter the algorithms to ensure they worked best for our application.

When we worked with large PDF files, the program tried to locate the image in each page one by one. Thus, we needed to shorten the time for PDFs to be fully scanned to make sure our application performs its tasks in a viable time period. Hence, we introduced threading into the project to reduce the scanning time when working with large PDF files as each page was scanned simultaneously. Although we have come to the realisation that threading is not ideal as the multi-processing greatly depends on the number of CPU cores of the user’s system. In an ideal world we would implement parallel processing instead of threading.
## Inspiration

Our inspiration for creating this AI project stemmed from the desire to leverage current technology to enhance the lives of our community members, particularly the elderly. Recognizing the difficulties faced by aging individuals, including our own grandparents, to embrace technological advancements, and their concerns about health conditions, we sought a way to integrate technology and healthcare services seamlessly into their lives. Thus, SageWell was conceived—an AI companion designed to provide personalized support to seniors by navigating medical information alongside their healthcare providers. At SageWell, we believe in accessibility of AI for all age groups.

## What it does

SageWell enables users to seek medical information through natural language interactions, allowing elderly individuals to pose questions verbally and receive spoken responses, mimicking human conversation.

## How we built it

SageWell leverages the capabilities of two Monster APIs: OpenAI-Whisper Large-v2 for speech-to-text conversion and Meta's llama-2-7b-chat-hf for refining our reinforcement learning model. Our model was trained on MedQuAD, a comprehensive dataset containing over 47,000 medical question-answer pairs, Drugs, Side Effects and Medical Condition dataset, and Drugs Related to Medical Conditions dataset. The frontend of the web app was built using React, a JavaScript library, and the service we integrated into our app to store all the audio files was Firebase.

## Challenges we ran into

In the process of developing SageWell, we encountered several challenges. Since there were many integrations in our application, from speech-to-text transcription and MonsterAPIs for fine-tuned LLMs, to storage service providers, we faced difficulties trying to link all the individual pieces together. We learned several new tools and technologies and also spent time fine-tuning models to provide medical information. Additionally, we encountered setbacks when we were finishing up our project as we were facing CORS errors when making API calls from the browser - we were able to work around this by adding a proxy, which served as a bridge between our client and server.

## Accomplishments that we're proud of

We empowered our primary user demographic, the elderly, to engage with SageWell through voice interactions instead of having to type text. We addressed the challenges that many seniors encounter when typing on digital devices, thereby increasing the accessibility of the process of seeking medical information and ensuring inclusivity in AI for all age groups, especially the elderly.

Our reinforcement learning model has demonstrated effectiveness, evidenced by the decrease in training loss after each iteration and to 0.756 at the end. This indicates that the model fits well on the training data.

We merged three datasets: the Medical Question Answering data set, the Drugs, Side Effects and Medical Condition data set, and the Drugs Related to Medical Conditions data set. By training our model for SageWell on these datasets obtained from NIH websites and drugs.com, we allowed SageWell to provide medical information to users based on reliable, comprehensive databases.

## What we learned

Through this project, we gained valuable insights into the startup ecosystem and developer space. We expanded our skill set by refining our model, working with new APIs and storage providers, and creating a solution that addresses the specific challenges faced by our target audience.

## What's next for SageWell

The journey of SageWell is just beginning. Moving forward, we aim to expand its capabilities to assist in additional areas crucial for the well-being of the elderly, such as medication reminders and guidance on accessing support for domestic chores. Furthermore, we envision integrating features that facilitate connections between seniors and younger generations, including their grandchildren and other youth in their communities. By fostering these intergenerational connections, SageWell will expand its targeting market size by not only keeping the elderly engaged with their loved ones but also ensuring they remain connected to the evolving world around them. Through SageWell, we look forward to continuing to push for accessibility of AI for all age groups.
# Babble: PennAppsXVIII

PennApps Project Fall 2018

## Babble: Offline, Self-Propagating Messaging for Low-Connectivity Areas

Babble is the world's first and only chat platform that is able to be installed, setup, and used 100% offline. This platform has a wide variety of use cases such as use in communities with limited internet access like North Korea, Cuba, and Somalia. Additionally, this platform would be able to maintain communications in disaster situations where internet infrastructure is damaged or sabotaged. ex. Warzones, Natural Disasters, etc.

### Demo Video

See our project in action here: <http://bit.ly/BabbleDemo>

[![](https://res.cloudinary.com/devpost/image/fetch/s--mKp50BXW--/c_limit,f_auto,fl_lossy,q_auto:eco,w_900/http://img.youtube.com/vi/M5dz9_pf2pU/0.jpg)](http://www.youtube.com/watch?v=M5dz9_pf2pU)

## Offline Install & Setup

Babble (a zipped APK) is able to be sent from one user to another via Android Beam. From there it is able to be installed. This allows any user to install the app just by tapping their phone to that of another user. This can be done 100% offline.

## Offline Send

All Babble users connect to all nearby devices via the creation of a localized mesh network created using the Android Nearby Connections API. This allows for messages to be sent directly from device to device via m to n peer to peer as well as messages to be daisy chain sent from peer to peer to ... to peer to peer.

Each Babble user's device keeps a localized ledger of all messages that it has sent and received, as well as an amalgamation of all of the ledgers of every device that this instance of Babble has been connected directly to via Android Nearby.

The combination of the Android Nearby Connections API with this decentralized, distributed ledger allows for messages to propagate across mesh networks and move between isolated networks as users leave one mesh network and join another.

## Cloud Sync when Online

Whenever an instance of Babble gains internet access, it uploads a copy of its ledger to a MongoDB Atlas Cluster running on Google Cloud. There the local ledger is amalgamated with the global ledger which contains all messages sent world wide. From there the local copy of the ledger is updated from the global copy to contain messages for nearby users.

## Use Cases

### Internet Infrastructure Failure: Natural Disaster

Imagine a natural disaster situation where large scale internet infrastructure is destroyed or otherwise not working correctly. Only a small number of users of the app would be able to distribute the app to all those affected by the outage and allow them to communicate with loved ones and emergency services. Additionally, this would provide a platform by which emergency services would be able to issue public alerts to the entire mesh network.

### Untraceable and Unrestrictable Communication in North Korea

One of the future directions we would like to take this would be a Ethereum-esq blockchain based ledger. This would allow for 100% secure, private, and untraceable messaging. Additionally the Android Nearby Connections API is able to communicate between devices via, cellular network, Wifi, Bluetooth, NFC, and Ultrasound which makes our messages relatively immune to jamming. With the mesh network, it would be difficult to block messaging on a large scale.

As a result of this feature set, Babble would be a perfect app to allow for open and unobstructed, censored, or otherwise unrestricted communication inside of a country with heavily restricted internet access like North Korea.

### Allowing Cubans to Communicate with Family and Friends in the US

Take a use case of a Cuba wide roll out. There will be a limited number of users in large cities like Havana or Santiago de Cuba that will have internet access as well as a number of users distributed across the country who will have occasional internet access. Through both the offline send and the cloud sync, 100% offline users in cuba would be able to communicate with family stateside.

## Future Goals and Directions

Our future goals would be to build better stability and more features such as image and file sharing, emergency messaging, integration with emergency services and the 911 decision tree, end to end encryption, better ledger management, and conversion of ledger to Ethereum-esq anonymized blockchain to allow for 100% secure, private, and untraceable messaging.

Ultimately, the most insane use of our platform would be as a method for rolling out low bandwidth internet to the offline world.

Name creds go to Chris Choi