# Inspiration 💡

Every day, doctors across the globe use advanced expertise and decades of medical breakthroughs to diagnose patients and craft unique prescriptions. The inspiration for this mobile application stems from the irony that the result of such precision is without fail, the chicken scratch found on a doctor’s note. The physician not only entrusts that the patient will keep track of the crumpled paper but they require that the pharmacist on-call will understand their professional scribble. The plan is to create a platform that leverages technology to streamline the prescription filling process, making it easier for doctors to authenticate their work and for patients and pharmacists alike to be confident in their prescriptions.

# What it does 🚀

The mobile application is designed to streamline prescription filling for patients, physicians and pharmacists. It starts with the written doctor’s note which is scanned via the mobile app and transcribed in real-time, allowing for them to confirm the prescription, directly edit it or retake the scan. This allows physicians to authenticate the interpretation of their prescription before shipping it off to the pharmacy via a shared patient database. During registration, the patient volunteers personal information which populates the database and ensures that general questions such as age, address and insurance are only to be answered once. In conjunction with the transcribed prescription, this information will be used to fill any necessary pharmaceutical forms that are scanned via the app. With completed paperwork, a transcribed description and verified patient information, pharmacists are significantly less likely to make errors fulfilling patient orders.

# How we built it 🏗️

To produce this mobile application, we utilized a diverse technology stack to integrate various components and create an uninterrupted product experience. Using our defined user types (patient, physician and pharmacist), we derived the necessary functions for each, prioritizing and placing them to create an intuitive UI. This paved the way for early design wireframes and an eventual high-fidelity Figma prototype which directed our front-end development in ReactNative. On the back-end, the light-weight Python framework, Flask, was used to handle registration, data transferring and transcription. Our application required keeping track of a large amount of data, which we stored/accessed within a Redis Cloud database. In order to accurately interpret text from forms and notes provided by doctors/pharmacies, we utilised the Google Cloud Vision OCR API as well as Gemini Pro, providing users with accurate transcriptions within images that were processed with the Pillow API. Then, in order to autocomplete forms with efficiency and accuracy, we deployed the OpenAI API as an LLM model, generating prompts through information found within the Redis Cloud database to fill out forms with the correct answers. The back-end of our project was developed parallel to our front-end, which was entirely built using react-native, capable of supporting both android and IOS devices. By using the navigation library, the various components of the application are split into their own pages with styling and functions unique to each one.

## Form filling process:

* Read form
	+ Extract text and location using Google Cloud OCR
	+ Group text into coherent groups (Vertical + horizontal coordinate comparison and LLMs
	+ Detect fillable fields and clean punctuation (LLM and Python)
* Answer fields (LLM)
* Write to form (Python Pillow library)

# Challenges we ran into 🧩

Throughout our project, we faced several obstacles which we ultimately overcame with perseverance and targeted learning. Before writing a single line of code, we discussed our interests, skillsets and project ambitions, finding clear differences which would necessitate a degree of compromise. Moreover, we decided to implement a tech stack with Flask on the backend and React-native on the frontend, which proved to be frustrating as they could have been much more complementary. As a result, we had a handful of avoidable complications given a better-composed tech stack. Given the shorter nature of this particular hackathon, we were very much forced to stick with the decisions we made early on without much room to pivot, greatly improving our critical thinking and debugging skills. Another issue we ran into was the inconsistency that we found within practices in the medical industry, as doctors notes and pharmaceutical forms often differed widely from one person/company to another, meaning we had to build scripts that would satisfy a wide variety of possibilities.

# Accomplishments that we're proud of 🏆

Ultimately, we are extremely proud that we were able to successfully build a full-stack mobile application within a 24-hour window, especially given that 3 of our members were not particularly experienced in app development before this hackathon. However, we were all able to find ways to contribute to the project whether it be through design elements, programming, or pitching. We are also extremely happy with the number of different libraries/APIs that we were able to put to use in this project. Having the experience of working with these APIs, many of which were for the first time, was extremely exciting for us and allowed us to build a product that we found to be extremely interesting.

# What we learned 🧠

Through this experience, all members of our group came away with an enhanced skillset. We each improved our developing expertise as despite our lack of experience with the chosen technologies and tools, we ultimately built a strong project that applied those tools well. Beyond just coding, we also improved our product thinking, prioritizing the needs of each end-user to design a well-rounded and truly valuable mobile application. Above all, we learned the importance of programming as a collaborative process, finding success by sharding responsibility between members and optimizing workloads with everyone's unique skill set. We are beyond excited to apply our updated mindset to participate in future hackathons.

# What's next for PharmFill 🚀

We are interested in continuing our work with PharmFill, with a focus on adding more features and polishing those already implemented. Our belief in PharmFill's potential to significantly impact the medical industry fuels our enthusiasm. A key area of development is the creation of more complex form-filling algorithms, capable of handling all types of forms. This advancement will not only enhance the app's functionality but also solidify its position as a transformative tool in healthcare.
## Inspiration

Staying healthy is a goal that everyone strives to achieve, but sometimes life just gets in the way. For the average person, slacking off for a few days is not a big deal and is something that is often done. However, for the more vulnerable population, staying on top their health is crucial and can often feel like a burden.

In 2017, a study looked into people's ability to adhere to their medication. This [study](https://www.sciencedirect.com/science/article/abs/pii/S016926071630579X) found that 80% of the time individuals delayed taking their medication, while 45% of the time they completely forgot!

We set out to create a personalized medication management app that would be both accessible and easy-to-use app. We wanted to specifically focus on the more vulnerable population (those with disabilities: Alzheimer's, severe arthritis, etc) of senior citizens to easily keep on top of their health.

Another common issue we found from anecdotal experience, is that senior citizens often dislike using technology and find it to be overwhelming. We wanted to bridge this gap and make the user experience for seniors as seamless as possible and take away the burden of unfamiliarity.

## What it does

PrescribePal is a smart health app with an emphasis on accessibility. Its main features are:

* Creating prescription order requests using OCR and voice interactions
* Giving friendly reminders to users to take their medication
* Easily connecting patients with professionals at a moment's notice

PrescribePal allows the user to opt-in to receiving timely notifications throughout the day or week. These notifications remind users to take their specified medication and amount.

Our app will also automate the process of refilling medication and supplements. Users can upload pictures of their medication to our app, and trust that PrescribePal will be able to automatically refill that medication upon request. Upon making a request, PrescribePal will also send out a text message to a caregiver or doctor for the appropriate approval.

Finally, users will be able to get in touch with their medical professionals at the click of a button. Our app will feature an online meeting platform where users can meet virtually with their doctors.

## How we built it

We built PrescribePal using the MERN stack as well as the following:

Socket.io and peer.js for video calling

Tesseract.js to extract text from images taken by patients

Cohere.ai to tokenize large chunks of text found on prescription labels

Puppeteer to find relevant product data

Alan.ai to create an easy to use voice assistant for patients

AWS to host and fetch our images

## Challenges we ran into

Trying to integrate our project with various different API's and libraries proved to be a difficult challenge. Working to integrate the entire end-to-end flow of our app starting from an image and transforming that into request for medication as well as connecting with Amazon. Our team barely had any prior experience working with these technologies in small applications, let alone creating a full scale project. Additionally, learning to work with so many requests and passing them between different libraries was extremely time consuming. After a significantly underestimated amount of effort, our team was finally able to integrate our Tesseract.js image conversion, Cohere AI text tokenization, Puppeteer.js web-scraper, and Twilio SMS notifications all into the desired degree of completeness.

## What we learned

Having spent so much time working with API's and formatting requests, our team can confidently say we are much more experienced. Understanding how to properly parse through hundreds of lines of JSON objects was a big lesson. Team members each got their own opportunity to refine and improve upon previously lacking skills. Further exposure to front-end development was a learning experience for one, and for the other it was greater understanding of push notifications from a server-client aspect.

## What's next for PrescribePal

To further improve the functionality and accessibility of PrescribePal:

* We'd like to improve the user interface to make it even more accessible for users
* Create a more personalized virtual video calling room for doctors with features like note taking, text chat and white boarding
* Reduce latency on notifications being received by patients for a more seamless experience
**check out the project demo during the closing ceremony!**
<https://youtu.be/TnKxk-GelXg>

## Inspiration

On average, half of patients with chronic illnesses like heart disease or asthma don’t take their medication. Reports estimates that poor medication adherence could be costing the country $300 billion in increased medical costs.

So why is taking medication so tough? People get confused and people forget.

When the pharmacy hands over your medication, it usually comes with a stack of papers, stickers on the pill bottles, and then in addition the pharmacist tells you a bunch of mumble jumble that you won’t remember.

<http://www.nbcnews.com/id/20039597/ns/health-health_care/t/millions-skip-meds-dont-take-pills-correctly/#.XE3r2M9KjOQ>

## What it does

The solution:
How are we going to solve this? With a small scrap of paper.

NekoTap helps patients access important drug instructions quickly and when they need it.

On the pharmacist’s end, he only needs to go through 4 simple steps to relay the most important information to the patients.

1. Scan the product label to get the drug information.
2. Tap the cap to register the NFC tag. Now the product and pill bottle are connected.
3. Speak into the app to make an audio recording of the important dosage and usage instructions, as well as any other important notes.
4. Set a refill reminder for the patients. This will automatically alert the patient once they need refills, a service that most pharmacies don’t currently provide as it’s usually the patient’s responsibility.

On the patient’s end, after they open the app, they will come across 3 simple screens.

1. First, they can listen to the audio recording containing important information from the pharmacist.
2. If they swipe, they can see a copy of the text transcription. Notice how there are easy to access zoom buttons to enlarge the text size.
3. Next, there’s a youtube instructional video on how to use the drug in case the patient need visuals.
Lastly, the menu options here allow the patient to call the pharmacy if he has any questions, and also set a reminder for himself to take medication.

## How I built it

* Android
* Microsoft Azure mobile services
* Lottie

## Challenges I ran into

* Getting the backend to communicate with the clinician and the patient mobile apps.

## Accomplishments that I'm proud of

Translations to make it accessible for everyone! Developing a great UI/UX.

## What I learned

* UI/UX design
* android development