## Inspiration

Getting tested for Covid19 is crucial to preventing the spread of the disease. In many countries such as Bangladesh, Myanmar, and Pakistan, testing is not as readily available nor is it as accessible as compared to wealthier countries like the USA or Canada. By taking advantage of our knowledge of machine learning and signal processing, we aim to bridge that gap by providing equal and equitable access to testing, using the internet as a means to reach places where getting access to a Covid19 test might be difficult.

## What it does

Our web app allows a user to upload an audio file containing the sounds of their coughing (whether it’s a forced cough or one that’s caused by some unidentified respiratory illness) to our site. Upon pressing the submission button, the audio file is processed, padded, and inserted into our trained model, which then predicts the likeliness that the audio file contains coughing caused by Covid19. This information is then displayed for the user to see.

## How we built it

We first scoured the internet for any sort of dataset on Covid19 and audio. We quickly found the [Project Coswara](https://github.com/iiscleap/Coswara-Data) dataset from the Indian Institute of Science, containing approximately 1300 individuals’ worth of coughing, vowel phonation, and breathing. Then we downloaded this data and processed it using Pandas to access the csv data that contained each individuals’ Covid19 postivity status (whether or not they had Covid) and using Librosa to process the audio. We converted each audio file into a Mel-frequency cepstrum (MFC) which is an approximation of a human’s sensitivity to sound. Then, we normalized all the MFCs to make them all the same length, and we fed them into our neural network, a relatively simple feed-forward network.
We trained our model over our data in in Google Colab, saved it, and downloaded it for use in our web application. We then created our web application to host the model and interface with our users.

## Challenges we ran into

Our first challenge was that we were unfamiliar with librosa and audio processing. This meant that our data processing step took much longer than we anticipated. Through intuition, a lot of Googling (thank God for StackOverflow), and a lot of elbow grease, we managed to process all of our data.

Our next challenge was actually training our network on our data. All of our computers are relatively slow, so training our model on our own computer was taking forever. Our solution was to upload our serialized input data and expected outputs to Google Drive, mount it to Google Colab, and train from there. Once training was completed, we were able to download our serialized model, and then mess around with our architecture to test different models.

Our last and final challenge was actually creating our web app. None of us had much experience with creating web apps reliant on a backend component, so we ended up learning Flask in order to create our web app.

## Accomplishments that We’re proud of

We’re most proud of actually being able to create our model in the first place. Starting off with this project, none of us had any experience working with Pytorch or Librosa. We didn’t even know if our project was feasible, and we encountered many obstacles along the way. Despite all of this, we were able to create a model with 93% accuracy.

## What we learned

We learned a lot about neural networks, signal processing, and most importantly – working together in a team.

## What's next for CoughDrop

Our next steps would be to continue working on our model. One of our ideas for our model originally was to implement ensemble learning and stacking multiple models under another neural network, then have those sub-models all inspect different sounds. One model might detect Covid19 in phonation of the vowel ‘a’, while another detects Covid19 in breathing, another in coughing, etc. Then the metalearner would take in all the outputs of those sub-models to form a more informed classification of the patients’ likeliness for having Covid19.

Another thing we’d like to work on would be making our project available to people in other languages. According to [the Harvard Business Review](https://hbr.org/2012/05/global-business-speaks-english) in 2012, approximately 1.75 billion people “speak English at a useful level”. This means that the large majority of the world will not be able to access our web app solely due to the issue of a language barrier.
## Inspiration:

Coronavirus is our modern-day villain. Millions of tests are conducted around the world every day. The tests come with long, tiring lines and pose an even greater threat to our wellbeing. As developers, we felt it was our duty to make this tiresome process safer (and quicker) with the help of technology! To limit exposure, we created Covid-19 Checkin so that you can report symptoms to your healthcare provider from the comfort of your own home — simply by talking to your Google Assistant!

## What it does:

Our application provides a Covid-19 check-up through Google Assistant by asking the user symptom related questions. The answers are directly transferred to a database where a health providing company (and the user themselves) can review and see if the patient should go do a Covid-19 test.

## How we built it:

We split the work into multiple roles: Shrey created the voiceFlow application for the "Covid-19 Checkin" and integrated it with google actions so that it could run on a Google home mini. He also created the backend for the website using node.js and Express. Rushil worked on the python feature to send an email to the doctor with the patients CSV file from voiceflow. Tasfiq worked with data provided from the VoiceFlow application and imported it into a Google database, which was Google Sheets. It was able to update old data and create new data to be viewed in a formal structure. Lastly, Ahmed created a website using Javascript, HTML, CSS and firebase. To talk more about the application, and provide details about what technology we used for our application.

## Challenges we ran into:

Working virtually was tougher than we thought it would have been as there is an obvious barrier for communication, exchanging ideas as well as the feeling of a community. With the time constraint and of course the sleep deprivation, we found it relatively stressful to make a large application in such a short amount of time.

## Accomplishments that we are proud of:

We successfully developed and launched a fully functional website with user-friendly UI, cross-platform. We also managed to create an advanced voice recognition AI app to minimize face-to-face interactions that might take place at the hospital reception which at the beginning of this hackathon none of us know how to do!!

## What we learned:

The most important thing we have learned is how to effectively time manage the tasks and collaborate with other teammates. We were able to do this by creating short and concise tasks for our team to work on for 4 hours before coming back to regroup.
In terms of technical skills, we've also honed our web development skill set including HTML, CSS and Javascript for the front-end. Then we deployed the front-end on Firebase. We also learned Voice input technology.

## What’s next for SafeCheckIn:

We believe that SafeCheckIn has the potential to be the go-to platform for the patient's initial interaction before visiting the clinic. Though they're just a few things that we believe that we must do before it can reach that level. These things are working on the website a little more as well as coming up with a few other features for users to interact with including specific requests by the patient. We also hope to bring more content to our website in terms of informative news articles. We would also love to implement some interactive graphs for users to interact with so that they can understand how using our platform can save lives!
# **Cough It**

#### COVID-19 Diagnosis at Ease

## Inspiration

As the pandemic has nearly crippled all the nations and still in many countries, people are in lockdown, there are many innovations in these two years that came up in order to find an effective way of tackling the issues of COVID-19. Out of all the problems, detecting the COVID-19 strain has been the hardest so far as it is always mutating due to rapid infections.

Just like many others, we started to work on an idea to detect COVID-19 with the help of cough samples generated by the patients. What makes this app useful is its simplicity and scalability as users can record a cough sample and just wait for the results to load and it can give an accurate result of where one have the chances of having COVID-19 or not.

## Objective

The current COVID-19 diagnostic procedures are resource-intensive, expensive and slow. Therefore they are lacking scalability and retarding the efficiency of mass-testing during the pandemic. In many cases even the physical distancing protocol has to be violated in order to collect subject's samples. Disposing off biohazardous samples after diagnosis is also not eco-friendly.

To tackle this, we aim to develop a mobile-based application COVID-19 diagnostic system that:

* provides a fast, safe and user-friendly to detect COVID-19 infection just by providing their cough audio samples
* is accurate enough so that can be scaled-up to cater a large population, thus eliminating dependency on resource-heavy labs
* makes frequent testing and result tracking efficient, inexpensive and free of human error, thus eliminating economical and logistic barriers, and reducing the wokload of medical professionals

Our [proposed CNN](https://dicova2021.github.io/docs/reports/team_Brogrammers_DiCOVA_2021_Challenge_System_Report.pdf) architecture also secured Rank 1 at [DiCOVA](https://dicova2021.github.io/) Challenge 2021, held by IISc Bangalore researchers, amongst 85 teams spread across the globe. With only being trained on small dataset of 1,040 cough samples our model reported:

Accuracy: 94.61%
Sensitivity: 80% (20% false negative rate)
AUC of ROC curve: 87.07% (on blind test set)

## What it does

The working of **Cough It** is simple. User can simply install the app and tap to open it. Then, the app will ask for user permission for external storage and microphone. The user can then just tap the record button and it will take the user to a countdown timer like interface. Playing the play button will simply start recording a 7-seconds clip of cough sample of the user and upon completion it will navigate to the result screen for prediction the chances of the user having COVID-19

## How we built it

Our project is divided into three different modules -->

#### **ML Model**

Our machine learning model ( CNN architecture ) will be trained and deployed using the Sagemaker API which is apart of AWS to predict positive or negative infection from the pre-processed audio samples. The training data will also contain noisy and bad quality audio sample, so that it is robust for practical applications.

#### **Android App**

At first, we prepared the wireframe for the app and decided the architecture of the app which we will be using for our case. Then, we worked from the backend part first, so that we can structure our app in proper android MVVM architecture. We constructed all the models, Retrofit Instances and other necessary modules for code separation.

The android app is built in Kotlin and is following MVVM architecture for scalability. The app uses Media Recorder class to record the cough samples of the patient and store them locally. The saved file is then accessed by the android app and converted to byte array and Base64 encoded which is then sent to the web backend through Retrofit.

#### **Web Backend**

The web backend is actually a Node.js application which is deployed on EC2 instance in AWS. We choose this type of architecture for our backend service because we wanted a more reliable connection between our ML model and our Node.js application.

At first, we created a backend server using Node.js and Express.js and deployed the Node.js server in AWS EC2 instance. The server then receives the audio file in Base64 encoded form from the android client through a POST request API call. After that, the file is getting converted to .wav file through a module in terminal through command. After successfully, generating the .wav file, we put that .wav file as argument in the pre-processor which is a python script. Then we call the AWS Sagemaker API to get the predictions and the Node.js application then sends the predictions back to the android counterpart to the endpoint.

## Challenges we ran into

#### **Android**

Initially, in android, we were facing a lot of issues in recording a cough sample as there are two APIs for recording from the android developers, i.e., MediaRecorder, AudioRecord. As the ML model required a .wav file of the cough sample to pre-process, we had to generate it on-device. It is possible with AudioRecord class but requires heavy customization to work and also, saving a file and writing to that file, is a really tedious and buggy process. So, for android counterpart, we used the MediaRecorder class and saving the file and all that boilerplate code is handled by that MediaRecorder class and then we just access that file and send it to our API endpoint which then converts it into a .wav file for the pre-processor to pre-process.

#### **Web Backend**

In the web backend side, we faced a lot of issues in deploying the ML model and to further communicate with the model with node.js application.

Initially, we deployed the Node.js application in AWS Lamdba, but for processing the audio file, we needed to have a python environment as well, so we could not continue with lambda as it was a Node.js environment. So, to actually get the python environment we had to use AWS EC2 instance for deploying the backend server.

Also, we are processing the audio file, we had to use ffmpeg module for which we had to downgrade from the latest version of numpy library in python to older version.

#### **ML Model**

The most difficult challenge for our ml-model was to get it deployed so that it can be directly accessed from the Node.js server to feed the model with the MFCC values for the prediction. But due to lot of complexity of the Sagemaker API and with its integration with Node.js application this was really a challenge for us. But, at last through a lot of documentation and guidance we are able to deploy the model in Sagemaker and we tested some sample data through Postman also.

## Accomplishments that we're proud of

Through this project, we are proud that we are able to get a real and accurate prediction of a real sample data. We are able to send a successful query to the ML Model that is hosted on Sagemaker and the prediction was accurate.

Also, this made us really happy that in a very small amount we are able to overcome with so much of difficulties and also, we are able to solve them and get the app and web backend running and we are able to set the whole system that we planned for maintaining a proper architecture.

## What we learned

Cough It is really an interesting project to work on. It has so much of potential to be one of the best diagnostic tools for COVID-19 which always keeps us motivated to work on it make it better.

In android, working with APIs like MediaRecorder has always been a difficult position for us, but after doing this project and that too in Kotlin, we feel more confident in making a production quality android app. Also, developing an ML powered app is difficult and we are happy that finally we made it.

In web, we learnt the various scenarios in which EC2 instance can be more reliable than AWS Lambda also running various script files in node.js server is a good lesson to be learnt.

In machine learning, we learnt to deploy the ML model in Sagemaker and after that, how to handle the pre-processing file in various types of environments.

## What's next for Untitled

As of now, our project is more focused on our core idea, i.e., to predict by analysing the sample data of the user. So, our app is limited to only one user, but in future, we have already planned to make a database for user management and to show them report of their daily tests and possibility of COVID-19 on a weekly basis as per diagnosis.

## Final Words

There is a lot of scope for this project and this project and we don't want to stop innovating. We would like to take our idea to more platforms and we might also launch the app in the Play-Store soon when everything will be stable enough for the general public.

Our hopes on this project is high and we will say that, we won't leave this project until perfection.