## Inspiration

COVID19 hit business really hard and they are struggling to be profitable, and in fact, few are worried not to get into losses. Procurement is one of the important business aspects that are affected by COVID19 because investors and clients back off due to fear of getting into losses.

In business, procurement is the process of searching for, negotiating, and purchasing goods and services from suppliers. **Datafy** aims to support businesses by providing the details of the top legit companies that are crowned by The Fortune500 website, to whom we can approach for negotiating or business proposal.

Ever toured websites such as **Linkedin**, **Fortune500**, **Forbes** and believed it would have been exceptional if you could perceive all the *data* existing in the website in an easy-to-use format preferable a *CSV* or *JSON*, and then apply your codes or statistical analysis?
It's a **yes** for several *data scientists* and *business analysts* who frequently invest their time trying to find suitable data and regenerate it into a suitable format so that they can apply *data visualization* and other *data analytics*.
**Datafy** originated at that instant when I was once challenged by my internship company to accumulate the data of top retailing companies in the world. After a day of googling, I landed at a conclusion to use the **Fortune500** website as my reservoir of data. But why stop at one domain when I can apply it for all potential circumstances? And that's how it all started.

## What it does

Datafy is a **web scraped data provider website**, effortlessly download *CSV* and *JSON* Data of Companies listed in **Fortune 500**

## Datafy for:

1. **Developers**: CSV and JSON format make data convenient to handle and implement various data science tools, PowerBI, and it stands developer-friendly
2. **Procurement & Spend Analytics**: Datafy lets businesses discover the apt vendor or clients that can help their company reach the precise target audience and be lucrative
3. **Investors**: Datafy exhibits revenue change, profit rate, and other former statistics of companies which assists the investors and stockholders get sound shrewdness through data, and statistics
4. **Seekers**: Datafy provides the records of prestigious organizations honored by Fortune 500, where every student and a job seeker endeavors to be an employee

## How I built it

1. **Web Scrapping**: I used **Selenium**, **Chrome driver**, **python** for web scrapping the data from The Fortune500 website. [Github link to website code](https://github.com/Shakthi-Dhar/DatafyWebsite), [Github link to web scrapping tool](https://github.com/Shakthi-Dhar/DataScrapping_Fortune500), [GITHUB code repository](https://github.com/Shakthi-Dhar/DatafyWebsite)
2. **Database**: This data that has been scrapped is converted into *JSON* and *CSV* format and stored in **Firebase Cloud Storage**

3.**Website**: Developed a *beautiful web application* (for best UI use in laptop/desktop) that sends a request to download the data from the firebase cloud storage and saves it on the user's local system. [Check out the website](https://datafy-fortune500.herokuapp.com/)

## Challenges I ran into

1. **Deployment**: *Heroku* does not allow any request that lasts for more than *30sec* and *selenium web scrapping* is a time-consuming process, hence I need to switch to **Firebase** as my server and send a download request to the database rather than creating one instantaneously.
2. **UI**: The UI is not very compatible with mobile screen resolutions
3. **Download**: Creating a download request in *JSON* and *CSV* format was giving some internal server error when hosted.

## Accomplishments that I'm proud of

1. I have learned to use the **Firebase cloud storage** and how to send a download and insert request.
2. Making a **dark-themed UI** that is compatible with most of the device is a great achievement for me
3. Putting my web scrapping skills into help for other data scientists is a commendable achievement

## What I learned

1. Flask Cloud storage
2. HTML, CSS scroll button feature and cards system

## What's next for DATAFY

Currently, Datafy focuses on extracting data from *The Fortune500 website*, but in **LinkedIn**, there are more than **50 million companies** which I aim to extract information from and present it to all the Data scientists out there in the world.
## Inspiration

The idea behind the inspiration was from Raj, who invests periodically and enjoys playing around in the market. One thing he enjoys doing is looking at specific industries and finding the best value's in these specific industries. Currently, to do this he compares many different factors, balance sheets, and news stories while also looking at hundreds of companies trying to determine the best company in an industry for his investment. After utilizing the Marquee API and realizing that the API provides percentile data on calculated statistics that utilize many of the data points he considers when looking to invest, it was determined the the API could be utilized along with the company's industry to find how each company placed and changed with respect to time.

## What it does

The user is allowed to select an industry and different factors that are provided through the Marquee API. Whenever one of the selections is changed the charts change in real-time along with if the date range that is available is changed. The user can then visualize the companies in a specific industry and how they stack against each other in the chosen factor vs time. As two charts can be used, the user can compare multiple factors when having a deeper look into the specified industry.

## How we built it

Back-End: To get the information needed to visualize the marquee API is used to grab all the historical data points for the given range provided by the user through a call. This then runs a function which takes in the date-range and the industry and grabs the companies in the industry by utilizing pandas to find the companies in the given industry in a csv file. To get the companies with their specific industry, some web-scrapping was first done through websites that contain GICS classifications and appends a column to each company in a .csv and then then this csv file is used in conjunction with the given industry to grab all the companies into a list. This list can then be used to grab gsid which then can be used to request the data from the Marquee dataset. Then conversion of the data is done to put it into proper formatting for the front-end using pandas conversions and the eventual dataframe is converted to a csv for use with Dash
Front-End: Utilizing Dash's complex yet easy to use framework we were able to take in the values provided by the csv file and create an interactive API for the user. There are two graphs present on the main page as of now, for each of them you select what you want as your y-axis, the x-axis will be represented by time. Thereafter, the data is displayed with a slider in the middle of the two, this slider controls the date range and can help investors see trends for the companies within the same sector. To allow for more interactivity, we added hover features that allow users to see the values of all the companies within an industry on a set month.

## Challenges we ran into

Specifically in the backend, it seemed that we were constantly running into issues utilizing the Marquee API with limits and formatting to get the information we needed. When we eventually did get the information, there was some repeat data that needed to be filtered out. During the web scrapping process sometimes the GICS classifications were not as needed or not present due to some companies going under/getting acquired in the dataset so some research needed to be done to classify them manually.
I think for front-end the main issue was finding the best tool to use to help in visualizing the data. At first, we decided to use Dash as it was the only open source data tool that runs on python, the language most people in our team are comfortable. Then, we decided to switch over to React,js and its many libraries to represent the data however we found many constraints when trying to use this. So, we decided to switch back over to Dash as it was something we felt more familiar with and we had about 10 hours at this point.

## Accomplishments that we're proud of

* Figuring out Dash and its data vis. solutions
* Understanding Marquee API
* Having a SICK API!
* Not giving up when we saw us going nowhere

## What we learned

* "You always want to check you assumptions" --Max Guo(Goldman)

## What's next for Marqulityics

Ideally we want to be able to visualize more industries, show major events in said industries (mergers, acquisitions, etc.), and perhaps show more plots. One plot that is likely desired is one that plots all companies as a scatter and labels them according to industry by color. Then have a time progression that shows how these companies move against time so investors can be informed on trends of industries in relation to one another. Another big consideration for the future is implementing a way to baseline our plots utilizing something such as an index to better show how industries change with changes to the index.
## Inspiration

After learning about the current shortcomings of disaster response platforms, we wanted to build a modernized emergency services system to assist relief organizations and local governments in responding faster and appropriately.

## What it does

safeFront is a cross between next-generation 911 and disaster response management. Our primary users are local governments and relief organizations. The safeFront platform provides organizations and governments with the crucial information that is required for response, relief, and recovery by organizing and leveraging incoming disaster related data.

## How we built it

safeFront was built using React for the web dashboard and a Flask service housing the image classification and natural language processing models to process the incoming mobile data.

## Challenges we ran into

Ranking urgency of natural disasters and their severity by reconciling image recognition, language processing, and sentiment analysis on mobile data and reported it through a web dashboard. Most of the team didn't have a firm grasp on React components, so building the site was how we learned React.

## Accomplishments that we're proud of

Built a full stack web application and a functioning prototype from scratch.

## What we learned

Stepping outside of our comfort zone is, by nature, uncomfortable. However, we learned that we grow the most when we cross that line.

## What's next for SafeFront

We'd like to expand our platform for medical data, local transportation delays, local river level changes, and many more ideas. We were able to build a fraction of our ideas this weekend, but we hope to build additional features in the future.