## 💡Inspiration

Every year, around 40 million tons of food is wasted in the United States. That is estimated to be around 35 percent, of the entire US food supply. When we waste food, we also waste all the energy and water it takes to grow, harvest, transport, and package it. And if food goes to the landfill and rots, it produces methane—a greenhouse gas that is 26x more potent than carbon dioxide.
Drilling down further into the problem of food waste, research shows that 43% of all waste in United States actually comes from homes, which is more than food services and grocery stores combined.
Reducing food waste is one of the easiest and most, powerful actions anyone can take to help sustainability, so this weekend, our team chose to build a IoT and mobile solution that tackles food waste at homes.

## 🤔 What it does

**GreenFridge** is a end-to-end IoT, data and mobile application solution that tracks what users have in their fridge, remind users of food that is about to go bad, and educate users on ways they can do their part in minimizing their carbon footprint. Its key features include:

* Mobile feature 1: Displays live view of food items currently in the fridge and their shelf life
* Mobile feature 2: Displays fridge power usage, temperature and humidity data, CO2 emissions per week, correlation between number of times fridge door was open and power.
* Mobile feature 3: Recommends sustainable alternatives and way to offset carbon dioxide emissions of food items currently in fridge
* Integration 1: Twilio messages with reminders of food that is going to go bad today and recommends ways to preserve them

[![IMG-2177.jpg](https://i.postimg.cc/NM561vVm/IMG-2177.jpg)](https://postimg.cc/cgGtS2dL)

[![Screen-Shot-2022-09-04-at-7-22-21-AM.jpg](https://i.postimg.cc/B6qysbjs/Screen-Shot-2022-09-04-at-7-22-21-AM.jpg)](https://postimg.cc/Z9MjxTPD)

## 🦾 How we built it

* **IoT:** Raspberry Pi Camera Local Object Detection, Humidity Sensor, Temperature Sensor
* **Backend:** Postman, Google Cloud Platform, Flask
* **Web Scraping:** Python
* **Google Cloud Services:** AutoML, Vision, Vertex AI, Compute Engine, Maps
* **Database:** Cockcroach DB, Cloud Storage, FireStore
* **Twilio:** Auto SMS
* **Frontend:** React, JS, HTML, CSS, Figma

[![Screen-Shot-2022-09-04-at-6-06-14-AM.jpg](https://i.postimg.cc/k4vd7Wjj/Screen-Shot-2022-09-04-at-6-06-14-AM.jpg)](https://postimg.cc/5Yyr3QWv)

## 👨🏻‍🤝‍👨🏽 (PennApps Track) Sustainability Prize

We believe, and a lot of research also points to that, reducing food waste is the easiest and most powerful way to make a dent in sustainability. Like the Nest Thermostat that saves users ~13% on energy bills (usage), we envision GreenFridge to be in every home, helping users on food waste reduction and educating them on carbon footprint reduction, thus bringing outsized impact on sustainability.

## 🛠 (Sponsored by Citadel & Penn Detkin Lab) Best Hardware Hack

Thank you Penn Detkin Lab for lending us Raspberry Pi camera and other tools late at night, so that we can use local object detection to enable the rest of our solution!

## ☁️ Best Use of Google Cloud - MLH

Our application uses Vertex AI (Auto ML) for Custom ML model, Compute Engine to deploy flask server, Cloud Storage to store images for training, FireStore to update data of items in fridge in real time, Connectors AI.

## 🤖 Most Creative Use of Twilio - MLH

We used Twilio SMS to serve reminders of food that is going to go bad on that day and recommendations on ways to preserve them. Twilio was so intuitive to use, even for someone who is completely new!

[![Screen-Shot-2022-09-04-at-7-13-02-AM.jpg](https://i.postimg.cc/Y0PQXn5Z/Screen-Shot-2022-09-04-at-7-13-02-AM.jpg)](https://postimg.cc/p5D9rZXJ)

## Challenges we ran into

* We tried hard and followed documentation, but could not get Cockroach DB network setup

## Accomplishments that we're proud of

* Finding an object detection algorithm that is efficient on Raspberry Pi
* Interfacing humidity and temperature sensors, modeling surge in power consumption over time
* Training custom computer vision model and deploying for real time predictions
* Setting up Google Cloud infrastructure to enable efficient use of their tech stack - compute engine, API services, ML toolkit, data storage

## What we learned

* We were surprised by technology development that allow even learners to deploy complex technologies
* We were surprised by how big the opportunity it is to make a dent in sustanability through reducing food waste
# Omakase

*"I'll leave it up to you"*

## Inspiration

On numerous occasions, we have each found ourselves staring blankly into the fridge with no idea of what to make. Given some combination of ingredients, what type of good food can I make, and how?

## What It Does

We have built an app that recommends recipes based on the food that is in your fridge right now. Using Google Cloud Vision API and Food.com database we are able to detect the food that the user has in their fridge and recommend recipes that uses their ingredients.

## What We Learned

Most of the members in our group were inexperienced in mobile app development and backend. Through this hackathon, we learned a lot of new skills in Kotlin, HTTP requests, setting up a server, and more.

## How We Built It

We started with an Android application with access to the user’s phone camera. This app was created using Kotlin and XML. Android’s ViewModel Architecture and the X library were used. This application uses an HTTP PUT request to send the image to a Heroku server through a Flask web application. This server then leverages machine learning and food recognition from the Google Cloud Vision API to split the image up into multiple regions of interest. These images were then fed into the API again, to classify the objects in them into specific ingredients, while circumventing the API’s imposed query limits for ingredient recognition. We split up the image by shelves using an algorithm to detect more objects. A list of acceptable ingredients was obtained. Each ingredient was mapped to a numerical ID and a set of recipes for that ingredient was obtained. We then algorithmically intersected each set of recipes to get a final set of recipes that used the majority of the ingredients. These were then passed back to the phone through HTTP.

## What We Are Proud Of

We were able to gain skills in Kotlin, HTTP requests, servers, and using APIs. The moment that made us most proud was when we put an image of a fridge that had only salsa, hot sauce, and fruit, and the app provided us with three tasty looking recipes including a Caribbean black bean and fruit salad that uses oranges and salsa.

## Challenges You Faced

Our largest challenge came from creating a server and integrating the API endpoints for our Android app. We also had a challenge with the Google Vision API since it is only able to detect 10 objects at a time. To move past this limitation, we found a way to segment the fridge into its individual shelves. Each of these shelves were analysed one at a time, often increasing the number of potential ingredients by a factor of 4-5x. Configuring the Heroku server was also difficult.

## Whats Next

We have big plans for our app in the future. Some next steps we would like to implement is allowing users to include their dietary restriction and food preferences so we can better match the recommendation to the user. We also want to make this app available on smart fridges, currently fridges, like Samsung’s, have a function where the user inputs the expiry date of food in their fridge. This would allow us to make recommendations based on the soonest expiring foods.
## Inspiration

One of the greatest challenges facing our society today is food waste. From an environmental perspective, Canadians waste about *183 kilograms of solid food* per person, per year. This amounts to more than six million tonnes of food a year, wasted. From an economic perspective, this amounts to *31 billion dollars worth of food wasted* annually.

For our hack, we wanted to tackle this problem and develop an app that would help people across the world do their part in the fight against food waste.

We wanted to work with voice recognition and computer vision - so we used these different tools to develop a user-friendly app to help track and manage food and expiration dates.

## What it does

greenEats is an all in one grocery and food waste management app. With greenEats, logging your groceries is as simple as taking a picture of your receipt or listing out purchases with your voice as you put them away. With this information, greenEats holds an inventory of your current groceries (called My Fridge) and notifies you when your items are about to expire.

Furthermore, greenEats can even make recipe recommendations based off of items you select from your inventory, inspiring creativity while promoting usage of items closer to expiration.

## How we built it

We built an Android app with Java, using Android studio for the front end, and Firebase for the backend. We worked with Microsoft Azure Speech Services to get our speech-to-text software working, and the Firebase MLKit Vision API for our optical character recognition of receipts. We also wrote a custom API with stdlib that takes ingredients as inputs and returns recipe recommendations.

## Challenges we ran into

With all of us being completely new to cloud computing it took us around 4 hours to just get our environments set up and start coding. Once we had our environments set up, we were able to take advantage of the help here and worked our way through.

When it came to reading the receipt, it was difficult to isolate only the desired items. For the custom API, the most painstaking task was managing the HTTP requests. Because we were new to Azure, it took us some time to get comfortable with using it.

To tackle these tasks, we decided to all split up and tackle them one-on-one. Alex worked with scanning the receipt, Sarvan built the custom API, Richard integrated the voice recognition, and Maxwell did most of the app development on Android studio.

## Accomplishments that we're proud of

We're super stoked that we offer 3 completely different grocery input methods: Camera, Speech, and Manual Input. We believe that the UI we created is very engaging and presents the data in a helpful way. Furthermore, we think that the app's ability to provide recipe recommendations really puts us over the edge and shows how we took on a wide variety of tasks in a small amount of time.

## What we learned

For most of us this is the first application that we built - we learned a lot about how to create a UI and how to consider mobile functionality. Furthermore, this was also our first experience with cloud computing and APIs. Creating our Android application introduced us to the impact these technologies can have, and how simple it really is for someone to build a fairly complex application.

## What's next for greenEats

We originally intended this to be an all-purpose grocery-management app, so we wanted to have a feature that could allow the user to easily order groceries online through the app, potentially based off of food that would expire soon.

We also wanted to implement a barcode scanner, using the Barcode Scanner API offered by Google Cloud, thus providing another option to allow for a more user-friendly experience. In addition, we wanted to transition to Firebase Realtime Database to refine the user experience.

These tasks were considered outside of our scope because of time constraints, so we decided to focus our efforts on the fundamental parts of our app.