## Inspiration

When working in the kitchen, we have to handle many different ingredients, wet foods, and big messes. Constantly searching for the small things you need constantly takes time and is often frustrating. Even more so, to switch back and forth between wet and dry ingredients means frequent hand washing. Messy situations like these are prime examples of where the internet of things and voice commands are most useful, yet there are disappointingly few tools to handle even the most basic tasks. Aroma addresses these challenges by providing a way to automatically manage your inventory of spices and seasonings while also automatically delivering them precisely to you.

## What it does

Aroma is a seasoning delivery tool that can hold multiple spices simultaneously while quickly and precisely providing them to you. To use it, all you have to do is place a bowl or plate under the device and then select what you would like. You can do this using either the in-built touch screen, or by asking a Google Home to dole out specific quantities. Even better, if you're currently working on a recipe, the Google Assistant can automatically provide the seasonings you need at the moment you need them.

## How we built it

The hardware for Aroma is built atop Android Things. Using a custom designed chassis, we developed an interlocking mechanism that can attach to the lids of standard seasoning containers and then automatically measure and dispense the contents. A great deal of time was spent making the system easy to use as a standalone platform. As one might expect, all you need to start is to select the spice from the embedded touch screen. Once the mechanism has adjusted itself, a button becomes available that allows you to keep delivering the selected spice while it is held. Furthermore, it will indicate in real time exactly how much has been dispensed.

However, since our main focus was to make this usable totally hands free, we designed an extensive Google Assistant backend. Not only can the Aroma assistant dispense exact quantities of seasoning, it can also read your recipes aloud and provide the appropriate seasoning where necessary. This was implemented using the actions-on-google API hosted on Firebase.

## Challenges we ran into

Engineering the mechanisms to be able to hold and dispense spices was tricky. In order for the system to be useful, it had to know exactly how much was being released, and could only deliver when it was supposed to. This required a high degree of engineering that isn't usually necessary during a hackathon. As such, the hardest challenge we faced was developing the delivery mechanism. We went through nearly 6 designs before settling on the current one.

## Accomplishments that we're proud of

Not only were we able to deliver a single spice, we were able to deliver any of three spices without requiring a separate servor for each. As such, this technique can be trivially extended to handle far more spices.

## What we learned

Integrating actions-on-google with Android Things was surprisingly easy. We were able to develop reactive message passing with only minimal work. This was also our first project where we did any significant manufacturing, and we learned how to quickly design and iterate to develop a functioning system.

## What's next for Aroma

The implementation of Aroma's interlocking mechanism is somewhat crude due to the materials available. However, with access to better fabrication tools, the whole design could be shrunk signifcantly, allowing denser storage of spices. In addition, there are many ways we could make Aroma smarter, such as determining from context when you might need an ingredient.
## Inspiration

IoT devices are extremely useful; however, they come at a high price. A key example of this is a smart fridge, which can cost thousands of dollars. Although many people can't afford this type of luxury, they can still greatly benefit from it. A smart fridge can eliminate food waste by keeping an inventory of your food and its freshness. If you don't know what to do with leftover food, a smart fridge can suggest recipes that use what you have in your fridge. This can easily expand to guiding your food consumption and shopping choices.

## What it does

FridgeSight offers a cheap, practical solution for those not ready to invest in a smart fridge. It can mount on any existing fridge as a touch interface and camera. By logging what you put in, take out, and use from your fridge, FridgeSight can deliver the very same benefits that smart fridges provide. It scans barcodes of packaged products and classifies produce and other unprocessed foods. FridgeSight's companion mobile app displays your food inventory, gives shopping suggestions based on your past behavior, and offers recipes that utilize what you currently have.

## How we built it

The IoT device is powered by Android Things with a Raspberry Pi 3. A camera and touchscreen display serve as peripherals for the user. FridgeSight scans UPC barcodes in front of it with the Google Mobile Vision API and cross references them with the UPCItemdb API in order to get the product's name and image. It also can classify produce and other unpackaged products with the Google Cloud Vision API. From there, the IoT device uploads this data to its Hasura backend.

FridgeSight's mobile app is built with Expo and React Native, allowing it to dynamically display information from Hasura. Besides using the data to display inventory and log absences, it pulls from the Food2Fork API in order to suggest recipes. Together, the IoT device and mobile app have the capability to exceed the functionality of a modern smart fridge.

## Challenges we ran into

Android Things provides a flexible environment for an IoT device. However, we had difficulty with initial configuration. At the very start, we had to reflash the device with an older OS because the latest version wasn't able to connect to WiFi networks. Our setup would also experience power issues, where the camera took too much power and shut down the entire system. In order to avoid this, we had to convert from video streaming to repeated image captures. In general, there was little documentation on communicating with the Raspberry Pi camera.

## Accomplishments that we're proud of

Concurring with Android Things's philosophy, we are proud of giving accessibility to previously unaffordable IoT devices. We're also proud of integrating a multitude of APIs across different fields in order to solve this issue.

## What we learned

This was our first time programming with Android Things, Expo, Hasura, and Google Cloud - platforms that we are excited to use in the future.

## What's next for FridgeSight

We've only scratched the surface for what the FridgeSight technology is capable of. Our current system, without any hardware modifications, can notify you when food is about to expire or hasn't been touched recently. Based on your activity, it can conveniently analyze your diet and provide healthier eating suggestions. FridgeSight can also be used for cabinets and other kitchen inventories. In the future, a large FridgeSight community would be able to push the platform with crowd-trained neural networks, easily surpassing standalone IoT kitchenware. There is a lot of potential in FridgeSight, and we hope to use PennApps as a way forward.
## Inspiration

The idea for Refrigerator Ramsay came from a common problem we all face: food waste. So many times, we forget about ingredients in the fridge, or we don't know how to use them, and they go bad. We wanted to create something that could help us not only use those ingredients but also inspire us to try new recipes and reduce food waste in the process.

## What it does

Refrigerator Ramsay is an AI-powered kitchen assistant that helps users make the most of their groceries. You can take a picture of your fridge or pantry, and the AI will identify the ingredients. Then, it suggests recipes based on what you have. It also has an interactive voice feature, so you can ask for cooking tips or get help with recipes in real-time.

## How we built it

We used **Google Gemini AI** for image recognition to identify ingredients from photos. and then, to come up with creative meal ideas. We also added **Hume EVI**, which is an emotionally intelligent voice AI, to make the experience more interactive. This lets users chat with the AI to get tips, learn new cooking techniques, or personalize recipes.

## Challenges we ran into

One challenge we faced was working with **Hume EVI** for the first time. It was really fun to explore, but there was definitely a learning curve. We had to figure out how to make the voice assistant not just smart but also engaging, making sure it could deliver cooking tips naturally and understand what users were asking. It took some trial and error to get the right balance between information and interaction.

Another challenge was training the **Google Gemini AI** to be as accurate as possible when recognizing ingredients. It wasn't easy to ensure the AI could reliably detect different foods, especially when items were stacked or grouped together in photos. We had to spend extra time fine-tuning the model so that it could handle a variety of lighting conditions and product packaging.

## Accomplishments that we're proud of

We're proud of creating a solution that not only helps reduce food waste but also makes cooking more fun. The AI's ability to suggest creative recipes from random ingredients is really exciting. We're also proud of how user-friendly (and funny) the voice assistant turned out to be, making it feel like you're cooking with a friend.

## What we learned

We learned a lot about how AI can be used to solve everyday problems, like reducing food waste. Working with **Hume EVI** taught us about building conversational AI, which was new for us. It was really fun, but there was definitely a challenge in figuring out how to make the voice interaction feel natural and helpful at the same time.

We also learned about the importance of **training AI models**, especially with the **Gemini AI**. Getting the image recognition to accurately identify different ingredients required us to experiment a lot with the data and train the model to work in a variety of environments. This taught us that accuracy is key when it comes to creating a seamless user experience.

## What's next for Refrigerator Ramsay

Next, we want to improve the AI’s ability to recognize even more ingredients and maybe even offer nutritional advice. We’re also thinking about adding features where the AI can help you plan meals for the whole week based on what you have. Eventually, we’d love to partner with grocery stores to suggest recipes based on store deals, helping users save money too!