## Inspiration

About a week ago, when I was preparing to move out from my old dorm to a new one, I felt ashamed seeing eight large bags of "trash." Most of these items were impulsive purchases that I used once or twice and then discarded. Since the individual items were not of high value and I was sure I wouldn't use them in the future, I decided to throw them away. I noticed that this is a common issue among my friends as well—everyone has to painfully discard many items when moving. This inspired me to create Hand to Hand, a second-hand goods exchange platform that allows people to meet each other's needs, thereby reducing waste and avoiding unnecessary new purchases.

You might wonder, how is this different from second-hand websites like eBay? The biggest difference lies in its eco-friendly concept. On Hand to Hand, all items are acquired through exchanges instead of money, which significantly reduces unnecessary waste. Additionally, in the main use case of Hand to Hand—campuses—platforms like eBay have several well-known issues:

1. Students find it difficult to spend a lot of time and effort uploading item information to second-hand websites.
2. High shipping costs make it impractical to trade items worth less than $20 because the shipping could be more expensive than the item itself.
3. Ensuring product quality is difficult.

To address these issues, Hand to Hand offers the following features:

### For “Sellers”

All you need to do is lay all your stuff on the ground and take a picture. Hand to Hand uses image segmentation and visual understanding models to automatically crop the item images and assess the condition of the second-hand goods. Using Selenium, Hand to Hand scrapes the original prices and images of the items from Amazon and generates a brief description for your items. Remember, all you need to do is take a picture!

### During the Exchange

Traditional bartering usually occurs between two people. However, considering the supply and demand in a campus might not always match, I designed an algorithm to facilitate possible multi-person (more than two) exchanges. Simply put, I store all items and users as nodes in a graph and use an optimized Depth-First Search (DFS) to find a complete loop. Since loops formed by multi-person exchanges might be unstable, I trained a machine learning model to determine item similarity, so when one edge of the loop is removed, the most similar item can automatically replace it. This recommendation system greatly enhances the chances of forming multi-person exchange chains.

### For “Buyers”

Since users live on the same campus, meeting offline becomes the best way to trade. I integrated Google Calendar to automatically find common free times between users for transactions. To further save time, I designed a feature called “trade along your way.” In simple terms, Hand to Hand automatically invokes the Google Map API and, combined with the schedule in Google Calendar, helps you arrange trades during times when users’ routes intersect. This maximizes time efficiency and minimizes waste.

## How We Built It

I will explain my tech stack based on the functionality of the project.

### Function 1: Process Item Information with One Click

First, I deployed the YOLO V8 model using OpenVino for instance segmentation, which significantly boosts the speed of visual models on a CPU. After obtaining the segmentation masks, I used OpenCV and Selenium for web scraping. OpenCV detects the search icon in the browser and simulates mouse clicks, while Selenium retrieves specific item information (including name, price, image, category, description, etc.) from Amazon. Then, using GPT-4-O’s visual capabilities, we compare images of new and second-hand items to estimate their condition (Excellent, Like New, Good, Acceptable, Not Very Well) and appropriate price. The processed information is stored in JSON format and sent to MongoDB. For all images, we used Amazon S3 Bucket to handle image and URL conversion.

### Function 2: Find Possible Exchanges Between More Than Two People

Here, we store all items and users as nodes, connected by directed edges (if the direction is from user to item, it means User\_X owns Item\_X; if from item to user, it means User\_X wants Item\_X). By doing this, we transform the problem into a mathematical one: we need to find a closed loop where users and items are mutually connected. I used a Depth-First Search (DFS) algorithm to find the shortest path loop based on the existing edges, thus forming a feasible trade chain.

### Function 3: Smart Replacement

For situations where a loop cannot be formed, similar items can be substituted. For this, I trained an autoencoder model using Pytorch GPU in Intel IDC Jupyter Notebook to determine item similarity. The dataset used is the Kaggle Amazon Sales Dataset, which contains detailed information on 1,466 transactions from Amazon in 2023.

#### Model Training Steps:

1. **Data Preprocessing**: Standardizing numerical features, One-Hot encoding categorical features, and TF-IDF vectorizing text features.

	* **Numerical Features (Price)**: Standardizing item prices to enable comparison across different price ranges.
	* **Categorical Features (Category)**: One-Hot encoding to convert item categories into numerical features for model training.
	* **Text Features (Name and Description)**: TF-IDF vectorization to extract key information from the text.
2. **Autoencoder Training**: Comprising an encoder and a decoder.

	* **Encoder**: Compresses high-dimensional input features into low-dimensional representations (embeddings), extracting main information and reducing noise and redundancy.
	* **Decoder**: Restores low-dimensional representations to high-dimensional input, ensuring key information from the original data is retained.
3. **Similarity Calculation**: Using the encoder to extract low-dimensional embeddings, simplifying item features while retaining key information. Cosine similarity is used to calculate item similarity, normalized to a 0-1 range.
4. **Recommendation Generation**: Based on the similarity matrix, generating a recommendation list for each item by finding those with the highest similarity.

After training the model, with Intel IDC’s powerful computing capabilities, I expanded the dataset to include 100,000 shopping records and used Optuna for automatic fine-tuning of model parameters for 100 rounds, resulting in a highly effective similarity and item recommendation model. Additionally, to better understand text information, I experimented with fine-tuning the Llama3 8B model using Intel IDC, but due to time constraints, the fine-tuning is still in progress.

### Function 4: Smart Meetup with Automatic Schedule Setup

We simulated two users with different Google Calendar IDs. First, we searched for common free slots shorter than one hour (to avoid disrupting relaxation) in both schedules. Then, by invoking the Google Map API, we obtained the routes of the two users during these time slots. We segmented the routes and looked for locations where the latitude and longitude differences are less than 0.0001 (approximately 14 meters) at the same time. Finally, we added the location and time slots to both users’ Google Calendars.

### Other Features

For the front end, I used templates from Figma and Anima tools to automatically convert the designs into HTML and CSS, saving a lot of time.

## Challenges we ran into

Although the functional design was established early, we faced numerous technical challenges during implementation. For example, since YOLO V8 is a category-based model, its initial performance in instance segmentation was suboptimal, causing me to waste a significant amount of time. The connection issues with MongoDB were also quite frustrating. When using the OpenAI API, we frequently encountered rate limit issues with base64 image encoding. I resolved this by switching from base64 to using Amazon S3 to convert images to URLs.

Another challenge was managing the task progress in an orderly manner. Since I completed the entire project alone, going thirty consecutive hours without sleep often caused my brain to crash and cease functioning. To address this, I alternated between coding and documentation. After implementing each feature, I immediately wrote the documentation and created the PPT, which effectively alleviated my drowsiness at 3 AM.

Integrating all functionalities also proved difficult. The code directly converted from Figma for the front-end design had serious layout issues in actual use. Additionally, the CORS (Cross-Origin Resource Sharing) settings for the Flask server were unstable, often causing page content to fail to load, which was a constant source of frustration.

## Accomplishment that we're proud of

Firstly, my execution capability is something I am most proud of. I am pleased that I was able to implement almost all the initially planned features. Previously, I had only done separate front-end and back-end development and had not truly integrated them. This full-stack development experience has made me very happy and proud.

Most importantly, the features I implemented perfectly addressed the problems faced by eBay in the campus context. Brief Recap:

1. As students, spending a lot of time and effort uploading item information to second-hand websites is difficult; (✅ Solved by one-click upload of all item information)
2. High shipping costs make items under $20 not worth trading (since shipping could cost more than the item itself); (✅ Solved by face-to-face transactions, eliminating shipping costs)
3. Ensuring the quality of items is challenging; (✅ Solved by face-to-face transactions, allowing buyers to inspect item quality in person)

## What we learned

I learned to more proficiently call vision models and use the OpenAI API to process images. I tried integrating Google Calendar and Google Maps for the first time. I applied search algorithms (DFS) learned in class to real-world scenarios for the first time. In terms of model training, I discovered a very user-friendly and powerful tool—Intel IDC (better than Colab).

## Whats's nest for Hand to Hand

Next, I will seek partners and engineers for collaboration. Additionally, I will implement a more mature backend framework for Hand to Hand and design more aesthetically pleasing front-end interfaces. I plan to promote this project at Stanford University and further optimize it based on market feedback.
## Inspiration

We inherently know importance of recycling, but we get lazy and sometimes forget. People often throw their waste in the wrong recycling bin (e.g. bottles for compost or food waste for landfill) which negates the entire purpose of recycling. Therefore, we wanted to create a project to solve this issue.

## What it does

Introducing **sorta**, a project that combines computer vision and hardware components to make recycling more convenient and effective. Our system utilizes image recognition via a mounted camera to recognize waste and identify which kind of recycling bin it best belongs in. Our system will communicate with the recycling bins to ensure that only the identified bin will open for the user. Therefore, people have no choice but to correctly dispose their waste.

## How we built it

For capturing images and opening specific bins properly, our system utilizes a Raspberry Pi 3 Microcontroller with an Arducam, an ultrasonic range sensor, and three servo motors. The Arducam captures images and the microcontroller sends the images to a storage account in Microsoft Azure. The Arducam only takes pictures when a user is within a certain range of the waste disposal system so that we do not have excessive, superfluous data; our system detects range with the ultrasonic range sensor. Finally, our system manages the opening and closing of the recycling bins through three standard servo motors.

In terms of software, after the Raspberry Pi takes the picture of the piece of waste it is uploaded into a Microsoft blob storage so that the list of pictures taken are easily iterated through and turned into the proper format (URL). The system then utilises the Microsoft cognitive services, computer vision API to detect the type of waste it is (eg. plastic bottle, plastic bag, can). A category hierarchy is then built on top of it to sort the detected objects into more general categories. This builds up until each object is sorted into the 3 categories of a typical recycling bin in Stanford: Compostable (fruits, vegetables), Recycle (glass, plastic, metal, paper), landfill. For extra accuracy the system then also runs the pictures through the Microsoft custom vision API which we manually trained with a mixture of categorised waste pictures from ImageNet and manually taken pictures of food wrapper waste lying around in the Hackathon. Our original plan also involved training this API to detect brand names and company logos.

The data coming in from both APIs are then stored up in a Microsoft cosmos db with Mongodb. If company brand logos, type of waste and location (trivial data because the physical system is stationary and it can be hardcoded) are successfully obtained. This has potential to be used in data analytics applications (eg. displaying in a heatmap showing which company is producing what type of waste at high levels in different locations)

## Challenges we ran into

Waste sorting isn’t completely a visual problem. People sort waste according to what the wastes are, but what the wastes are depends not only on our vision of those objects. A piece of metal is categorized into recycled because it feels like a metal in our hands. Therefore, we are working on problems that contain more factors that we can tackle currently. Thus, we have to optimize under inevitably biased sub-optimal situations.

No one in our team had a strong background regarding hardware. Therefore, implementing all the Raspberry Pi features was a challenge. Although we managed to implement the camera capture with aspects of distance detection and image transfer to the Microsoft Azure data storage space, we were not able to implement the opening and closing of recycling bins through the servo motors.

Data selection was challenging, as we didn’t have access to database with good quality control. In the end, we have to manually select general and valid photos of glass, plastic, metal, paper, etc. so that our model developed with the Microsoft custom vision API could be better trained.

Along the way we came across many small bugs that really tripped up the teams progress. Many of those bugs, involved calling APIs because the errors returned were generally unclear on which parameter was missing in the call. So a large amount of time it came down to trial and error on guessing the conventions of these API calls.

## Accomplishments that we're proud of

For all the members in our team, this event was the first hackathon we ever attended. Therefore, we’re very proud of submitting an idea and project we felt could an beneficial impact on society.

Although the learning curve at times was steep, we are proud that we stuck with it throughout the whole event. We definitely learned a lot and were exposed to a ton of amazing resources we can use for future projects.
We also had a blast discovering the collaborative, innovative, and inspiring nature of hackathons. We met a lot of people this weekend that we otherwise would never have the chance to meet. To conclude, we are really looking forward to attending our next hackathon!

## What we learned

Hardware, Database, Raspberry Pi, Microsoft Cognitive Services (including computer vision API and custom vision API)

## What's next for sorta

We could integrate more sensors into our system so that it’s easier to distinguish certain material like metal.

We could also integrate motors into our system that we were not able to finish this time, so that the user only needs to place the trash on a plate, and our system can categorize it and place it into the right trash bin.

If company logo brand and location data is also collected at a larger scale with sharpener waste classification, we believe that the data collected can be very useful. It can be used to detect consumer patterns in different locations as well as waste data for environmental purposes.
## 💡 Inspiration 💯

Have you ever faced a trashcan with a seemingly endless number of bins, each one marked with a different type of recycling? Have you ever held some trash in your hand, desperately wondering if it can be recycled? Have you ever been forced to sort your trash in your house, the different bins taking up space and being an eyesore? Inspired by this dilemma, we wanted to create a product that took all of the tedious decision-making out of your hands. Wouldn't it be nice to be able to mindlessly throw your trash in one place, and let AI handle the sorting for you?

## ♻️ What it does 🌱

IntelliBin is an AI trashcan that handles your trash sorting for you! Simply place your trash onto our machine, and watch it be sorted automatically by IntelliBin's servo arm! Furthermore, you can track your stats and learn more about recycling on our React.js website.

## 🛠️ How we built it 💬

Arduino/C++ Portion: We used C++ code on the Arduino to control a servo motor and an LED based on serial input commands. Importing the servo library allows us to access functions that control the motor and turn on the LED colours. We also used the Serial library in Python to take input from the main program and send it to the Arduino. The Arduino then sent binary data to the servo motor, correctly categorizing garbage items.

Website Portion: We used React.js to build the front end of the website, including a profile section with user stats, a leaderboard, a shop to customize the user's avatar, and an information section. MongoDB was used to build the user registration and login process, storing usernames, emails, and passwords.

Google Vision API: In tandem with computer vision, we were able to take the camera input and feed it through the Vision API to interpret what was in front of us. Using this output, we could tell the servo motor which direction to turn based on if it was recyclable or not, helping us sort which bin the object would be pushed into.

## 🚧 Challenges we ran into ⛔

* Connecting the Arduino to the arms
* Determining the optimal way to manipulate the Servo arm, as it could not rotate 360 degrees
* Using global variables on our website
* Configuring MongoDB to store user data
* Figuring out how and when to detect the type of trash on the screen

## 🎉 Accomplishments that we're proud of 🏆

In a short span of 24 hours, we are proud to:

* Successfully engineer and program a servo arm to sort trash into two separate bins
* Connect and program LED lights that change colors varying on recyclable or non-recyclable trash
* Utilize Google Cloud Vision API to identify and detect different types of trash and decide if it is recyclable or not
* Develop an intuitive website with React.js that includes login, user profile, and informative capabilities
* Drink a total of 9 cans of Monsters combined (the cans were recycled)

## 🧠 What we learned 🤓

* How to program in C++
* How to control servo arms at certain degrees with an Arduino
* How to parse and understand Google Cloud Vision API outputs
* How to connect a MongoDB database to create user authentification
* How to use global state variables in Node.js and React.js
* What types of items are recyclable

## 🌳 Importance of Recycling 🍀

* Conserves natural resources by reusing materials
* Requires less energy compared to using virgin materials, decreasing greenhouse gas emissions
* Reduces the amount of waste sent to landfills,
* Decreasesdisruption to ecosystems and habitats

## 👍How Intellibin helps 👌

**Efficient Sorting:** Intellibin utilizes AI technology to efficiently sort recyclables from non-recyclables. This ensures that the right materials go to the appropriate recycling streams.

**Increased Recycling Rates:** With Intellibin making recycling more user-friendly and efficient, it has the potential to increase recycling rates.

**User Convenience:** By automating the sorting process, Intellibin eliminates the need for users to spend time sorting their waste manually. This convenience encourages more people to participate in recycling efforts.

**In summary:** Recycling is crucial for environmental sustainability, and Intellibin contributes by making the recycling process more accessible, convenient, and effective through AI-powered sorting technology.

## 🔮 What's next for Intellibin⏭️

The next steps for Intellibin include refining the current functionalities of our hack, along with exploring new features. First, we wish to expand the trash detection database, improving capabilities to accurately identify various items being tossed out. Next, we want to add more features such as detecting and warning the user of "unrecyclable" objects. For instance, Intellibin could notice whether the cap is still on a recyclable bottle and remind the user to remove the cap. In addition, the sensors could notice when there is still liquid or food in a recyclable item, and send a warning. Lastly, we would like to deploy our website so more users can use Intellibin and track their recycling statistics!