# My Eyes

Helping dyslectics read by using vision and text-to-speech.

## Mission Statement

Dyslexia is a reading disability that affects 5% of the population. Individuals with dyslexia have difficulty decoding written word with speed and accuracy. To help those afflicted with dyslexia, many schools in BC provide additional reading classes. But even with reading strategies, it can take quite a bit of concentration and effort to comprehend text.

Listening to an audio book is more convenient than reading a physical one. There are text-to-speech services which can read off digital text on your tablet or computer. However, there aren't any easily accessible services which offer reading off physical text.

Our mission was to provide an easily accessible service that could read off physical text. Our MOBILE app at 104.131.142.126:3000 or eye-speak.org allows you to take a picture of any text and play it back. The site's UI was designed for those with dyslexia in mind. The site fonts and color scheme were purposely chosen to be as easily read as possible.

This site attempts to provide an easy free service for those with reading disabilities.

## Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

### Installing

Create a file called '.env' and a folder called 'uploads' in the root folder

Append API keys from [IBM Watson](https://www.ibm.com/watson/services/text-to-speech/)

Append API keys from Google Cloud Vision

1. [Select or create a Cloud Platform project](https://console.cloud.google.com/project)
2. [Enable billing for your project](https://support.google.com/cloud/answer/6293499#enable-billing)
3. [Enable the Google Cloud Vision API API](https://console.cloud.google.com/flows/enableapi?apiid=vision.googleapis.com)
4. [Set up authentication with a service account so you can access the API from your local workstation](https://cloud.google.com/docs/authentication/getting-started)

.env should look like this when you're done:

```
USERNAME=<watson_username>
PASSWORD=<watson_password>
GOOGLE_APPLICATION_CREDENTIALS=<path_to_json_file>

```

Install dependencies and start the program:

```
npm install
npm start

```

Take a picture of some text and and press play to activate text-to-speech.

## Built With

* [Cloud Vision API](https://cloud.google.com/vision/) - Used to read text from images
* [Watson Text to Speech](https://console.bluemix.net/catalog/services/text-to-speech) - Used to read text from images to speech

## Authors

* **Zachary Anderson** - *Frontend* - [ZachaRuba](https://github.com/ZachaRuba)
* **Håvard Estensen** - *Google Cloud Vision* - [estensen](https://github.com/estensen)
* **Kristian Jensen** - *Backend, IBM Watson* - [HoboKristian](https://github.com/HoboKristian)
* **Charissa Sukontasukkul** - *Design*
* **Josh Vocal** - *Frontend* - [joshvocal](https://github.com/joshvocal)
## Inspiration

Alex K's girlfriend Allie is a writer and loves to read, but has had trouble with reading for the last few years because of an eye tracking disorder. She now tends towards listening to audiobooks when possible, but misses the experience of reading a physical book.

Millions of other people also struggle with reading, whether for medical reasons or because of dyslexia (15-43 million Americans) or not knowing how to read. They face significant limitations in life, both for reading books and things like street signs, but existing phone apps that read text out loud are cumbersome to use, and existing "reading glasses" are thousands of dollars!

Thankfully, modern technology makes developing "reading glasses" much cheaper and easier, thanks to advances in AI for the software side and 3D printing for rapid prototyping. We set out to prove through this hackathon that glasses that open the world of written text to those who have trouble entering it themselves can be cheap and accessible.

## What it does

Our device attaches magnetically to a pair of glasses to allow users to wear it comfortably while reading, whether that's on a couch, at a desk or elsewhere. The software tracks what they are seeing and when written words appear in front of it, chooses the clearest frame and transcribes the text and then reads it out loud.

## How we built it

**Software (Alex K)** -
On the software side, we first needed to get image-to-text (OCR or optical character recognition) and text-to-speech (TTS) working. After trying a couple of libraries for each, we found Google's Cloud Vision API to have the best performance for OCR and their Google Cloud Text-to-Speech to also be the top pick for TTS.

The TTS performance was perfect for our purposes out of the box, but bizarrely, the OCR API seemed to predict characters with an excellent level of accuracy individually, but poor accuracy overall due to seemingly not including any knowledge of the English language in the process. (E.g. errors like "Intreduction" etc.) So the next step was implementing a simple unigram language model to filter down the Google library's predictions to the most likely words.

Stringing everything together was done in Python with a combination of Google API calls and various libraries including OpenCV for camera/image work, pydub for audio and PIL and matplotlib for image manipulation.

**Hardware (Alex G)**: We tore apart an unsuspecting Logitech webcam, and had to do some minor surgery to focus the lens at an arms-length reading distance. We CAD-ed a custom housing for the camera with mounts for magnets to easily attach to the legs of glasses. This was 3D printed on a Form 2 printer, and a set of magnets glued in to the slots, with a corresponding set on some NerdNation glasses.

## Challenges we ran into

The Google Cloud Vision API was very easy to use for individual images, but making synchronous batched calls proved to be challenging!
Finding the best video frame to use for the OCR software was also not easy and writing that code took up a good fraction of the total time.

Perhaps most annoyingly, the Logitech webcam did not focus well at any distance! When we cracked it open we were able to carefully remove bits of glue holding the lens to the seller’s configuration, and dial it to the right distance for holding a book at arm’s length.

We also couldn’t find magnets until the last minute and made a guess on the magnet mount hole sizes and had an *exciting* Dremel session to fit them which resulted in the part cracking and being beautifully epoxied back together.

## Acknowledgements

The Alexes would like to thank our girlfriends, Allie and Min Joo, for their patience and understanding while we went off to be each other's Valentine's at this hackathon.
We all believe in harnessing the power of data to help people make informed decisions. One of the biggest decisions that any American makes is where to live or which home to buy. We sought to use real estate data made available through Nasdaq datasets to make this decision easier and better informed.

## What it does

Given any zip code, our web application shows the desired data and has interactive charts to display housing data for a given zip code and time frame. It also provides links to information on local schools in the community.

## How we built it

We used Python/Flask for the backend, and wrote simple HTML frontend to display the form and graphs. We used web scraping to build the link that directs users to the top schools in the area.

## Challenges we ran into

There were a few challenges. The first was getting the graph to display -- there was an issue with the mpld3 python module, and we had to manually change the library to render the graph correctly. We also ran into challenges in how to collect information about local schools.

## What we learned

* Real estate data is a valuable resource for making crucial decisions about where to live.
* Having the right data at everyone's disposal can help people find affordable areas to live.
* How to display time series graphs in an interactive way.
* How to use large datasets like the Nasdaq datasets and visualize them in an easily understandable way.