## Inspiration

We were motivated to tackle linguistic challenges in the educational sector after juxtaposing our personal experience with current news.

There are currently over 70 million asylum seekers, refugees, or internally-displaced people around the globe, and this statistic highlights the problem of individuals from different linguistic backgrounds being forced to assimilate into a culture and language different than theirs. As one of our teammates was an individual seeking a new home in a new country, we had first hand perspective at how difficult this transition was. In addition, our other team members had volunteered extensively within the educational system in developing communities, both locally and globally, and saw a similar need with individuals being unable to meet the community’s linguistics standards.

We also iterated upon our idea to ensure that we are holistically supporting our communities by making sure we consider the financial implications of taking the time to refine your language skills instead of working.

## What it does

Fluently’s main purpose is to provide equitable education worldwide. By providing a user customized curriculum and linguistic practice, students can further develop their understanding of their language. It can help students focus on areas where they need the most improvement. This can help them make progress at their own pace and feel more confident in their language skills while also practicing comprehension skills. By using artificial intelligence to analyze pronunciation, our site provides feedback that is both personalized and objective.

## How we built it

Developing the web application was no easy feat.

As we were searching for an AI model to help us through our journey we stumbled upon OpenAI, specifically Microsoft Azure’s cognitive systems that utilize OpenAI’s comprehensive abilities in language processing. This API gave us the ability to analyze voice patterns and fluency and transcribe passages that are mentioned in the application. Figuring out the documentation as well as how the AI will be interacting with the user was most important for us to execute properly since the AI would be acting as the tutor/mentor for the students in these cases. We developed a diagram that would break down the passages read to the student phonetically and give them a score of 100 for how well each word was pronounced based on the API’s internal grading system. As it is our first iteration of the web app, we wanted to explore how much information we could extract from the user to see what is most valuable to display to them in the future.

Integrating the API with the web host was a new feat for us as a young team. We were confident in our python abilities to host the AI services and found a library by the name of Flask that would help us write html and javascript code to help support the front end of the application through python. By using Flask, we were able to host our AI services with python while also continuously managing our front end through python scripts.

This gave room for the development of our backend systems which are Convex and Auth0. Auth0 was utilized to give members coming into the application a unique experience by having them sign into a personalized account. The account is then sent into the Convex database to be used as a storage base for their progress in learning and their development of skills over time. All in all, each component of the application from the AI learning models, generating custom passages for the user, to the backend that communicated between the Javascript and Python server host that streamlines the process of storing user data, came with its own challenges but came together seamlessly as we guide the user from our simple login system to the passage generator and speech analyzer to give the audience constructive feedback on their fluency and pronunciation.

## Challenges we ran into

As a majority beginning team, this was our first time working with many of the different technologies, especially with AI APIs. We need to be patient working with key codes and going through an experiment process of trying different mini tests out to then head to the major goal that we were headed towards. One major issue that we faced was the visualization of data to the user. We found it hard to synthesize the analysis that was done by the AI to translate to the user to make sure they are confident in what they need to improve on. To solve this problem we first sought out how much information we could extract from the AI and then in future iterations we would simply display the output of feedback.

Another issue we ran into was the application of convex into the application. The major difficulty came from developing javascript functions that would communicate back to the python server hosting the site. This was resolved thankfully; we are grateful for the Convex mentors at the conference that helped us develop personalized javascript functions that work seamlessly with our Auth0 authentication and the rest of the application to record users that come and go.

## Accomplishments that we're proud of:

One accomplishment that we are proud of was the implementation of Convex and Auth0 with Flask and Python. As python is a rare language to host web servers in and isn't the primary target language for either service, we managed to piece together a way to fit both services into our project by collaboration with the team at Convex to help us out. This gave way to a strong authentication platform for our web application and for helping us start a database to store user data onto.

Another accomplishment was the transition of using a React Native application to using Flask with Python. As none of the group has seen Flask before or worked for it for that matter, we really had to hone in our abilities to learn on the fly and apply what we knew prior about python to make the web app work with this system.

Additionally, we take pride in our work with OpenAI, specifically Azure. We researched our roadblocks in finding a voice recognition AI to implement our natural language processing vision. We are proud of how we were able to display resilience and conviction to our overall mission for education to use new technology to build a better tool.

## What we learned

As beginners at our first hackathon, not only did we learn about the technical side of building a project, we were also able to hone our teamwork skills as we dove headfirst into a project with individuals we had never worked with before.

As a group, we collectively learned about every aspect of coding a project, from refining our terminal skills to working with unique technology like Microsoft Azure Cognitive Services. We also were able to better our skillset with new cutting edge technologies like Convex and OpenAI.

We were able to come out of this experience not only growing as programmers but also as individuals who are confident they can take on the real world challenges of today to build a better tomorrow.

## What's next?

We hope to continue to build out the natural language processing applications to offer the technology in other languages. In addition, we hope to hone to integrate other educational resources, such as videos or quizzes to continue to build other linguistic and reading skill sets. We would also love to explore the cross section with gaming and natural language processing to see if we can make it a more engaging experience for the user. In addition, we hope to expand the ethical considerations by building a donation platform that allows users to donate money to the developing community and pay forward the generosity to ensure that others are able to benefit from refining their linguistic abilities. The money would then go to a prominent community in need that uses our platform to fund further educational resources in their community.

## Bibliography

United Nations High Commissioner for Refugees. “Global Forced Displacement Tops 70 Million.” UNHCR, UNHCR, The UN Refugee Agency, <https://www.unhcr.org/en-us/news/stories/2019/6/5d08b6614/global-forced-displacement-tops-70-million.html>.
## Inspiration

With the constant evolution of the internet and web technologies, web developers are spread thinner than ever. Accessibility is often overlooked. We wanted to make a tool which would provide an accessibility layer on top of webpages -- providing users with accessibility automatically without developers having to spend painstaking hours on such a thing.

## What it does

The project is a Google Chrome Extension which provides an accessibility layer on top of web pages in two important ways

* Automatically generate and add alt tags to images without them
* Automatically make texts contrast with their backgrounds with a contrast ratio of at least 4.5:1

## How we built it

The project is a Google Chrome Extension with a server-less architecture. We utilized AWS's API Gateway to communicate with our AWS Lambda computational unit. The Lambda function is responsible for generating the alt text for the image. This utilizes Azure's Computer Vision API to do so. Although computationally expensive, our software is **heavily optimized** and implements a 2-layer cache system, with the 1st layer as warm Lambda storage and the 2nd layer as AWS DynamoDB. In this way, once a page's alt texts have been generated and cached once, any users of the page will effectively instantly have access to them. Very little data is passed between the client & API Gateway (only a URL and a dictionary of alt texts), meaning that alt texts will (generally) be available before the images actually load. Thus, it also benefits people with limited internet access. Lastly, the automatic text contrasting is done in the frontend by "hacking" the mathematics of RGB coloring.

We also built a static site [hosted by Amazon S3](https://bad-website.s3.amazonaws.com/index.html) for testing purposes.

## Challenges we ran into

Web accessibility is an enormously difficult problem. There is a reason a tool like this doesn't exist already. The standards aren't clear cut from a programmatic perspective, making automated testing difficult. Specifically, automatically *fixing* the issues is a huge undertaking. Thankfully, with the help of our Microsoft Mentor -- Morgan, we discovered [Accessibility Insights](https://accessibilityinsights.io). We used this to test our extension's components to ensure they were working correctly.

Technical Challenges

* utilizing CORS within a Chrome Extension; it requires a background thread, message sharing with that thread, and explicit permissions to make such requests
* identifying the background color of a generic HTML tag; we ended up traversing the DOM upwards until we could safely identify a background color

## Accomplishments that we're proud of

* Integrating many different complex technologies together including 4 AWS Services, Microsoft's Computer Vision Azure API, and a Google Chrome Extension
* Implementing a 2-layered caching system for optimized usage of our extension
* Tackling an extremely important issue and being able to create a usable product in 24 hours

## What we learned

* By using [Accessibility Insights](https://accessibilityinsights.io), we learned many websites don't comply with the Web Content Accessibility Guidelines (WCAG); this was shocking and motivated us to help make a difference
* A lot of amazing tech tools exist nowadays, but it's still up to us to use them and make an impact
* It's quite motivating and fun to hack Assistive Tech

## What's next for Extend a H.A.N.D.

1. We plan to implement more features in order to make websites automatically more compliant with the WCA Standards.
2. We'd like to build extensions for other browsers.
3. We'd like to make a developer tool so that developers can easily and automatically make their sites accessible in the future.
## Inspiration

20% of people worldwide have a learning difference such as dyslexia or ADHD, representing 70 million people in the US alone. All of us had observed and were familiar with the problems such students face. For example, our teammate Samim started an NGO in Germany that started offering workshops at German schools when COVID-19 started. During these workshops he began to realize how difficult it is for every student to keep pace with educational/instructional materials, especially those with learning difference such as dyslexia or ADHD, and even more during the pandemic. As a PhD student in AI,he knew the power of software applications and AI in creating support tools and a personalized learning experience.

## What it does

As of present, Lexy provides a suite of support tools for students with learning impairments or difficulties based on transformations of a given text input to an output. For example, students having trouble processing a certain material have the option to choose whether they want a text to be summarized or to be transformed into flashcards/categorized by heading etc. It is thus entirely a user-specific experience at the moment and they get to modulate how they want the app to support them, and the app will perform the natural language processing based on that to obtain the output. No other software offers/integrates such a comprehensive suite of support tools for struggling students and students with learning impairments, making our solution unique.

## How we built it

1. Design Prototypes in Figma
2. Set up the GPT-3 API (from OpenAI) in Flask to perform a specific task when prompted
3. Extended our NLP configuration to all the support tools we were hoping to cover
4. Implemented text to speech transformation of text for individuals with learning impairments who prefer to listen to the text. Completed backend at this point.
5. Built on top of our backend to add a UI using React, CSS and Bootstrap.
6. Began working on the personalization elements of the project, wherein individual datapoints regarding a student would be used to transform the material accordingly. The datapoints would be provided either by the teacher (who would fill out a survey) or externally through some LMS. Then, based on the datapoints the customization of the output will take place.

## Challenges we ran into

Deciding on what to build took us a while until Saturday morning so the challenge of making the best out of our time became very significant

Frontend and tech stack proved to be much more tedious than expected, taking away crucial time which prevented us from acheiving the project's true potential of a personalized learning experience.

## Accomplishments that we're proud of

Our use of NLP and OpenAI's GPT-3 API in particular :) Coming in, we didnt know much about these technologies but are extremely glad to have gotten hands on experience in working with LLMs.

## What we learned

Ideating before you start. be prepared before you start and align on the idea, tech stack and team roles.

If you fail to plan you plan to fail. As time is limited make sure to have a generally roadmap and time plan. Compensating the lack of time for sleep has it's limit and the team needs a good rest to perform at it's best.

Fight & be nice. You might disagree on priorities or certain directions you need to go together as a team. Conflict is useful to debate for the best solution but never forget the fun!

## What's next for Lexy: Duolingo for Students with Learning Disabiltiies

The personalization elements of the project need to be completed so that Lexy can be taken to its full potential. Here, individual datapoints regarding a student would be used to transform the material accordingly. The datapoints would be provided either by the teacher (who would fill out a survey) or externally through some LMS. Then, based on the datapoints the customization of the output will take place. For example, someone who is identified as having dyslexia but specifically struggles with reading for long periods could be given the ability to use the text to speech function, while someone who specifically struggles with direct or indirect comprehension could be given the text broken down by categories with each one having a heading. Thus, this personalized learning experience would be the main USP of our app going forward.

Additionally, in the future we would like to make Lexy a plugin or Chrome Extension so that it can be used while someone is reading digitally in another application, making the lives of those students much smoother and efficient.