## Inspiration

We are all software/game devs excited by new and unexplored game experiences. We originally came to PennApps thinking of building an Amazon shopping experience in VR, but eventaully pivoted to Project Em - a concept we all found mroe engaging. Our swtich was motivated by the same force that is driving us to create and improve Project Em - the desire to venture into unexplored territory, and combine technologies not often used together.

## What it does

Project Em is a puzzle exploration game driven by Amazon's Alexa API - players control their character with the canonical keyboard and mouse controls, but cannot accomplish anything relevant in the game without talking to a mysterious, unknown benefactor who calls out at the beginning of the game.

## How we built it

We used a combination of C++, Pyhon, and lots of shell scripting to create our project. The client-side game code runs on Unreal Engine 4, and is a combination of C++ classes and Blueprint (Epic's visual programming language) scripts. Those scripts and classes communicate an intermediary server running Python/Flask, which in turn communicates with the Alexa API. There were many challenges in communicating RESTfully out of a game engine (see below for more here), so the two-legged approach lent itself well to focusing on game logic as much as possible. Sacha and Akshay worked mostly on the Python, TCP socket, and REST communication platform, while Max and Trung worked mainly on the game, assets, and scripts.

The biggest challenge we faced was networking. Unreal Engine doesn't naively support running a webserver inside a game, so we had to think outside of the box when it came to networked communication.

The first major hurdle was to find a way to communicate from Alexa to Unreal - we needed to be able to communicate back the natural language parsing abilities of the Amazon API to the game. So, we created a complex system of runnable threads and sockets inside of UE4 to pipe in data (see challenges section for more info on the difficulties here). Next, we created a corresponding client socket creation mechanism on the intermediary Python server to connect into the game engine. Finally, we created a basic registration system where game clients can register their publicly exposed IPs and Ports to Python.

The second step was to communicate between Alexa and Python. We utilitzed [Flask-Ask](https://flask-ask.readthedocs.io/en/latest/) to abstract away most of the communication difficulties,. Next, we used [VaRest](https://github.com/ufna/VaRest), a plugin for handing JSON inside of unreal, to communicate from the game directly to Alexa.

The third and final step was to create a compelling and visually telling narrative for the player to follow. Though we can't describe too much of that in text, we'd love you to give the game a try :)

## Challenges we ran into

The challenges we ran into divided roughly into three sections:

* **Threading**: This was an obvious problem from the start. Game engines rely on a single main "UI" thread to be unblocked and free to process for the entirety of the game's life-cycle. Running a socket that blocks for input is a concept in direct conflict with that idiom. So, we dove into the FSocket documentation in UE4 (which, according to Trung, hasn't been touched since Unreal Tournament 2...) - needless to say it was difficult. The end solution was a combination of both FSocket and FRunnable that could block and certain steps in the socket process without interrupting the game's main thread. Lots of stuff like this happened:

```
while (StopTaskCounter.GetValue() == 0)
    {
        socket->HasPendingConnection(foo);
        while (!foo && StopTaskCounter.GetValue() == 0)
        {
            Sleep(1);
            socket->HasPendingConnection(foo);
        }
        // at this point there is a client waiting
        clientSocket = socket->Accept(TEXT("Connected to client.:"));
        if (clientSocket == NULL) continue;
        while (StopTaskCounter.GetValue() == 0)
        {
            Sleep(1);
            if (!clientSocket->HasPendingData(pendingDataSize)) continue;
            buf.Init(0, pendingDataSize);
            clientSocket->Recv(buf.GetData(), buf.Num(), bytesRead);
            if (bytesRead < 1) {
                UE_LOG(LogTemp, Error, TEXT("Socket did not receive enough data: %d"), bytesRead);
                return 1;
            }
            int32 command = (buf[0] - '0');
            // call custom event with number here
            alexaEvent->Broadcast(command);
            clientSocket->Close();
            break; // go back to wait state
        }
    }

```

Notice a few things here: we are constantly checking for a stop call from the main thread so we can terminate safely, we are sleeping to not block on Accept and Recv, and we are calling a custom event broadcast so that the actual game logic can run on the main thread when it needs to.

The second point of contention in threading was the Python server. Flask doesn't natively support any kind of global-to-request variables. So, the canonical approach of opening a socket once and sending info through it over time would not work, regardless of how hard we tried. The solution, as you can see from the above C++ snippet, was to repeatedly open and close a socket to the game on each Alexa call. This ended up causing a TON of problems in debugging (see below for difficulties there) and lost us a bit of time.

* **Network Protocols**: Of all things to deal with in terms of networks, we spent he largest amount of time solving the problems for which we had the least control. Two bad things happened: heroku rate limited us pretty early on with the most heavily used URLs (i.e. the Alexa responders). This prompted two possible solutions: migrate to DigitalOcean, or constantly remake Heroku dynos. We did both :). DigitalOcean proved to be more difficult than normal because the Alexa API only works with HTTPS addresses, and we didn't want to go through the hassle of using LetsEncrypt with Flask/Gunicorn/Nginx. Yikes. Switching heroku dynos it was.

The other problem we had was with timeouts. Depending on how we scheduled socket commands relative to REST requests, we would occasionally time out on Alexa's end. This was easier to solve than the rate limiting.

* **Level Design**: Our levels were carefully crafted to cater to the dual player relationship. Each room and lighting balance was tailored so that the player wouldn't feel totally lost, but at the same time, would need to rely heavily on Em for guidance and path planning.

## Accomplishments that we're proud of

The single largest thing we've come together in solving has been the integration of standard web protocols into a game engine. Apart from matchmaking and data transmission between players (which are both handled internally by the engine), most HTTP based communication is undocumented or simply not implemented in engines. We are very proud of the solution we've come up with to accomplish true bidirectional communication, and can't wait to see it implemented in other projects. We see a lot of potential in other AAA games to use voice control as not only an additional input method for players, but a way to catalyze gameplay with a personal connection.

On a more technical note, we are all so happy that...

THE DAMN SOCKETS ACTUALLY WORK YO

## Future Plans

We'd hope to incorporate the toolchain we've created for Project Em as a public GItHub repo and Unreal plugin for other game devs to use. We can't wait to see what other creative minds will come up with!

### Thanks

Much <3 from all of us, Sacha (CIS '17), Akshay (CGGT '17), Trung (CGGT '17), and Max (ROBO '16). Find us on github and say hello anytime.
## Inspiration

The inspiration for our project stems from the increasing trend of online shopping and the declining foot traffic in physical stores. Our goal was to provide a unique and engaging experience for customers, encouraging them to visit physical stores and rediscover the joy of in-person shopping. We wanted to create an interactive and entertaining shopping experience that would entice customers to visit stores more frequently and foster a deeper connection between them and the store's brand.

## What it does

Our project is an AR scavenger hunt experience that gamifies the shopping experience. The scavenger hunt encourages customers to explore the store and discover new products they may have otherwise overlooked. As customers find specific products, they can earn points which can be redeemed for exclusive deals and discounts on future purchases. This innovative marketing scheme not only provides customers with an entertaining experience but also incentivizes them to visit stores more frequently and purchase products they may have otherwise overlooked.

## How we built it

To create the AR component of our project, we used Vuforia and Unity, two widely used platforms for building AR applications. The Vuforia platform allowed us to create and track image targets, while Unity was used to design the 3D models for the AR experience. We then integrated the AR component into an Android application by importing it as a Gradle project. Our team utilized agile development methodologies to ensure efficient collaboration and problem-solving throughout the development process.

## Challenges we ran into

One of the challenges we faced was integrating multiple APIs and ensuring that they worked together seamlessly. Another challenge was importing the AR component and creating the desired functionality within our project. We also faced issues with debugging and resolving technical errors that arose during the development process.

## Accomplishments that we're proud of

Despite the challenges we faced, we were able to achieve successful teamwork and collaboration. Despite forming the team later than other groups, we were able to effectively communicate and work together to bring our project to fruition. We are proud of the end result, which was a polished and functional AR scavenger hunt experience that met our objectives.

## What we learned

We learned how difficult it is to truly ship out software, and we are grateful to have joined the hackathon. We gained a deeper understanding of the importance of project planning, effective communication, and collaboration among team members. We also learned that the development process can be challenging and unpredictable, and that it requires perseverance and problem-solving skills. Additionally, participating in the hackathon taught us valuable technical skills such as integrating APIs, creating AR functionality, and importing projects onto an Android application.

## What's next for Winnur

Looking forward, we plan to incorporate Computer Vision technology into our project to prevent potential damage to our product's packaging. We also aim to expand the reach of our AR scavenger hunt experience by partnering with more retailers and enhancing the user interface and experience. We are excited about the potential for future development and growth of Winnur.
## Inspiration

Our focus is to connect people around the globe by building a platform that hosts conferences in virtual reality. Imagine the convenience of meeting anyone, from anywhere, and at any time -- all from the comfort of your home or office. Large, popular, and influential events like Google I/O, Grace Hopper, etc. often get sold out before everyone who wants to can purchase a ticket. Our infinitely scalable platform eliminates this problem. We disrupt the constraints of the physical world (space, time), and open up the opportunity for you to connect with your stakeholders and audience in an enhanced reality. We see virtual reality as an elegant solution to not only language barriers, but also the inefficiencies, expenses, and discomforts of physically traveling to interact with others.

The potential of our project is as scalable as our platform. Virtual reality is not only a solution, it is the future. What’s most exciting is that our platform opens up a new marketplace for businesses around the world to connect, network, and make an impact.

## What it does

Full conference and live communication in virtual reality from anywhere in the world! In our prototype, we've created a lobby, a webinar room, and a conference room where participants can talk and engage in real time.

# Significant Problems Addressed:

Discriminatory barriers in terms of social, geographic, and financial conditions.
High costs with regard to travel time and conference cost
Limited participants

## What we learned

This was a very ambitious project for our team -- especially considering that none of us have worked with VR before. Each and every one of us had to overcome steep learning curves in order to work with multiple different technologies with which we were not familiar.

Main things we learned:
-Working with Unity
-Multiplayer communication and networking
-3D design for virtual reality

# Problems we ran into:

We ran into a lot of incompatibility problems, as our project involves several different pieces of technology. So we had to think outside the box in order to make our devices compatible.
-First, our Alienware laptops had an embedded graphic card that is incompatible with the current version of the Oculus runtime. With that, we had to hack our way to downgrade the current run time on our computer, edit some of the registry logs, and disable error reporting in order for the Oculus to be compatible.
-Finding an ultimate back-end platform for our multiplayer platform and enabling voice so that everyone inside our platform can talk to each other was also a challenge.
-When we tried to set up the speech to text, text to speech API, our backend platform did not accept sound so we had to convert our audio clip’s 0.5 second frame -> convert to float -> convert to byte -> send it to the rest of the players of the network -> converted back to float -> converted back to sound so there was a little delay when it come to receiving the sound.
-Setting up the Oculus and getting it to display properly along with the position tracker was extremely challenging.

## What's next for ConferenceVR

Web Interface for corporations, schools, etc. to customize their conference area.
Live speech translation to allow for language-independent global communication.

We hope to add in more features such as Facebook authorization, having profile pictures that are parsed from Facebook, population/attendance for data visualization, and much more. There is no end to the number of new features that can be integrated with our platform.