Leave a comment

Meet JARVIS: Facebook CEO's smart home A.I. can do things you won't believe


At the start of 2016, Facebook CEO Mark Zuckerberg decided to take on his very own personal challenge - build a "simple" artificial intelligence (AI) assistant that will "help him run his home and help him with his work."

He decided to name this AI helper Jarvis, after Tony Stark's computer butler in the Iron Man films.

Fun fact: In Marvel lore, JARVIS stands for Just A Rather Very Intelligent System.

"Simple" may be a relative term since some of his plans for "Jarvis" are voice recognition for controlling everything in his home, facial recognition at the front door for security purposes, and Virtual Reality data visualization for his work-related activities. A tall order, indeed.

Now, almost a year has passed and Zuckerberg has finally posted a Jarvis project update through a Facebook note titled "Building Jarvis."

"My goal was to learn about the state of artificial intelligence -- where we're further along than people realize and where we're still a long ways off," he wrote. "These challenges always lead me to learn more than I expected, and this one also gave me a better sense of all the internal technology Facebook engineers get to use, as well as a thorough overview of home automation."

Here's a video and a rundown of what Zuckerberg's Jarvis does around his home:

Control smart appliances

Zuckerberg revealed that he uses a Crestron home automation system for his lights, thermostat and doors, a Samsung TV and a Nest cam for monitoring his daughter Max.

In the video, he showed how Jarvis helps him get ready in the morning. Upon waking up, he can set up Jarvis to prep his living space by controlling his home's window curtains and setting its temperature. It also makes his breakfast toast and even instantly delivers him a fresh (gray, of course) T-shirt via a launcher cannon.

Jarvis diagram

Zuckerberg admits that his Jarvis challenge was easier than he expected but the most complicated aspect was connecting and communicating with all the different systems in his house.

"Before I could build any AI, I first needed to write code to connect these systems, which all speak different languages and protocols," he explained in his post.

"...and of course my work is connected to Facebook's systems. I had to reverse engineer APIs for some of these to even get to the point where I could issue a command from my computer to turn the lights on or get a song to play."

There's also his issue with controlling traditional non-smart appliances that are not connected to the internet yet. For his "smart" breakfast toaster, he ended up using an old 1950s toaster that he rigged with an internet connected power switch. He also mentions it required hardware modifications for his T-shirt launcher and dog food dispenser to work.

Virtual Secretary/Babysitter

In the video, Jarvis also acted as his virtual secretary of sorts. He gave him a rundown of his calendar meetings for the day and even prepped a room for a video conference call.

The AI butler also acts like a virtual babysitter by monitoring his daughter Max's room via a Nest cam, giving him updates about her and by "entertaining" her, possibly with music.

With his daughter's presence, he stresses the importance of context understanding for any AI.


Fun fact: Jarvis is currently voiced by Morgan Freeman.

Facial Recognition at the front door

For security, Zuckerberg rigged his front door with a few cameras for facial recognition. This system automatically recognizes familiar faces (like his parents'), let them in and informs Zuckerberg about the visitors.

He states that he is using Facebook's own facial recognition technology to scan his front door but it has its own share of challenges.

Jarvis front door

"Face recognition is a particularly difficult version of object recognition because most people look relatively similar compared to telling apart two random objects -- for example, a sandwich and a house. But Facebook has gotten very good at face recognition for identifying when your friends are in your photos. That expertise is also useful when your friends are at your door and your AI needs to determine whether to let them in."

Zuckerberg's front door camera system runs on a server that continuously watches it and runs face detection first, then facial recognition to identify who the visitor is. Once it identifies a familiar face, it automatically lets him/her in if he's at home and informs him about the visitor.

Play music according to preferences

For music, he uses a Sonos connected speaker system and Jarvis helps pick the music according to preference. Since the AI was trained to recognize both his and his wife Priscilla's voices, the system can play specific music tailored to the requester's tastes.

Zuckerberg said that Jarvis looks at past listening patterns to decide what appropriate music to play at any given time. If it gets the mood wrong, it can both learn the classification (read: keyword) and adjust accordingly for that moment and catalogs it for future reference.

Voice requests for music are also filled with subtle variations so Jarvis' context understanding again comes into play.

"Music is a more interesting and complex domain for natural language because there are too many artists, songs and albums for a keyword system to handle. The range of things you can ask it is also much greater. Lights can only be turned up or down, but when you say 'play X,' even subtle variations can mean many different things."

Control Jarvis via text message

Although Zuckerberg thinks that voice recognition will play an important role in controlling AIs, he thinks that text messaging is more important that people realize. The most useful advantage that voice has over text is speed, he said, but he texts his commands more than expected.

Jarvis text


His issue with voice commands and AI voice responses is their disruptiveness. He admits that he would much rather send his commands via text message and likewise receive Jarvis' responses via another text message. This gives him more control of when he wants to view the response.

The future of Jarvis

Now that 2016 is almost over and Zuckerberg's Jarvis Challenge is ending, does this mean he will stop and abandon the project altogether?

Not at all. He said that he'll continue improving Jarvis since he uses it on a daily basis. His next goals include building an Android app, more terminals around his home and connecting more appliances.

His long-term goal is to explore how to make Jarvis learn new skills by itself rather than having a human program it to perform specific tasks.

Over time, he said, he will try and find ways to make his AI available to the world. He even admits that he has considered open-sourcing his code but it is too tightly integrated with his own home at the moment. If he can make Jarvis home automation functionality more general, he may release it and it can be a great foundation for a future product.

Read his full Facebook "Building Jarvis" post here.

What do you think? Does Mark Zuckerberg's Jarvis blow your mind? Drop us a comment below!

More articles to read:

Zuckerberg demos the future of Facebook and it looks amazing

Amazon Echo vs. Google Home - Which virtual assistant is better?

Facebook Messenger finally rolls out the feature you've been waiting for

Next Story
View Comments ()
This eBay listing is taking the internet by storm - What's so special about it?
Previous Happening Now


Facebook Messenger finally rolls out the feature you've been waiting for
Next Happening Now

Facebook Messenger finally rolls out the feature you've been waiting for