Smartwatches can now track your finger in mid-air using sonar

Smartwatches can now track your finger in mid-air using sonar
FingerIO, a new sonar technology developed by UW computer scientists and electrical engineers, allows you to interact with mobile devices and tiny smartwatch screens by writing or gesturing on any nearby surface -- a tabletop, piece of paper or even in mid-air. Credit: Dennis Wise, University of Washington

As mobile and wearable devices such as smartwatches grow smaller, it gets tougher for people to interact with screens the size of a matchbook.

That could change with a new sonar technology developed by University of Washington computer scientists and electrical engineers that allows you to interact with by writing or gesturing on any nearby surface—a tabletop, a sheet of paper or even in mid-air.

FingerIO tracks fine-grained finger movements by turning a smartphone or smartwatch into an active sonar system using the 's own microphones and speakers.


In a paper to be presented in May at the Association for Computing Machinery's CHI 2016 conference in San Jose, California, the UW team demonstrates that FingerIO can accurately track two-dimensional finger movements to within 8mm, which is sufficiently accurate to interact with today's mobile devices. The work was recognized with an honorable mention award by the conference.

"You can't type very easily onto a smartwatch display, so we wanted to transform a desk or any area around a device into an input surface," said lead author Rajalakshmi Nandakumar, a UW doctoral student in computer science and engineering. "I don't need to instrument my fingers with any other sensors—I just use my finger to write something on a desk or any other surface and the device can track it with high resolution."

Smartwatches can now track your finger in mid-air using sonar
In user testing with off-the-shelf smartphones, the finger location computed by FingerIO (in green) was accurate to within 8mm of the actual motion recorded on a separate touchscreen (shown in black). Credit: University of Washington


FingerIO turns a smartwatch or smartphone into a sonar system using the device's own speaker to emit an inaudible sound wave. That signal bounces off the finger, and those "echoes" are recorded by the device's microphones and used to calculate the finger's location in space.

Using sound waves to track finger motion offers several advantages over cameras—which don't work without line-of-sight when the device is hidden by fabric or another obstructions—and other technologies like radar that require both custom sensor hardware and greater computing power, said senior author and UW assistant professor of computer science and engineering Shyam Gollakota.

"Acoustic signals are great—because travel much slower than the radio waves used in radar, you don't need as much processing bandwidth so everything is simpler," said Gollakota, who directs the UW's Networks and Mobile Systems Lab. "And from a cost perspective, almost every device has a speaker and microphones so you can achieve this without any special hardware."

University of Washington computer science and engineering and electrical engineering researchers demonstrate FingerIO, a new technology from CSE's Networks and Mobile Systems Lab that employs sonar to enable users to interact with their smartphones and smartwatches by gesturing or writing on any nearby surface. Credit: University of Washington Department of Computer Science & Engineering



To test their approach, the researchers created a FingerIO prototype app for Android devices and downloaded it to an off-the-shelf Samsung Galaxy S4 smartphone and a smartwatch customized with two microphones, which are needed to track in two dimensions. Today's smartwatches typically only have one, which can be used to track a finger in one dimension.

The researchers asked testers to draw shapes such as stars, squiggles or figure 8s on a touchpad next to a smartphone or smartwatch running FingerIO. Then they compared the touchpad tracings to the shapes created by FingerIO's tracking.


"Given that your finger is already a centimeter thick, that's sufficient to accurately interact with the devices," said co-author and electrical engineering graduate student Vikram Iyer.


Explore further

Electromagnets and sensors track the motions of fingers

Citation: Smartwatches can now track your finger in mid-air using sonar (2016, March 15) retrieved 29 July 2021 from https://phys.org/news/2016-03-smartwatches-track-finger-mid-air-sonar.html

Feedback to editors

User comments