Sensor Turns Any Surface, Even the Human Body, Into Input Device
A research team from the University of Michigan and Meta Reality Labs has developed a system to turn any surface—clothes, countertops, even clothing—into an input device for swipes, taps, and scrolls.
Researchers from the University of Michigan Interactive Sensing & Computing (ISC) Lab and Meta Reality Labs recently presented a novel input device robust against environmental noise at the 2023 Conference on Human Factors in Computing Systems (CHI ’23).
Capacitive input sensing makes the smartphone touch screen possible. It allows users to use a variety of gestures to open or close apps, quickly write a message, or dismiss alerts. In the time before touchscreen phones, number keypads would require 13 button presses (4-4-3-3-5-5-5-5-5-5-6-6-6) just to write hello.
SAWSense recognizes different types of touch input on soft or irregularly-shaped surfaces, even a person's coat sleeve. Image used courtesy of ISC Lab
Now, in the next improvement to capacitive sensing technology, the University of Michigan team and its Meta partners have demonstrated how their SAWSense system can repurpose technology commonly used in bone-conduction microphones to pick up acoustic waves that travel along the surface of objects. This system is said to operate on non-flat surfaces, like a human arm or a toy, and on soft fabrics like furniture or clothing. In addition to working in noisy environments, the researchers also reported that the system recognizes different inputs (swipes, taps, and scratches, for instance) with extremely high accuracy—all without the need for other common input methods, like touch screens, cameras, or pads.
The Challenges of Existing Input Methods
Since the first touchscreen phone, voice assistants have made verbal input possible; other sensors have enabled hand gestures or eye tracking to interact with our devices.
However, many input methods are limited in what kinds of gestures or inputs can be sensed. For example, accelerometers usually have a sampling bandwidth of less than 250 Hz and piezoelectric microphones may be limited to frequencies between 2–4 kHz. Camera-based methods may not distinguish between close contact and actual contact, while microphones can struggle with ambient noise.
SAWSense attempts to overcome these limitations by focusing on surface-acoustic waves (SAW), 2D waves that only exist and propagate at the surface-to-air boundary. SAWs can be detected at a wide frequency bandwidth, making it possible to distinguish between a scratch and a swipe on a surface—even from faraway distances.
SAWSense Leverages Voice Pick-up Units to Detect Surface-acoustic Waves
The research team developed SAWSense by repurposing voice pick-up units (VPUs), which are used in microphones to translate vibrations from the vocal cords to the inner ear. Designed to detect SAWs, VPUs are hermetically sealed, ensuring that air-based sound waves are not detected. Only acoustic waves from surfaces in contact with the sensor are recognized.
Voice pick-up unit on SAWSense. Image used courtesy of ISC Lab
When the VPUs detect SAWs, it feeds those signals through signal processing software that uses machine learning to classify the type of gesture with great than 97% accuracy.
When a user flings his fingernail against a table, the motion generates surface-acoustic waves detectable by VPUs, which are then interpreted with a machine-learning pipeline. Image used courtesy of ISC Lab
In the team’s experiment, SAWSense detected tapping, swiping, and flicking on a tabletop surface in both noisy and quiet environments. By using two VPUs on a table surface, researchers also demonstrated how a webpage can either be scrolled or navigated by swiping or tapping.
Compatible With Odd Geometries and Materials
SAWSense was also demonstrated on different surfaces to show that it’s not limited to flat table-top-like environments—even proving operable with the human body as an input source.
Turning a toy into an input device, SAWSense detects different motions like belly rubs. Image used courtesy of ISC Lab
In one experiment, the system could distinguish petting, belly rubs, and pretend feeding motions on a toy. When attached to a coat sleeve, SAWSense detected various gestures like squeezing. On a speaker, the sensor could detect scratches or taps even when music was playing.
This study opens up different possibilities for what can be considered an input device and the different types of interactions users can have with it. While the researchers conceded that vibrations or surface-acoustic waves can be detected with other tools, like accelerometers or piezoelectric sensors, SAWSense operates in a higher frequency bandwidth, allowing it to capture a much broader range of signals and reject external waves.
Turning Any Surface Into an Input Receiver
The team sees SAWSense as an alternative input method that can provide greater privacy. Since SAWs can only propagate at the surface-to-air boundary, a personal desk in an office, for instance, can be used as a more private input surface compared to voice inputs.
The team plans to use SAWSense with multiple VPUs to localize inputs on a surface and to combine them with other input types, such as vision-based ones, to correct input errors. The researchers also suggest that SAWSense could eventually be used in medical applications where the system could detect subtle sounds, such as joints cracking and popping, to provide insight for diagnosis and treatment.
Perhaps the most promising use cases for SAWSense, the researchers say, are in a smart home. SAWSense can be used to turn a kitchen countertop into a surface sensor, allowing it to identify when it's being used to blend, whisk, chop, or stir. In fact, in demonstrations, the system successfully classified 16 different kitchen activities (chopping, closing a microwave door, using a food processor, etc.).
Because smart-home technologies using SAWSense would rely on surface-acoustic waves instead of cameras or microphones, they offer an attractive privacy-focused alternative to traditional input methods.
