New Wearable Prototype from Cornell’s SciFi Lab Senses Hand Position, Even Through Objects

July 23, 2020 by Gary Elinoff

By means of four wrist-mounted thermal cameras, the detailed positions of 20 points on the human hand can be tracked in real time.

Researchers from Cornell and the University of Wisconsin have developed FingerTrak, a wrist-mounted device that can quantify and record the complete and complex positioning of the entire human hand without the need to actually see the fingers directly.

Instead, the outline or the contour of the hand are inferred from the wrist through the use of four tiny, low-resolution thermal cameras aided by artificial intelligence. 

FingerTrak. Image used courtesy of Cornell University


The system transmits the images garnered from the thermal cameras to a deep neural network customized to “stitch” the images together to estimate 20 joint positions in 3D space. 

“This was a major discovery by our team—that by looking at your wrist contours, the technology could reconstruct in 3D, with keen accuracy, where your fingers are,” according to Assistant professor Cheng Zhang, the director of Cornell’s Smart Computer Interfaces for Future Interactions (SciFi) Lab.

While the SciFi Lab itself is involved with multiple projects related to man-machine interactions related to the human hand, FingerTrak is a notable milestone. What makes this system unique is that this is the first time the contours of the wrist have been used to reconstruct full hand posture.

A paper describing the results of the work, “FingerTrak: Continuous 3D Hand Pose Tracking by Deep Learning Hand Silhouettes Captured by Miniature Thermal Cameras on Wrist,” was published in the Proceedings of the Association for Computing Machinery on Interactive, Mobile, Wearable and Ubiquitous Technologies


Previous Implementations of Wrist-Mounted Cameras

Previous deployments of wrist-mounted cameras to track hand movements have depended on the use of cameras to directly capture the position of the fingers. These proved to be too bulky to be of practical use, and could anyway only capture limited information. By contrast, FingerTrak employs a dyad of thermal imaging and machine learning to virtually reconstruct the hand. 

The device is built around four pea-sized thermal cameras, each slightly larger than a third of an inch, take multiple “silhouette” images to form an outline of the hand. These images are then fed to a custom deep neural network which reconstructs a virtual 3D rendering of the entire hand. Through FingerTrack, the 3D image of the entire hand pose can be captured, even when the hand is holding an actual physical object.

Findings of the Study

FingerTrak proved capable of continuously reconstructing entire hand postures, consisting of 20 finger joints positions, without needing to image all fingers. Rather, the system was able to estimate the entire hand posture by observing only the outline of the hand, from the vantage point of the wrist, using low-resolution (32 x 24) thermal cameras.

FingerTrak achieved an average angular error of 6.46° when tested under the same background, and 8.06° when there was a changed background.


Video used courtesy of Cornell University


The picture above is taken from a video explaining the function of FingerTrack. The images on the lower left are the actual images captured by the thermal cameras. The hand with the red joints is the estimated hand pose given by the system and superimposed on the actual human hand. 

Finally, the blue artificial hand illustrated on the lower right is being controlled by FingerTrack. Viewers of the video will note the astonishingly accurate manner in which the artificial hand copies the movement, in real time, of the human hand.


Potential Uses for FingerTrack Technology

The potential applications for FingerTrak include communications, medical wearables, and gesture recognition. 

Professor Zhang sees sign language translation as an important potential for this work. 

Additionally, according to Yin Li, one of the paper’s authors, “How we move our hands and fingers often tells about our health condition.” Conditions such as Parkinson's and Alzheimer's diseases may be identified earlier by studying how early signs and symptoms manifest in hand movements.

And, of course, FingerTack has big implications for gaming and all other sorts of alternate and virtual reality applications.