Wearables are really indispensable devices; in fact, help us during sports training, monitor our vital functions, and increase the connectivity of the smartphone. Researchers continue to work on solutions that may offer more detailed hand tracking.
Now, a team of researchers at Cornell University’s SciFi Lab with assistance from the University of Wisconsin-Madison has designed a wrist-mounted device that continuously tracks the entire human hand in 3D. The device called FingerTrak uses three or four miniature, low-resolution thermal cameras, and the potential of Machine Learning to detect and interpret hand movements and positions.
The miniature thermal cameras are mounted closely on a form-fitting wristband, each the size of about a pea. They read contours on the wrist and snap multiple “silhouette” images to form an outline of the hand. A customized deep neural network then stitches these multi-view silhouette images together and estimate 20 finger joints positions in 3D space to create the 3D model of the hand.
When tested under a different background, Cheng Zhang, assistant professor of information science, and his fellow researchers were able to capture the entire hand pose, even when the hand is holding an object. FingerTrak also has the potential to reconstruct some of the complicated poses.
The 3D hand-sensing wristband has many potential applications. It could have an impact on health care applications, specifically in monitoring disorders that affect fine-motor skills. A device like this might be used to better understand how the elderly use their hands in daily life, helping to detect early signs of diseases like Parkinson’s and Alzheimer’s, explained Yin Li, who contributed to the software behind FingerTrak.
Besides, the researchers highlight that this device can be used in sign language translation, virtual reality, mobile health, human-robot interaction, and other areas, since it is lightweight and harness the potential of AI.
“The most novel technical finding in this work is discovering that the contours of the wrist are enough to accurately predict the entire hand posture,” Zhang said. “This finding allows the reposition of the sensing system to the wrist, which is more practical for usability.”
In prototype form, FingerTrak is already quite small. Nonetheless, this one could easily get even smaller with further engineering. At the moment, this project is in its initial stage, so we will have to wait to see it in action. However, what is promising here is that the device uses affordable components and is already very accurate.
Journal Reference:
- FingerTrak: Continuous 3D Hand Pose Tracking by Deep Learning Hand Silhouettes Captured by Miniature Thermal Cameras on Wrist. DOI: 10.1145/3397306