Different people have different learning styles. While some may prefer visual or auditory aids, others find it more effective to learn through touch and hands-on experience. Technology is constantly evolving to cater to learners’ diverse needs and preferences.
Researchers have developed smart gloves that cater to tactile learners by incorporating haptic feedback and AI technology. With the help of such innovative tools, learners can gain new skills, enhance their precision, and even control robots remotely.
Researchers from MIT‘s Computer Science and Artificial Intelligence Laboratory (CSAIL) and elsewhere have developed an embroidered smart glove that can capture, reproduce, and relay touch-based instructions.
The team also designed a machine-learning agent that can adapt to different users’ reactions to tactile feedback, which is truly remarkable. With the potential to improve responsive robot teleoperation, assist with training in virtual reality, and teach people physical skills, this new system could have far-reaching benefits.
“Humans engage in a wide variety of tasks by constantly interacting with the world around them,” says Yiyue Luo MS ’20, lead author of the paper, Ph.D. student in MIT’s Department of Electrical Engineering and Computer Science (EECS), and CSAIL affiliate. “We don’t usually share these physical interactions with others. Instead, we often learn by observing their movements, like with piano-playing and dance routines.
“The main challenge in relaying tactile interactions is that everyone perceives haptic feedback differently,” adds Luo. “This roadblock inspired us to develop a machine-learning agent that learns to generate adaptive haptics for individuals’ gloves, introducing them to a more hands-on approach to learning optimal motion.”
To create their smart glove, the researchers used a digital embroidery machine to embed tactile sensors and haptic actuators into textiles, making the gloves ready for use in 10 minutes. This technology can capture the hand and finger movements of a teacher, surgeon, or machine operator and reproduce them as haptic feedback for the learner. Even more impressive, the machine-learning agent behind the smart glove can tailor feedback responses for each individual user in just 15 seconds.
The smart glove technology can teach people how to play the piano. In a demonstration, an expert recorded a simple tune using the glove, which captured the sequence of finger presses on the keys. This sequence was then converted into haptic feedback by a machine-learning agent and fed into the students’ gloves as instructions.
The actuators on the fingers corresponding to the keys below then vibrated in response. The system even optimizes the directions for each user based on their touch interactions.
In two other experiments, tactile directions with time-sensitive feedback were transferred to users sporting the gloves while playing laptop games. In a rhythm game, the players learned to follow a narrow, winding path to bump into a goal area, and in a racing game, drivers collected coins and maintained the balance of their vehicle on their way to the finish line. The team found that participants earned the highest game scores through optimized haptics, as opposed to without haptics and with unoptimized haptics.
“This work is the first step to building personalized AI agents that continuously capture data about the user and the environment,” says senior author Wojciech Matusik, MIT professor of electrical engineering and computer science and head of the Computational Design and Fabrication Group within CSAIL. “These agents then assist them in performing complex tasks, learning new skills, and promoting better behaviors.”
This wearable technology can provide a hands-on experience for users and can be extended beyond fingers to guide other body parts like feet, hips, and more.
Luo noted that the potential applications for this technology are vast, and with a more complex artificial intelligence agent, it could assist with more complicated tasks like manipulating clay or driving an airplane.
Journal reference:
- Yiyue Luo, Chao Liu, Young Joong Lee, Joseph DelPreto, Kui Wu, Michael Foshey, Daniela Rus, Tomás Palacios, Yunzhu Li, Antonio Torralba, and Wojciech Matusik. Adaptive tactile interaction transfer via digitally embroidered smart gloves. Nature Communications, 2024; DOI: 10.1038/s41467-024-45059-8