MIT scientists have taken a step forward to leverage wireless signal’s (that flows around us and bouncing off our bodies) reflections with the aim of gaining valuable details on people’s behavior and health.
They have developed a system called Marko that works by transferring low-power radio-frequency (RF) signal into an environment. Those signals then get back to the system with certain changes if it has bounced off a moving human.
The system is originally based on algorithms that detect changing reflections and associate them with specific individuals. The system then checks whether each individual’s movement around a digital floor plan. Comparing their movement patterns, scientists can get insights on how people interact with each other and the environment.
Scientists describe the system and its real-world use in six locations: two assisted living facilities, three apartments inhabited by couples, and one townhouse with four residents. The case studies demonstrated the system’s ability to distinguish individuals based solely on wireless signals — and revealed some useful behavioral patterns.
In one assisted living facility, with permission from the patient’s family and caregivers, the researchers monitored a patient with dementia who would often become agitated for unknown reasons. Over a month, they measured the patient’s increased pacing between areas of their unit — a known sign of agitation. By matching increased pacing with the visitor log, they determined the patient was agitated more during the days following family visits. This shows Marko can provide a new, passive way to track functional health profiles of patients at home.
First author Chen-Yu Hsu, a Ph.D. student in the Computer Science and Artificial Intelligence Laboratory (CSAIL) said, “These are interesting bits we discovered through data. We live in a sea of wireless signals, and the way we move and walk around changes these reflections. We developed a system that listens to those reflections … to better understand people’s behavior and health.”
While testing in the home, scientists shoot out an RF signal. The signal gets rebounded and created a type of heat map cut into vertical and horizontal “frames,” which indicates where people are in a three-dimensional space.
In that heat map, people appear as bright blobs. Vertical frames capture the person’s height and build, whereas horizontal frames determine their general location. When a person walks, the system analyzes the RF frames — about 30 per second — to generate short trajectories, called tracklets.
Those tracklets are then used by a convolutional neural network to separate reflections by certain individuals. For each individual it senses, the system creates two “filtering masks,” which are small circles around the individual. These masks basically filter out all signals outside the circle, which locks in the individual’s trajectory and height as they move. Combining all this information — height, build, and movement — the network associates specific RF reflections with specific individuals.
For precise data, the system must be well trained. For a few days, individuals wear low-powered accelerometer sensors, which can be used to label the reflected radio signals with their respective identities.
While training, Marko primarily generated user’s tracklets. Then an algorithm correlates certain acceleration features with motion features. When users walk, for instance, the acceleration oscillates with steps but becomes a flat line when they stop. The algorithm finds the best match between the acceleration data and tracklet, and labels that tracklet with the user’s identity. In doing so, Marko learns which reflected signals correlate to specific identities.
The fantastic thing about the sensors, they do not require charging and after training, the individuals don’t need to wear them again. During the home deployment, the system was able to tag the identities of individuals in new homes with between 85 and 95 percent accuracy.
Scientists believe that the Marko could help health care facilities to positively monitor, how patients interact with family and caregivers, and whether patients receive medications on time. It also holds the potential to replace questionnaires and diaries currently used by psychologists or behavioral scientists to capture data on their study subjects’ family dynamics, daily schedules, or sleeping patterns, among other behaviors.
First author Chen-Yu Hsu, a Ph.D. student in the Computer Science and Artificial Intelligence Laboratory (CSAIL) said, “Most people aren’t that comfortable with being filmed all the time, especially in their own home. Using radio signals to do all this work strikes a good balance between getting some level of helpful information, but not making people feel uncomfortable.”
Scientists are planning to merge Marko with their prior work on inferring breathing and heart rate from the surrounding radio signals. This would help them better associate those biometrics with the corresponding individuals. It could also track people’s walking speeds, which is a good indicator of functional health in elderly patients.
Cecilia Mascolo, a professor of mobile systems in the Department of Computer Science and Technology at Cambridge University said, “The potential here is immense. With respect to imaging through cameras, it offers a less data-rich and more targeted model of collecting information, which is very welcome from the user privacy perspective. The data collected, however, is still very rich, and the paper evaluation shows accuracy which can enable a number of very useful applications, for example in elderly care, medical adherence monitoring, or even hospital care.”
“Yet, as a community, we need to aware of the privacy risks that this type of technology bring. Certain computation techniques, she says, should be considered to ensure the data remains private.”
he research is led by Dina Katabi, the Andrew and Erna Viterbi Professor of Electrical Engineering and Computer Science and director of the MIT Center for Wireless Networks and Mobile Computing (Wireless@MIT). Joining Katabi and Hsu on the paper are CSAIL graduate students Mingmin Zhao and Guang-He Lee and alumnus Rumen Hristov SM ’16.