Introducing LiftPose3D, a method to reconstruct 3D animal poses using only 2D poses

Turning 2D images into 3D models.


EPFL scientists have introduced new software called LiftPose3D to ease one of neuroscience’s most crucial yet laborious tasks: capturing 3D models of freely moving animals. It allows scientists to reconstruct 3D animal poses using only 2D poses from one camera.

Using this deep learning-based method, scientists could study brain mechanisms that control body movements.

Professor Pavan Ramdya at EPFL‘s School of Life Sciences said, “In the past, we used a deep neural network to perform this kind of ‘pose estimation’ in animals. Each camera acquired a single image of an animal, and multiple images across different cameras could then be triangulated to calculate 3-dimensional positions or poses.”

“But this triangulation of images requires multiple, synchronized cameras and elaborate calibration protocols, making it hard to adopt for neuroscientific studies of small animals.”

“The challenges we wanted to overcome here was, first to reduce the number of cameras needed to perform 3D pose estimation. Second, we wanted to address the problem of occlusions, where one of an animal’s body parts can move in front of another, obstructing the camera’s view and making full triangulation impossible.”

Animals are most likely to repeat the same behavior. This reproducibility allowed the scientists to train a neural network to map 2D poses onto 3D positions, reducing the number of cameras needed and overcoming the problem of occlusions.

Ramdya said“We use deep networks that track 2D poses from each camera view, and then another network that then maps these 2D positions or key points to a library of 3D poses.”

“Now we can use data from previous experiments, where people performed 3D pose estimation on animals, to train our network. We also performed a couple of tricks to generalize that mapping across datasets from different experimental systems and different laboratories. For example, in another lab, they might place their cameras in slightly different positions. So we trained our network to be able to generalize across these potential variations.”

“Another advantage of LiftPose3D is that it works with animals that are moving freely as opposed to tethered in a limited space–the usual practice in pose estimation studies. To understand the nervous system, one must also take into account the biomechanics involved in real behaviors. For example, when a cockroach runs up a hill, physical interactions between the animal and its environment are critical but cannot be captured if it is tethered. Now, with LiftPose3D, we can record 3D poses in freely behaving animals, capturing these body-environment interactions.

“Since it needs no specific hardware, LiftPose3D can make pose-estimation studies much easier and cheaper to perform. Our vision is to reverse engineer the nervous system and behavior to inform the design of robotics controllers. The most effective way to do that is using experimentally accessible animal models. We designed LiftPose3D specifically to probe these models with fewer cameras, allowing us to get closer to the goal of reverse-engineering the mechanisms that give rise to their complex behaviors.”

Journal Reference:
  1. Adam Gosztolai, Semih Günel et al. LiftPose3D, a deep learning-based approach for transforming 2D to 3D poses in laboratory animals. Nature Methods 05 August 2021. DOI: 10.1101/2020.09.18.292680