Drosophila is the most commonly used insect in the life sciences. In 2019, Professor Pavan Ramdya at EPFL’s School of Life Sciences and his group developed a deep-learning-based motion-capture software called DeepFly3d. The software, using multiple cameras, quantifies the movements of Drosophila in 3D space.
Later, in 2021, his team devised a method called LiftPose3D to reconstruct 3D animal poses from 2D images taken from a single camera. The team has created NeuroMechFly, the digital twin of the fruit fly, Drosophila melanogaster.
NeuroMechFly is a data-driven model of the Drosophila that combines: a physics-based simulation environment, a biomechanical exoskeleton, muscle models, and neural network controllers. The model realistically simulates the movements of the animal.
The team built NeuroMechFly using two kinds of data: First, they took a real fly and performed a CT scan to build a morphologically realistic biomechanical model. The second source of data was the real limb movements of the fly. The movements were obtained using pose estimation software.
Why build a digital twin of Drosophila?
Ramya said, “How do we know when we’ve understood a system? One way is to be able to recreate it. We might try to build a robotic fly, but it’s much faster and easier to build a simulated animal. So one of the major motivations behind this work is to start building a model that integrates what we know about the fly’s nervous system and biomechanics to test if it is enough to explain its behavior.”
“When we do experiments, we are often motivated by hypotheses. We’ve relied upon intuition and logic to formulate hypotheses and predictions until now. But as neuroscience becomes increasingly complicated, we rely more on models that can bring together many intertwined components, play them out, and predict what might happen if you made a tweak here or there.”
NeuroMechFly is a valuable testbed for biomechanics and robotics research, but only to the extent that it accurately reflects the genuine animal in a virtual environment. Verifying this was one of the scientist’s main concerns. They performed validation experiments and demonstrated that the model closely replicated the behaviors of the actual animal.
Scientists first made 3D measurements of real walking and grooming flies. They then replayed those behaviors using NeuroMechFly’s biomechanical exoskeleton inside a physics-based simulation environment.
The experiments showed that the model could predict various movement parameters that are otherwise unmeasured. Finally, they discovered neural network and muscle parameters that allow the fly to “run” in optimum ways for both speed and stability using NeuroMechFly’s complete neuromechanical capabilities.
Ramya said, “These case studies built our confidence in the model. But we are most interested in when the simulation fails to replicate animal behavior, pointing out ways to improve the model.”
“NeuroMechFly has been and will continue to be a community project. The software is open source and freely available for scientists to use and modify. We built a tool, not just for us but also for others. Therefore, we made it open source and modular and provided guidelines on using and modifying it.”
Journal Reference:
- Victor Lobato RÃos, Shravan Tata Ramalingasetty, et al. NeuroMechFly, a neuromechanical model of adult Drosophila melanogaster. Nature Methods 11 May 2022. DOI: 10.1038/s41592-022-01466-7