Robotic manipulation planning relies critically on selecting continuous values, such as grasps and object placements, that satisfy complex geometric and physical constraints, such as stability and lack of collision.
Existing approaches have used separate samplers for each constraint type obtained through learning or optimization. This process can be impractically time-consuming, with a long sequence of actions and a pile of luggage to pack.
A diffusion model, a type of generative AI called Diffusion-CCSP, was employed by MIT researchers to resolve this issue more effectively. Each machine-learning model in their approach has been trained to reflect a particular constraint. The packing problem is solved using a combination of these models that account for all limitations.
Their approach delivered more successful solutions simultaneously and produced practical answers more quickly than other approaches. Their method could also tackle issues involving novel combinations of restrictions and more significant numbers of objects, which the models had yet to encounter during training.
Their method can be used to teach robots how to comprehend and adhere to the general limitations of packing problems, such as the significance of avoiding collisions or a desire for one object to be near another because of its generalizability. This method of training robots could be used to perform various complicated jobs in multiple settings, such as filling orders in a warehouse or arranging bookshelves in a home.
Zhutian Yang, an electrical engineering and computer science graduate student, said, “My vision is to push robots to do more complicated tasks that have many geometric constraints and more continuous decisions that need to be made — these are the kinds of problems service robots face in our unstructured and diverse human environments. With the powerful tool of compositional diffusion models, we can now solve these more complex problems and get great generalization results.”
Diffusion models iteratively improve their output to produce fresh data samples that resemble samples in a training dataset.
Diffusion models learn a process for incrementally improving a potential solution to achieve this. Then, to address an issue, they begin with an arbitrary, appalling solution and progressively improve it.
Consider, for instance, randomly overlapping plates and other serving pieces on a model table. While qualitative constraints will pull the dish to the center, align the salad and dinner forks, etc., collision-free controls will cause the objects to push each other apart.
Yang said, “Diffusion models are well-suited for this kind of continuous constraint-satisfaction problem because the influences from multiple models on the pose of one object can be composed to encourage the satisfaction of all constraints. The models can obtain a diverse set of good solutions by starting from a random initial guess each time.”
Each type of constraint is represented by a different diffusion model in the family that Diffusion-CCSP learns. Since the models were trained simultaneously, they have specific knowledge in common, such as the geometry of the packing materials.
The models then collaborate to identify answers, in this case, places to put the items that satisfy all the restrictions.
Training individual models for each constraint type and then combining them to make predictions dramatically reduces the required training data compared to other approaches.
However, training these models still requires much data demonstrating solved problems. Humans would need to solve each problem with traditional slow methods, making the cost of generating such data prohibitive.
Instead, scientists turned the process around by coming up with ideas first. To ensure tight packing, stable poses, and collision-free solutions, they quickly generated segmented boxes and fitted a variety of 3D objects into each segment using their fast algorithms.
Yang said, “With this process, simulation data generation is almost instantaneous. We can generate tens of thousands of environments where we know the problems are solvable.”
“Trained using these data, the diffusion models work together to determine locations objects should be placed by the robotic gripper that achieves the packing task while meeting all of the constraints.”
They conducted feasibility studies and then used a real robot to show how Diffusion-CCSP could solve various challenging issues, such as packing 3D objects with a robotic arm, stacking 2D shapes with stability constraints, and fitting 2D triangles into a box.
In numerous studies, their strategy outperformed competing approaches, yielding a higher proportion of efficient solutions that were stable and collision-free.
Yang and her colleagues plan to try Diffusion-CCSP in more challenging scenarios in the future, as with mobile robots. Additionally, they intend to eliminate the requirement for Diffusion-CCSP to undergo new data training to solve issues in other areas.
Danfei Xu, an assistant professor in the School of Interactive Computing at the Georgia Institute of Technology and a Research Scientist at NVIDIA AI, who was not involved with this work, said, “Diffusion-CCSP is a machine-learning solution that builds on existing powerful generative models. It can quickly generate solutions simultaneously satisfying multiple constraints by composing known individual constraint models. Although it’s still in the early phases of development, the ongoing advancements in this approach hold the promise of enabling more efficient, safe, and reliable autonomous systems in various applications.”
Journal Reference:
- Zhutian Yang, Jiayuan Mao, Yilun Du et al. Compositional Diffusion-Based Continuous Constraint Solvers. arXiv: 2309.00966v1