Robohub.org
 

Helping robots handle fluids


by
27 May 2023



share this:

Researchers created “FluidLab,” a simulation environment with a diverse set of manipulation tasks involving complex fluid dynamics. Image: Alex Shipps/MIT CSAIL via Midjourney

Imagine you’re enjoying a picnic by a riverbank on a windy day. A gust of wind accidentally catches your paper napkin and lands on the water’s surface, quickly drifting away from you. You grab a nearby stick and carefully agitate the water to retrieve it, creating a series of small waves. These waves eventually push the napkin back toward the shore, so you grab it. In this scenario, the water acts as a medium for transmitting forces, enabling you to manipulate the position of the napkin without direct contact.

Humans regularly engage with various types of fluids in their daily lives, but doing so has been a formidable and elusive goal for current robotic systems. Hand you a latte? A robot can do that. Make it? That’s going to require a bit more nuance. 

FluidLab, a new simulation tool from researchers at the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL), enhances robot learning for complex fluid manipulation tasks like making latte art, ice cream, and even manipulating air. The virtual environment offers a versatile collection of intricate fluid handling challenges, involving both solids and liquids, and multiple fluids simultaneously. FluidLab supports modeling solid, liquid, and gas, including elastic, plastic, rigid objects, Newtonian and non-Newtonian liquids, and smoke and air. 

At the heart of FluidLab lies FluidEngine, an easy-to-use physics simulator capable of seamlessly calculating and simulating various materials and their interactions, all while harnessing the power of graphics processing units (GPUs) for faster processing. The engine is “differential,” meaning the simulator can incorporate physics knowledge for a more realistic physical world model, leading to more efficient learning and planning for robotic tasks. In contrast, most existing reinforcement learning methods lack that world model that just depends on trial and error. This enhanced capability, say the researchers, lets users experiment with robot learning algorithms and toy with the boundaries of current robotic manipulation abilities.

To set the stage, the researchers tested said robot learning algorithms using FluidLab, discovering and overcoming unique challenges in fluid systems. By developing clever optimization methods, they’ve been able to transfer these learnings from simulations to real-world scenarios effectively. 

“Imagine a future where a household robot effortlessly assists you with daily tasks, like making coffee, preparing breakfast, or cooking dinner. These tasks involve numerous fluid manipulation challenges. Our benchmark is a first step towards enabling robots to master these skills, benefiting households and workplaces alike,” says visiting researcher at MIT CSAIL and research scientist at the MIT-IBM Watson AI Lab Chuang Gan, the senior author on a new paper about the research. “For instance, these robots could reduce wait times and enhance customer experiences in busy coffee shops. FluidEngine is, to our knowledge, the first-of-its-kind physics engine that supports a wide range of materials and couplings while being fully differentiable. With our standardized fluid manipulation tasks, researchers can evaluate robot learning algorithms and push the boundaries of today’s robotic manipulation capabilities.”

Fluid fantasia

Over the past few decades, scientists in the robotic manipulation domain have mainly focused on manipulating rigid objects, or on very simplistic fluid manipulation tasks like pouring water. Studying these manipulation tasks involving fluids in the real world can also be an unsafe and costly endeavor. 

With fluid manipulation, it’s not always just about fluids, though. In many tasks, such as creating the perfect ice cream swirl, mixing solids into liquids, or paddling through the water to move objects, it’s a dance of interactions between fluids and various other materials. Simulation environments must support “coupling,” or how two different material properties interact. Fluid manipulation tasks usually require pretty fine-grained precision, with delicate interactions and handling of materials, setting them apart from straightforward tasks like pushing a block or opening a bottle. 

FluidLab’s simulator can quickly calculate how different materials interact with each other. 

Helping out the GPUs is “Taichi,” a domain-specific language embedded in Python. The system can compute gradients (rates of change in environment configurations with respect to the robot’s actions) for different material types and their interactions (couplings) with one another. This precise information can be used to fine-tune the robot’s movements for better performance. As a result, the simulator allows for faster and more efficient solutions, setting it apart from its counterparts.

The 10 tasks the team put forth fell into two categories: using fluids to manipulate hard-to-reach objects, and directly manipulating fluids for specific goals. Examples included separating liquids, guiding floating objects, transporting items with water jets, mixing liquids, creating latte art, shaping ice cream, and controlling air circulation. 

“The simulator works similarly to how humans use their mental models to predict the consequences of their actions and make informed decisions when manipulating fluids. This is a significant advantage of our simulator compared to others,” says Carnegie Mellon University PhD student Zhou Xian, another author on the paper. “While other simulators primarily support reinforcement learning, ours supports reinforcement learning and allows for more efficient optimization techniques. Utilizing the gradients provided by the simulator supports highly efficient policy search, making it a more versatile and effective tool.”

Next steps

FluidLab’s future looks bright. The current work attempted to transfer trajectories optimized in simulation to real-world tasks directly in an open-loop manner. For next steps, the team is working to develop a closed-loop policy in simulation that takes as input the state or the visual observations of the environments and performs fluid manipulation tasks in real time, and then transfers the learned policies in real-world scenes.

The platform is publicly publicly available, and researchers hope it will benefit future studies in developing better methods for solving complex fluid manipulation tasks.

“Humans interact with fluids in everyday tasks, including pouring and mixing liquids (coffee, yogurts, soups, batter), washing and cleaning with water, and more,” says University of Maryland computer science professor Ming Lin, who was not involved in the work. “For robots to assist humans and serve in similar capacities for day-to-day tasks, novel techniques for interacting and handling various liquids of different properties (e.g. viscosity and density of materials) would be needed and remains a major computational challenge for real-time autonomous systems. This work introduces the first comprehensive physics engine, FluidLab, to enable modeling of diverse, complex fluids and their coupling with other objects and dynamical systems in the environment. The mathematical formulation of ‘differentiable fluids’ as presented in the paper makes it possible for integrating versatile fluid simulation as a network layer in learning-based algorithms and neural network architectures for intelligent systems to operate in real-world applications.”

Gan and Xian wrote the paper alongside Hsiao-Yu Tung a postdoc in the MIT Department of Brain and Cognitive Sciences; Antonio Torralba, an MIT professor of electrical engineering and computer science and CSAIL principal investigator; Dartmouth College Assistant Professor Bo Zhu, Columbia University PhD student Zhenjia Xu, and CMU Assistant Professor Katerina Fragkiadaki. The team’s research is supported by the MIT-IBM Watson AI Lab, Sony AI, a DARPA Young Investigator Award, an NSF CAREER award, an AFOSR Young Investigator Award, DARPA Machine Common Sense, and the National Science Foundation.

The research was presented at the International Conference on Learning Representations earlier this month.




MIT News





Related posts :



Robot Talk Episode 123 – Standardising robot programming, with Nick Thompson

  30 May 2025
In the latest episode of the Robot Talk podcast, Claire chatted to Nick Thompson from BOW about software that makes robots easier to program.

Congratulations to the #AAMAS2025 best paper, best demo, and distinguished dissertation award winners

  29 May 2025
Find out who won the awards presented at the International Conference on Autonomous Agents and Multiagent Systems last week.

Congratulations to the #ICRA2025 best paper award winners

  27 May 2025
The winners and finalists in the different categories have been announced.

#ICRA2025 social media round-up

  23 May 2025
Find out what the participants got up to at the International Conference on Robotics & Automation.

Robot Talk Episode 122 – Bio-inspired flying robots, with Jane Pauline Ramos Ramirez

  23 May 2025
In the latest episode of the Robot Talk podcast, Claire chatted to Jane Pauline Ramos Ramirez from Delft University of Technology about drones that can move on land and in the air.

Robot Talk Episode 121 – Adaptable robots for the home, with Lerrel Pinto

  16 May 2025
In the latest episode of the Robot Talk podcast, Claire chatted to Lerrel Pinto from New York University about using machine learning to train robots to adapt to new environments.

What’s coming up at #ICRA2025?

  16 May 2025
Find out what's in store at the IEEE International Conference on Robotics & Automation, which will take place from 19-23 May.

Robot see, robot do: System learns after watching how-tos

  14 May 2025
Researchers have developed a new robotic framework that allows robots to learn tasks by watching a how-to video



 

Robohub is supported by:




Would you like to learn how to tell impactful stories about your robot or AI system?


scicomm
training the next generation of science communicators in robotics & AI


©2025.05 - Association for the Understanding of Artificial Intelligence


 












©2025.05 - Association for the Understanding of Artificial Intelligence