Robohub.org
 

Robots that feel by seeing


by
06 March 2021



share this:

Humanoid robot hand. Futuristic cyborg concept.

While modern cameras provide machines with a very well-developed sense of vision, robots still lack such a comprehensive solution for their sense of touch. At ETH Zurich, in the group led by Prof. Raffaello D’Andrea at the Institute for Dynamic Systems and Control, we have developed a tactile sensing principle that allows robots to retrieve rich contact feedback from their interactions with the environment. I recently described our approach in a TEDx talk at the last TEDxZurich. The talk features a tech demo that introduces the novel tactile sensing technology targeting the next generation of soft robotic skins.

Design

The sensing technique is based on a camera that tracks fluorescent particles, which are densely and randomly distributed within a soft, deformable gel. The randomness of the patterns simplifies production of the gel and their density provides strain information at each pixel of the resulting image. In addition, the technique does not make any assumption about the shape of the sensing surface, which can exhibit an arbitrary geometry.

Data Processing

While the images capturing the particles’ motion are intuitive and to some extent visually interpretable, the extraction of accurate physical quantities is challenging. In order to overcome the complexity of modeling the behavior of soft materials in real-time, the information extracted from the images is mapped to the distribution of the applied (shear and pressure) contact forces in a data-driven fashion. Specifically, a neural network is employed, which is trained entirely via accurate finite-element simulations to extract the aforementioned mapping.

Applications

This technology has the potential to impact several application fields, the obvious one being robotic manipulation. In a recent proof-of-concept, we have demonstrated how highly dynamic manipulation tasks can be achieved via the sole use of tactile sensing, as shown in the video below.

In addition, the versatility of this approach makes it suitable for various products beyond the robotics domain. In fact, an artificial sense of touch may find applications in smart prosthetic systems, with the potential to restore tactile sensations to people who have lost limbs.



tags: , , ,


Carlo Sferrazza is a PhD candidate at the Institute for Dynamic Systems and Control, ETH Zurich, under the supervision of Prof. Dr. Raffaello D’Andrea. His current research interests include the design and the development of vision-based, data-driven tactile sensors, and learning-based model predictive control.
Carlo Sferrazza is a PhD candidate at the Institute for Dynamic Systems and Control, ETH Zurich, under the supervision of Prof. Dr. Raffaello D’Andrea. His current research interests include the design and the development of vision-based, data-driven tactile sensors, and learning-based model predictive control.

            AUAI is supported by:



Subscribe to Robohub newsletter on substack



Related posts :

Robot Talk Episode 171 – Programming robot behaviour, with Sebastian Castro

  20 Nov 2026
In the latest episode of the Robot Talk podcast, Claire chatted to Sebastian Castro about developing open-source software for robot task planning and behaviour.

Soft exosuit makes it easier to stand up

Robotic system helps users stand up and walk more efficiently.

Robot swarms trained by trial and error in simulation keep working on real robots

  17 Nov 2026
Researchers have developed a new approach to reinforcement learning that helps robot swarms, allowing behaviours learned in simulation to transfer reliably to real-world robots.

Watch the IROS 2026 plenary and keynote talks

  16 Nov 2026
Catch up on the invited presentations from the International Conference on Intelligent Robots and Systems.

Robot Talk Episode 170 – Robots in the real world, with Daniela Shchegelskaya

  13 Nov 2026
In the latest episode of the Robot Talk podcast, Claire chatted to Daniela Shchegelskaya from the University of Twente about getting robots to operate reliably in safety-critical environments.

AI100 report released: embodied AI highlighted as grand challenge

  11 Nov 2026
Embodied AI is listed as one of the most inspiring open grand challenges.

AIhub coffee corner: Are humanoids the future of embodied AI?

  09 Nov 2026
The AIhub coffee corner captures the musings of AI experts over a short conversation.

Robot Talk Episode 169 – Robot sensing inspired by nature, with Dexter Shepherd

  06 Nov 2026
In the latest episode of the Robot Talk podcast, Claire chatted to Dexter Shepherd from the University of Sussex about using artificial intelligence to help robots understand and adapt to their environment.


↑


AUAI is supported by:







Subscribe to Robohub newsletter on substack




 















©2026.05 - Association for the Understanding of Artificial Intelligence