Repeatable robotics: Standards and benchmarks from frontier science to applied technology

23 July 2015

share this:

factory_Making_robotsIn a recent article in Frontiers in Robotics and AI, Gianluca Antonelli suggests that the Grand Challenge of robotics is to systematize the way its research is conducted. At Robohub we believe this is an important topic that needs to trace the arc between academia, R&D and even commercial territory. Robotics is so fundamentally multidisciplinary that traditional scientific methods can be hard to apply, and lab experiments difficult to validate, reproduce, or even adequately catalogue within the confines of traditional journal publications.

Is robotics a science, or is it a technology, and how do we demonstrate/benchmark either? Scientific method exists to test science; standards and regulations exist to benchmark technologies. Researchers face the issue of repeatability at the experimental level. Product developers face the issue at the prototype level, and producers of consumer product face the issue at a commercial level. Robotics is an emerging technology – caught between the research lab and the real world – and frequently falls into the abyss between.

Publishing and peer review alone cannot keep pace with the full spectrum robotics developments, and not everything can be contained in a paper. Indeed, robotic systems entwine both the physical and the non-physical in ways that are difficult to catalogue entirely in journal format. For example, code can not be adequately cited or valued academically, yet is fundamental to robotics and AI research. At the same time, standards and regulatory frameworks push robotics in certain directions that might not be optimal. As open source code increasingly becomes the foundation for commercial products, the need to ensure repeatability, robustness and validation becomes essential, not just for commercially scaled producers, but for regulators as well.

At the same time, academic development requires flexibility, creativity, and unboundedness. How else could you mix chemistry and robotics, as in the recent paper on designing soft jumping robots published in Science? Breakthroughs require state-of-the-art knowhow that is sometimes difficult to reproduce outside the original lab setting, but shows potential for developments further down the line.

We know the robotics community understands this need, hence the call for research benchmarks, competitions, current ROS efforts, and new standards for collaborative robotics. But approaches are fragmented and neither widely established nor confirmed. We believe it is time to review these issues, and to examine the various solutions proposed to date. Over the next several months Robohub will be delving more deeply into how these issues are being played out between the research lab and the real world.

Interested in participating in the discussion? Leave your comments below, or send a draft proposal to info[at]

Relevant links:

tags: , , ,

Robohub Editors

Related posts :




Robotics Grasping and Manipulation Competition Spotlight, with Yu Sun

Yu Sun, previous chair of the Robotics Grasping and Manipulation Competition, speaks on the value that this competition brought to the robotics community.
21 May 2022, by



Early Days of ICRA Competitions, with Bill Smart

Bill Smart, one fo the early ICRA Competition Chairs, dives into the high-level decisions involved with creating a meaningful competition.
21 May 2022, by

New imaging method makes tiny robots visible in the body

Microrobots have the potential to revolutionize medicine. Researchers at the Max Planck ETH Centre for Learning Systems have now developed an imaging technique that for the first time recognises cell-​sized microrobots individually and at high resolution in a living organism.
20 May 2022, by

A draft open standard for an Ethical Black Box

Within the RoboTIPS project, we have developed and tested several model of Ethical Black Boxes, including one for an e-puck robot, and another for the MIRO robot.
19 May 2022, by

Unable to attend #ICRA2022 for accessibility issues? Or just curious to see robots?

There are many things that can make it difficult to attend an in person conference in the United States and so the ICRA Organizing Committee, the IEEE Robotics and Automation Society and OhmniLabs would like to help you attend ICRA virtually.
17 May 2022, by



Duckietown Competition Spotlight, with Dr Liam Paull

Dr. Liam Paull, cofounder of the Duckietown competition talks about the only robotics competition where Rubber Duckies are the passengers on an autonomous driving track.
17 May 2022, by

©2021 - ROBOTS Association


©2021 - ROBOTS Association