Robohub.org
 

Replicable and measureable robotics research: Back to the basics of the scientific method

by
30 September 2016



share this:
robot-and-scientist

In general, as any robotics and/or AI PhD student knows very well, replicating the research results of other labs is quite difficult. The information you can customary find in a reputed journal paper is usually not enough to reproduce the experimental results claimed by the authors, let alone to make comparisons of the strengths and weaknesses of the different methods proposed in the literature in term of performances.

This is a serious issue, as the possibility to reproduce results is the cornerstone of the scientific method. Science is defined by the possibility to experimentally verify, Karl Popper would say ‘falsify’, ‘theories’. This is even more striking when at least part of robotics research is more and more regarded as ‘science,’ as for example, it is witnessed by the recent launch by AAAS with the new journal Science Robotics. By the way, what’s a ‘theory’ in robotics and AI?

This uncomfortable situation not only makes the cumulative process of research difficult, it also severely impairs technology transfer and industrial exploitation. It is worth noting that while everybody includes a state-of-the-art section in grant applications or business plans, the sad truth is that state-of-the-art is (more or less) insightfully guessed as published results cannot in many, if not most, cases be checked objectively.

And what about the TRLs (Technology Readiness Levels) at the core of SPARC Strategic Research Agenda? The community has been aware of this problem for a long time. In 2008, the European Robotics Network (EURON) started a Special Interest Group on Good Experimental Methodology and Benchmarking (coordinated by me and co-chaired by John Hallam and Angel P. Del Pobil), and the following year within IEEE RAS, the TC Pebras started, that is still active today. In 2012, the Euron GEM SIG led to the establishment by euRobotics aisbl of the Topic Group on Replicable Robotics Research, Benchmarking and Competition, which I coordinate.

Series of workshops at IROS, ICRA and RSS have debated the related issues and proposed examples of reproducible experiments and measurable results. We are now at a point where we can provide concrete directions and guidelines for reproducible research in robotics and AI. In September 2015, the first ever “Special Issue on Reproducible Robotics Research” was published in the IEEE RAS Robotics and Automation Magazine. Reproducible research is becoming an IEEE priority, there are no more excuses to indulge in ‘proof by video’ and ‘it worked once in my lab’ attitudes.

Please check the links below if you want to learn more.

  1. Bonsignorio F., Del Pobil A., (Eds.), Replicable and Measurable Robotics Research, IEEE Robotics & Automation Magazine, 22(3), 2015 http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7254310
  2. Bonsignorio F., del Pobil A.P.,Toward Replicable and Measurable Robotics Research [From the Guest Editors], Robotics & Automation Magazine, 22 (3), 32-35, 2015 http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=7254280&punumber=100
  3. Stoelen M. F. , de Tejada V. F., Huete A. J., Balaguer C., Bonsignorio F., Distributed and Adaptive Shared Control Systems: Methodology for the Replication of Experiments, IEEE Robotics & Automation Magazine, 22(4),137âAS146, 2015 http://ieeexplore.ieee.org/document/7307132/
  4. The IEEE RAS TC Pebras website: http://www.ieee-ras.org/performance-evaluation
  5. The Euron GEM Sig website page still listing related events http://www.heronrobots.com/EuronGEMSig/gem-sig-events


tags: , ,


Fabio Bonsignorio is a professor in the BioRobotics Institute at the Scuola Superiore Sant'Anna (Pisa, Italy).
Fabio Bonsignorio is a professor in the BioRobotics Institute at the Scuola Superiore Sant'Anna (Pisa, Italy).





Related posts :



How robots learn to hike

A new control approach that enables a legged robot, called ANYmal, to move quickly and robustly over difficult terrain.
20 January 2022, by

How robots and bubbles could soon help clean up underwater litter

Everyone loves to visit the seaside, whether to enjoy the physical benefits of an exhilarating swim or simply to relax on the beach and catch some sun. But these simple life affirming pleasures are easily ruined by the presence of litter, which if persistent can have a serious negative impact on both the local environment and economy. However, help is at hand to ensure the pristine nature of our coastlines.
19 January 2022, by

Maria Gini wins the 2022 ACM/SIGAI Autonomous Agents Research Award

Congratulations to Maria Gini on winning this prestigious award, recognising her research and leadership in the field of robotics and multi-agent systems.
18 January 2022, by

UN fails to agree on ‘killer robot’ ban as nations pour billions into autonomous weapons research

Given the pace of research and development in autonomous weapons, the U.N. meeting might have been the last chance to head off an arms race.
16 January 2022, by

Science Magazine robot videos 2021

A compilation of Science Magazine videos featuring robotics research that were released during last year.
14 January 2022, by

CBQ: Commercial-grade Autonomous Mowers, Safety, and Dogfooding | Sense Think Act Podcast #11

In this episode, Audrow Nash speaks to Charles Brian Quinn (aka, CBQ), CEO and a Co-Founder of Greenzie. Greenzie make an autonomous driving system for commercial lawn mowers. We talk about Greenzie's...
11 January 2022, by and





©2021 - ROBOTS Association


 












©2021 - ROBOTS Association