Robohub.org
Surviving the paper deluge: a one-year study in learning from demonstration
By Emmet Cole
With the explosion of robotics research, staying current in fields like Learning from Demonstration (LfD) is a monumental challenge. Is AI the solution to the “paper deluge,” or is it part of the problem? Read the article preview below to learn more!
Download the full paper: Surviving the Paper Deluge.
Authors: Aude Billard, Renaud Detry, Nadia Figueroa, Maximilian Foriest, Dongheui Lee, Kunpeng Yao
Contributions: The five senior authors (A.B, R.D, N.F, D.L and K.Yao) collectively designed the study, read the papers, conducted the qualitative and quantitative analysis and writing of the paper. M. F. contributed scripts for LLM analysis and participated in LLM-Human comparison.
Summary
Scientists are expected to read newly published papers in their field to stay current and keep their work relevant. However, when faced with the massive number of publications, it may seem an overwhelming task to read all these papers, even if one were to reduce this to only a fraction related to one’s own area of research. As an example, in 2024 alone, IEEE published no less than 46,968 papers on “robotics” or “automation”, and IEEE publications represent only a fraction of the total research available online
To assess the magnitude of this challenge, as well as to evaluate how much genuine progress is reported in today’s publications, we undertook exactly this effort. For the task to be reasonable, we reduced our search to one particular subarea, learning from demonstration (LfD), that is methods whereby robots are taught by human experts. We monitor progress through both quantitative and qualitative metrics, offering a review on current trends and notable contributions. We also delineate areas of importance, but that seem to receive little attention and offer recommendations for promoting.
Our assessment was primarily based both on a human-eye assessment of all papers. We also explored the use of AI and other computing tools to do this task in our place. While scripts and large language models (LLMs) can be used fairly faithfully to provide general quantitative assessment, they fail when it comes to assessing the true importance of the research. They cannot recognize a paper revisiting a work that already had solutions. They fail to recognize when the abstract or claims of the paper are overstatements over the true contribution reported in the paper.
Our overall assessment led us to conclude that from a deck of more than 300 papers, only about 20% of the papers could be qualified as offering highly notable contributions, while the remainder of the papers offered a variety of incremental improvements over existing methods, or new domains of applications. The notable contributions did not correlate necessarily with a higher number of downloads or citations. Finding these gems is, however, essential to reduce the risk that novel work goes unnoticed and reduce duplication of efforts. We offer a few thoughts on how to best combine direct reading of the literature with automated approaches (scripts and LLMs) to streamline the review process. We close with a few recommendations: a) develop a research engine that restores the natural importance of work done by journal and conference editorial boards to rank papers based on evaluation scores and peer-reviewed status, in place of Google Scholar or IEEEXplore, that place all publications on equal footing, disregarding peer reviewing and the reputation of journals and conferences, b) consider establishing a blind publication model and topic-based social media posting, where authors’ name and institution are downplayed and become accessory to the paper to ensure that focus be on the content of the publication rather than secondary aspects, c) take a holistic approach to use of LLM in support of reviewing literature, using them for what they excel at, namely summarizing a piece of work and collecting precise quantitative information, but bearing in mind that, while today the tools cannot match expert capacity to assess true novelty, should they achieve this one day, this may have repercussion on our own ability to provide said expertise.
Publications growth
Over the past decade, the number of submissions to robotics journals has grown steadily on a yearly basis, with an explosive trend in 2023 (26%) and 2024 (31%), likely due to different factors, including growing interest in the public and private sectors and to the availability of AI tools supporting the writing of papers and code. The number of published papers has closely followed this trend, despite all efforts made by editorial boards to contain the growth by decreasing acceptance rates. Conferences have followed the same trend. For instance, ICRA doubled the number of papers it published in ten years, reaching approximately 1,800 in 2024. Simultaneously, the strong pressure exerted by the community to publish rapidly has led to a 50% decrease in the time window between the submission of a paper and its publication. The phenomenon is not particular to IEEE publications, and journals and conferences such as IJRR, RSS and CoRL have followed the same trend.
Clearly, it would be unrealistic to expect any researcher to read all of these publications. One might argue that researchers are typically interested in only a subset of the literature, for instance a specific domain or methodology, and would therefore read only a fraction of all published papers. Yet even this narrower scope may prove unmanageable. To assess how feasible it is for a researcher to stay current within their own area of expertise, we undertook the task of reading a large fraction of all papers published in our domain – learning from demonstration – over the course of a single year (2024).
This article originally appeared on IEEE RAS.
tags: IEEE RAS
AUAI is supported by:
