Robohub.org
 

AI can be a powerful tool for scientists. But it can also fuel research misconduct


by
31 March 2025



share this:

An Escher-like structure depicting the concept of AI model collapse. The image features a swirling, labyrinthine design, representing a recursive loop where algorithms feed on their own generated synthetic data. Elements of digital clutter and noise are interwoven throughout, highlighting the chaotic nature of the internet increasingly populated by AI-generated content. The visual metaphor of a Uroboros, a snake eating its own tail, symbolizes the self-referential cycle of AI training on its own outputs.Nadia Piet & Archival Images of AI + AIxDESIGN / Model Collapse / Licenced by CC-BY 4.0

By Jon Whittle, CSIRO and Stefan Harrer, CSIRO

In February this year, Google announced it was launching “a new AI system for scientists”. It said this system was a collaborative tool designed to help scientists “in creating novel hypotheses and research plans”.

It’s too early to tell just how useful this particular tool will be to scientists. But what is clear is that artificial intelligence (AI) more generally is already transforming science.

Last year for example, computer scientists won the Nobel Prize for Chemistry for developing an AI model to predict the shape of every protein known to mankind. Chair of the Nobel Committee, Heiner Linke, described the AI system as the achievement of a “50-year-old dream” that solved a notoriously difficult problem eluding scientists since the 1970s.

But while AI is allowing scientists to make technological breakthroughs that are otherwise decades away or out of reach entirely, there’s also a darker side to the use of AI in science: scientific misconduct is on the rise.

AI makes it easy to fabricate research

Academic papers can be retracted if their data or findings are found to no longer valid. This can happen because of data fabrication, plagiarism or human error.

Paper retractions are increasing exponentially, passing 10,000 in 2023. These retracted papers were cited over 35,000 times.

One study found 8% of Dutch scientists admitted to serious research fraud, double the rate previously reported. Biomedical paper retractions have quadrupled in the past 20 years, the majority due to misconduct.

AI has the potential to make this problem even worse.

For example, the availability and increasing capability of generative AI programs such as ChatGPT makes it easy to fabricate research.

This was clearly demonstrated by two researchers who used AI to generate 288 complete fake academic finance papers predicting stock returns.

While this was an experiment to show what’s possible, it’s not hard to imagine how the technology could be used to generate fictitious clinical trial data, modify gene editing experimental data to conceal adverse results or for other malicious purposes.

Fake references and fabricated data

There are already many reported cases of AI-generated papers passing peer-review and reaching publication – only to be retracted later on the grounds of undisclosed use of AI, some including serious flaws such as fake references and purposely fabricated data.

Some researchers are also using AI to review their peers’ work. Peer review of scientific papers is one of the fundamentals of scientific integrity. But it’s also incredibly time-consuming, with some scientists devoting hundreds of hours a year of unpaid labour. A Stanford-led study found that up to 17% of peer reviews for top AI conferences were written at least in part by AI.

In the extreme case, AI may end up writing research papers, which are then reviewed by another AI.

This risk is worsening the already problematic trend of an exponential increase in scientific publishing, while the average amount of genuinely new and interesting material in each paper has been declining.

AI can also lead to unintentional fabrication of scientific results.

A well-known problem of generative AI systems is when they make up an answer rather than saying they don’t know. This is known as “hallucination”.

We don’t know the extent to which AI hallucinations end up as errors in scientific papers. But a recent study on computer programming found that 52% of AI-generated answers to coding questions contained errors, and human oversight failed to correct them 39% of the time.

Maximising the benefits, minimising the risks

Despite these worrying developments, we shouldn’t get carried away and discourage or even chastise the use of AI by scientists.

AI offers significant benefits to science. Researchers have used specialised AI models to solve scientific problems for many years. And generative AI models such as ChatGPT offer the promise of general-purpose AI scientific assistants that can carry out a range of tasks, working collaboratively with the scientist.

These AI models can be powerful lab assistants. For example, researchers at CSIRO are already developing AI lab robots that scientists can speak with and instruct like a human assistant to automate repetitive tasks.

A disruptive new technology will always have benefits and drawbacks. The challenge of the science community is to put appropriate policies and guardrails in place to ensure we maximise the benefits and minimise the risks.

AI’s potential to change the world of science and to help science make the world a better place is already proven. We now have a choice.

Do we embrace AI by advocating for and developing an AI code of conduct that enforces ethical and responsible use of AI in science? Or do we take a backseat and let a relatively small number of rogue actors discredit our fields and make us miss the opportunity?The Conversation

Jon Whittle, Director, Data61, CSIRO and Stefan Harrer, Director, AI for Science, CSIRO

This article is republished from The Conversation under a Creative Commons license. Read the original article.




The Conversation is an independent source of news and views, sourced from the academic and research community and delivered direct to the public.
The Conversation is an independent source of news and views, sourced from the academic and research community and delivered direct to the public.





Related posts :



#ICRA2025 social media round-up

  23 May 2025
Find out what the participants got up to at the International Conference on Robotics & Automation.

Robot Talk Episode 122 – Bio-inspired flying robots, with Jane Pauline Ramos Ramirez

  23 May 2025
In the latest episode of the Robot Talk podcast, Claire chatted to Jane Pauline Ramos Ramirez from Delft University of Technology about drones that can move on land and in the air.

Robot Talk Episode 121 – Adaptable robots for the home, with Lerrel Pinto

  16 May 2025
In the latest episode of the Robot Talk podcast, Claire chatted to Lerrel Pinto from New York University about using machine learning to train robots to adapt to new environments.

What’s coming up at #ICRA2025?

  16 May 2025
Find out what's in store at the IEEE International Conference on Robotics & Automation, which will take place from 19-23 May.

Robot see, robot do: System learns after watching how-tos

  14 May 2025
Researchers have developed a new robotic framework that allows robots to learn tasks by watching a how-to video

AI-powered robots help tackle Europe’s growing e-waste problem

  12 May 2025
EU-funded researchers have developed adaptable robots that could transform the way we recycle electronic waste, benefiting both the environment and the economy.

Robot Talk Episode 120 – Evolving robots to explore other planets, with Emma Hart

  09 May 2025
In the latest episode of the Robot Talk podcast, Claire chatted to Emma Hart from Edinburgh Napier University about algorithms that 'evolve' better robot designs and control systems.

Robot Talk Episode 119 – Robotics for small manufacturers, with Will Kinghorn

  02 May 2025
In the latest episode of the Robot Talk podcast, Claire chatted to Will Kinghorn from Made Smarter about how to increase adoption of new tech by small manufacturers.



 

Robohub is supported by:




Would you like to learn how to tell impactful stories about your robot or AI system?


scicomm
training the next generation of science communicators in robotics & AI


©2025.05 - Association for the Understanding of Artificial Intelligence


 












©2025.05 - Association for the Understanding of Artificial Intelligence