Science journal retracts peer-reviewed article containing AI generated ‘nonsensical’ images

The paper is authored by Xinyu Guo and Dingjun Hao of the Department of Spine Surgery, Hong Hui Hospital at Xi’an Jiaotong University; as well as Liang Dong of the Department of Spine Surgery, Xi’an Honghui Hospital in Xi’an, China.

It was reviewed by Binsila B. Krishnan of the National Institute of Animal Nutrition and Physiology (ICAR) in India and Jingbo Dai of Northwestern Medicine in the United States, and edited by Arumugam Kumaresan at the National Dairy Research Institute (ICAR) in India.

VentureBeat reached out to all the authors and editors of the paper, as well as Amanda Gay Fisher, the journal’s Field Chief Editor, and a professor of biochemistry at the prestigious Oxford University in the UK, to ask further questions about how the article was published, and will update when we hear back.

Troubling wider implications for AI’s impact on science, research, and medicine

AI has been touted as a valuable tool for advancing scientific research and discovery by some of its makers, including Google with its AlphaFold protein structure predictor and materials science AI GNoME, recently covered positively by the press (including VentureBeat) for discovering 2 million new materials.

However, those tools are focused on the research side. When it comes to publishing that research, it is clear that AI image generators could pose a major threat to scientific accuracy, especially if researchers are using them indiscriminately, to cut corners and publish faster, or because they are malicious or simply don’t care.

The move to use AI to create scientific illustrations or diagrams is troubling because it undermines the accuracy and trust among the scientific community and wider public that the work going into important fields that impact our lives and health — such as medicine and biology — is accurate, safe, and screened.

Yet it may also be the product of the wider “publish or perish” climate that has arisen in science over the last several decades, in which researchers have attested they feel the need to rush out papers of little value in order to show they are contributing something, anything, to their field, and bolster the number of citations attributed to them by others, padding their resumes for future jobs.

But also, let’s be honest — some of these researchers on this paper work in spine surgery at a human hospital: would you trust them to operate on your spine or help with your back health?

And with more than 114,000 citations to its name, the journal Frontiers in Cell and Developmental Biology has now had its integrity of all of them called into question by this lapse: how many more papers published by it have AI-illustrated diagrams that have slipped through the review process?

Intriguingly, Frontiers in Cell and Developmental Biology is part of the wider Frontiers company of more than 230 different scientific publications founded in 2007 by neuroscientists Kamila Markram and Henry Markram , the former of whom is still listed as CEO.

The company says its “vision [is] to make science open, peer-review rigorous, transparent, and efficient and harness the power of technology to truly serve researchers’ needs,” and in fact, some of the tech it uses is AI for peer review.

As Frontiers proclaimed in a 2020 press release:

“In an industry first, Artificial Intelligence (AI) is being deployed to help review research papers and assist in the peer-review process. The state-of-the-art Artificial Intelligence Review Assistant (AIRA), developed by open-access publisher Frontiers, helps editors, reviewers and authors evaluate the quality of manuscripts. AIRA reads each paper and can currently make up to 20 recommendations in just seconds, including the assessment of language quality, the integrity of the figures, the detection of plagiarism, as well as identifying potential conflicts of interest.“

The company’s website notes AIRA debuted in 2018 as “The next generation of peer review in which AI and machine learning enable more rigorous quality control and efficiency in the peer review.”

And just last summer, an article and video featuring Mirjam Eckert, chief publishing officer at Frontiers, stated:

“At Frontiers, we apply AI to help build that trust. Our Artificial Intelligence Review Assistant (AIRA) verifies that scientific knowledge is accurately and honestly presented even before our people decide whether to review, endorse, or publish the research paper that contains it.

AIRA reads every research manuscript we receive and makes up to 20 checks a second. These checks cover, among other things, language quality, the integrity of figures and images, plagiarism, and conflicts of interest. The results give editors and reviewers another perspective as they decide whether to put a research paper through our rigorous and transparent peer review.“

Frontiers has also received favorably coverage of its AI article review assistant AIRA in such notable publications as The New York Times and Financial Times.

Clearly, the tool wasn’t able to effectively catch these nonsensical images in the article, leading to its retraction (if it was used at all in this case). But it also raises questions about the ability of such AI tools to detect, flag, and ultimately stop the publication of inaccurate scientific information — and the growing prevalence of its use at Frontiers and elsewhere across the publishing ecosystem. Perhaps that is the danger of being on the “frontier” of a new technology movement such as AI — the risk of it going wrong is higher than with the “tried and true,” human-only or analog approach.

VentureBeat also relies on AI tools for image generation and some text, but all articles are reviewed by human journalists prior to publication. AI was not used by VentureBeat in the writing, reporting, illustrating or publishing of this article.