乌鸦传媒

Practitioners are desperate for reliable and actionable information on COVID-19. They want to learn how they can protect patients and themselves, and they want to understand how the diverse and complex COVID-19-related clinical syndromes can be effectively treated. How can healthcare providers adopt best practices for COVID-19?

For decades, practitioners have relied on data and knowledge published in the peer reviewed medical literature. But the peer review process is not designed for the rapid dissemination of critical information during a time of crisis. Peer review is a deliberate, slow and imperfect process.

Typically, a paper that is submitted to a journal is vetted to determine if it meets formalistic technical requirements, and then, it is sent out to two to five experts, who have a few weeks to provide comments. Their feedback allows the editors to make a preliminary decision about the paper's potential value. Authors may be given a chance to revise the paper, and the revisions are often reviewed again by the external experts. Under optimal circumstances, the peer review process takes 4-6 weeks. But all too often, the process is tortured; it can take 4-6 months -- or much longer.

Such delays are not acceptable under noncrisis conditions, but they are morally reprehensible during a pandemic. In a matter of a few months, COVID-19 has afflicted over 4 million people and caused nearly 300,000 deaths. The current peer review process does not work effectively under such cataclysmic circumstances.

Accordingly, authors have turned to preprint servers to disseminate their work. The , which is sponsored by Yale University and led by Harlan Krumholz. It is part of the YODA project; YODA stands for Yale University Open Data Access. Harlan is a dear friend, who has been a champion of rapid data dissemination and data transparency for years. This project is his brainchild, and now, in a time of crisis, it is displaying its strengths.

Harlan is a former journal editor, who has written eloquently about the severe limitations of peer review journals. In his famed 2015 paper, he compellingly argued that journals no longer play an important role in the dissemination of medical information, because they are slow, expensive, limited, unreliable, focused on the wrong metric, powerful, parochial, static, and dependent on a flawed business model.

To rectify these deficiencies, Harlan (and others) launched medRxiv, and it has been the go-to source for new information on COVID-19. Hundreds, perhaps thousands, of papers have been posted, and many have been widely read and cited on social media, on medical news sites, and in news reports. They are vigorously discussed and fiercely debated. Little of this debate is taking place in the peer reviewed medical literature.

It is understood that the information posted on preprint servers may or may not be reliable. And even reliable data may be intentionally or unintentionally misunderstood by readers or distorted by those who post comments on social media, blogs and news reports.

Therefore, when data are sourced from papers posted on medRxiv, readers are routinely told that the data has "not undergone peer review." This standard disclaimer is provided as a "cautionary note," suggesting (perhaps) that the information should be taken with a grain of salt.

Is this caution taken seriously? I doubt it.

So is the dissemination of non-peer reviewed information problematic? It would be if we knew that the peer review process represented a reliable mechanism of quality control. The lack of peer review would mean that we could not be sure that the basic standards of scientific conduct and integrity had been upheld.

However, it is not clear that peer review was an effective quality-control mechanism BEFORE the time of COVID-19. For many years, many of us recognized that the peer review process was uneven at best. An author's ability to have his/her work published in any given journal often represented the luck of the draw, being highly dependent on the preferences of the handling editor and the choice of reviewers. Top-tier journals published awful papers and rejected good ones. Authors understood the fickleness of the process, but we did little to fix the problem. If a journal rejected a paper, the easiest path was to simply move onto a different journal.

If an author was persistent, there was an absolute certainty that their work would be eventually published somewhere, regardless of its quality. Many fee-based journals will publish any work as being "peer reviewed" as long as authors are willing to pay.

As a result, even before COVID-19, declaring that a study was published in a peer review journal was no longer an effective identifier of quality work. Of course, publishing work in a top-tier journal still had meaning, but all too often, only authors appreciated the distinction of presenting their findings in a first-rate venue.

Did papers published in top tier peer reviewed journals get more attention and accolades than those published on preprint servers without peer review? I do not know. Many papers go viral within days of being posted on preprint servers, and many excellent papers published in first-rate journals ."

So journal editors were in the midst of an existential crisis before COVID-19. Now, as the pandemic has grown, their crisis has worsened considerably.

Great editors have stepped up to meet the challenge. Many have made every effort to publish important findings on COVID-19 as their highest priority. Critical papers related to COVID-19 are granted an expedited process, which means that a review and decision is often made in days rather than months.

Yet, in many cases, quality thresholds have been relaxed. Normally, top-tier journals would prioritize studies if they had meaningful numbers of patients, included adequate controls, relied on clinically important metrics, and performed analyses that used rigorous methods. Now, the same journals often rush papers with serious methodological limitations through the process. The papers were still peer reviewed, but reviewers have been instructed to make their decisions quickly, and many reviewers sense that (at least with respect to COVID-19 papers) they should not be too picky.

Not surprisingly, some authors -- sensing the vulnerability of journal editors -- have seized the opportunity. They reasoned that some journals were prepared to publish work, even if it were second-rate, as long as it was related to COVID-19. All they needed to do was put COVID-19 in the title and ask for an expedited review.

The peer review process was not designed to function properly during such a period of chaos. It is not surprising that editors have made mistakes.

On March 20, the first paper describing the use of hydroxychloroquine and azithromycin was . The journal is the official journal of the International Society of Antimicrobial Chemotherapy. On April 3, two weeks after the publication, that same society indicating that the study did not meet its expected standard. So according to the society, the peer review process for that paper failed.

It is really hard for the peer review process to work during a time of chaos, especially since it did not work well before COVID-19. Accordingly, some authors have given up. If peer review is broken, why not simply submit your work to medRxiv, and hope that the news goes directly to Twitter? It takes far less time and effort, and the impact is considerable. (Some may think medRxiv was intended to precede, not replace, peer review, but its de facto function is rapidly evolving.)

Is the current state of affairs driving some journal editors to tears? Some wonder how they are going to fulfill their mission during the current crisis.

After all, what would you do if you were a journal editor, and you were receiving 20-40 full-length papers on COVID-19 every day? During the past several months, some journals have each received 500-1,000 papers on COVID-19. Someone needs to read these manuscripts, and in many cases, external reviewers need to judge which are worthwhile and which have little value. But no journal can do justice to such a deluge of material. Their bandwidth is too narrow, and the error rate is too high.

Please do not get me wrong. I am not denigrating the importance of peer review in principle. Journal editors are struggling to do their best, but they had an impossible job before COVID-19. The pandemic made things much more difficult.

Some of my colleagues contend that the war is already lost. Many physicians have stopped reading journals altogether. They no longer have the time, and many do not have the ability or the interest to decide what information is trustworthy, replicable and actionable.

In a previous blog, I predicted that the pandemic would greatly accelerate changes that were already taking place before COVID-19, and these changes would affect every aspect of our lives. This principle certainly applies to the dissemination of medical information.

Editors must now find a way to transform their journals into nimble and trustworthy vehicles for the publication of reliable and actionable medical data. The process of peer review must be made far more rapid, and it must be stripped of its current pettiness and fickleness. At the same time, peer review must maintain the highest possible standards, and it must be relevant; otherwise, it serves no useful purpose.

In the end, I imagine that only a handful of top-tier journals will survive this existential threat. Yet, their survival is critically important, because -- now, more than ever -- we need them to function as critical gatekeepers of scientific communication for the medical community, our governmental leaders and the general public.

Harlan Krumholz was right. For many and perhaps for most journals, this will be The End of Journals. As will be so often the case, COVID-19 just brought the future into the present.

Comment