Impact Factor and the Future of Medical Journals

Some research publications are getting away from flawed measures of influence that make it easy to game the system.

Medical publishing, one of the oldest continuous scientific traditions, is growing exponentially. New medical journals dedicated to increasingly niche microcosms are being created every day. Yet medical publishing finds itself at a vital cross roads. This year's Nobel Prize winner in physiology, Randy Scheckman, announced his decision to boycott the three major “luxury” journals: Science, Nature, and Cell. Criticism of researchers' inability to replicate major medical findings is also gaining traction beyond scientific circles.

For an industry based on advancement, medical journals are very rigid. Take for example the impact factor, defined as the number of citations divided by the number of papers published in the journal, which is a measure to convey the influence of journals and the research they carry. Even in medical school, my friends would compare the impact factor of the journals they had published in, as though that represented the quality of the work. It is akin to what the “mega-pixel” is for digital cameras, a pithy misleading sound bite that has become an industry gold standard.

While the impact factor was originally devised in the 1970's by librarians to help them make rational choices about what journals to subscribe to, the current use is very different from that original intent. Impact factors are frequently considered for academic promotions, pressuring academics to keep targeting higher impact journals. In fact, scientists in China are paid based on the impact factor of the journals they publish in.

Journals employ several strategies to artificially raise the impact factor, which initiates a positive feedback loop by incentivizing more scientists to submit to them. Some editors have been caught trying to induce authors to increase the number of citations from their journal to further raise the impact factor. One investigation found that a collaboration of Brazilian journals had agreed to highly cite each other’s articles to fraudulently raise each of the journals’ impact factors.

Critics argue that citations are barely a reflection of the quality of the research and that the impact factor is easily manipulated by taking articles that are more likely to be highly cited or ‘sexy’ than highly meaningful. Reviews of common topics can contain hundreds of references (and authors have been known to cram them with self citations) and are more highly cited than original research articles that may eventually lead to a new discovery or finding. Also, research in controversial areas, such as a study published in Science that demonstrated brain damage by the drug Ecstasy, or the infamous paper published in the Lancet which purported to show a link between vaccines and autism, are more likely to be highly cited because of interest generated in these subjects (both studies were subsequently retracted).

Medical publishing’s growth is actually one of its greatest challenges. With more than half a million papers published annually in biomedicine alone, it is no wonder that physicians are increasingly feeling overwhelmed with the avalanche of information they find themselves facing. The new hubs of publishing are India and China, where new journals are being formed almost daily. My inbox usually receives several invitations a day to join editorial boards of journals I've never heard of, completely unrelated to my specialty.

And all of these journals need to be concerned with maintaining accuracy. The current system of peer-review, which originated in the 18th century, is now stressed. After a journal decides to consider a paper, it is sent to unpaid, anonymous reviewers in the same specialty to review. Physicians are busy and it's difficult for them to be able to effectively analyze a paper for accuracy. The only real upside for the reviewer is the synthetic prestige that comes with being an anonymous reviewer, which may not be enough to muster a consistent and thorough effort. The flaws of the current system were exposed in a sting by a reporter for Science who had a fraudulent and comically bad paper, from a fake alias and institution, accepted in the vast majority of journals he had submitted to without any meaningful review.

In spite of their strong ties to tradition, journal editors are finally responding to the changing ecology. When I asked Howard Bauchner, editor-in-chief of the Journal of the American Medical Association, what the future of the medical journal was, he summed it up in just one word: “Digital.” With a website curated with author video interviews, twitter feeds and podcasts, JAMA addresses one of the medium’s biggest drawbacks—passivity. “Medical journals have traditionally been passive; readers read articles and experts go on to discuss them. The digital format opens space for debate that journals haven’t previously explored,” Bauchner told me.

Increasingly, journal editors are also recognizing the limitation of the impact factor. One of those voices is that of Dr Joseph Loscazlo, editor-in-chief of the American Heart Association’s journal Circulation, the highest impact journal for cardiovascular disease. In an editorial published in 2011, he wrote that relying on measures that only take citations into account “defies logic.”

Some journals, however, are starting to take more innovative approaches. One such journal is PLOS One, which provides individual article metrics to anyone who accesses the article. For a paper I published on childhood obesity in 2009, the journal not only reported citations, but showed the number of times it has been viewed and downloaded, and compared it to other papers published in that area. Instead of letting the reputation of the journal decide the impact of its papers, PLOS One provides information about the influence of the article on a more granular level. Other novel measures of impact including the h index (a metric that indicates how many papers with a minimum number of citations have been published by an individual author) continue to depend on citations as a surrogate for impact. In truth, history is the best judge of the impact of any research, and scientists will have to apply the same scientific standard to the measures of impact, which they apply to their research, to devise a measure that captures the quality and impact of investigative work.

To Loscalzo, the future of medical publishing is a democratic one. When I met him recently, he prophesized that “the medical journal will evolve to eliminate many of the artificial distinctions that currently exist between journals today. Smart software will decide based on largely open access journals which papers will be of most interest to a particular reader.” Such a system, he hopes, would ease the burden of information overload for individual physicians, by bringing the most relevant research straight to them. Regarding peer review, while he didn’t see the principle changing, he felt that there was room for change in the current paradigm. “Over time there may be more online reviews with no anonymity,” he said, a trend pioneered by Biology Direct, a journal that provides open peer review that is available for readers to read along with the article, with or without changes suggested by the reviewers.

Like a cancer that outgrows its blood supply, the tremendous growth of medical journals is likely to plateau. Yet for all their growth, medical journals are starting to go back to the lab and focus on the essentials of science: accuracy, replicability and generalizability—as well as equity and transparency—which will help them continue to command influence and respect.