Home
high-impact articles

Better metrics for high-impact research

Summary

This brief seeks to provide alternative metrics for identifying “high-impact” research articles. Providing a broader context for the metrics used in research evaluation will more accurately and holistically demonstrate the impact of IUPUI faculty publications. This is particularly important in the Promotion & Tenure process, as well as in hiring and award processes.

The most well-known metric used in evaluating publications is the Journal Impact Factor (JIF). The JIF is a publication metric reflecting the annual average (mean) number of citations to articles published in a particular journal. The JIF is a measure about a journal, not the articles published within the journal. Counter to common perception, the 2018 JIF reflects citations to articles published in 2016-2017. The 2018 JIF does not speak to the quality or to influence of the articles published in 2018.

A graphic portraying how the Journal Impact Factor is calculated.

Problem Statement

The following limitations documented in the bibliometric literature make the Journal Impact Factor the wrong metric for evaluating individuals for promotion and tenure, conducting search committees, and scoring grants.

The data are skewed: Only one-third of articles in most journals receive citations that equal or exceed the JIF value^[i],[ii].
The timeframe is too short: As a measure of impact on the scientific community, the timeframe for calculating the JIF (2 years) is a poor reflection of long-term citation rates. It takes papers in Biomedical Research 8 years to accumulate 50% of their citations, while it takes 9 years for Physics papers, 13 years for Psychology papers, and 14 years for Social Science papers^[iii].
Journal citations are not a measure of research rigor: The JIF is not a measure of quality, for the journal or the articles published in it. Peer review practices vary widely and are often insufficient to detect and address issues related to increasing complexity of study design and statistical analyses as well as image duplication or manipulation.
The source data excludes too many sources of valuable science: Many fields and non-English language journals are underrepresented or not represented at all in the data used to calculate the JIF^[iii].

Furthermore, using the JIF can cause damage to researchers, the institution itself, and its funders by:

Delaying the publication of science: the use of the JIF as an indicator of high-impact research encourages submissions to journals with higher JIFs that may not be the most appropriate venue for submission^[iii].
Increasing risks of retraction: Correlative analysis shows that several journals with very high JIF also have higher rates of retraction^[iii].
Discouraging innovation and research diversity: The JIF disadvantages scholars in fields with lower research volume and fields with publishing practices that do not reflect the inflexible criteria (such as a high volume of articles per volume) for inclusion in the citation databases provided by Clarivate Analytics (Web of Science) and Elsevier (Scopus). When institutions use the JIF as an indicator of high-impact research, they may be overlooking scholars publishing in journals that are not indexed in these citation databases.
Contributing to monopolistic pricing: The overwhelming majority of journals with highly ranked JIFs are owned by five publishers^[iv]. As universities incentivize publication in these journals, the publishers will continue to impose constant and unsustainable price increases. The scientific, technical, and medical publishing market in 2017 was estimated at $25.7 billion. For 2016, it was estimated that libraries alone spent more than $8.1 billion ^[v]. These costs redirect dollars from the university’s research infrastructure and support programs to a small group of large, for-profit publishing companies.

Recommended Alternatives to the Journal Impact Factor

Though alternatives to the Journal Impact Factor have been developed (e.g., Eigenfactor Score, CiteScore, Scimago Journal Rank), none have been widely adopted. While each of these was developed to address specific concerns about the JIF, none address the underlying issues regarding the data used to calculate the metrics. The publication indexes on which these metrics are based are biased towards English-language publications and high-volume fields of research. Inclusion in these indexes does not reflect journal quality. Furthermore, the majority of citations continue to be accrued by a small fraction of published articles (i.e., Matthew Effect).

As publishing has transitioned to a networked environment, it is now possible to capture much more information about how scholars access and interact with research articles. To improve evaluation of scholars and their articles at IUPUI, we propose the use of article metrics to broaden the conversation about “high impact” research. Two types of article metrics, already used by some research areas and campus departments, could be readily adopted:

Normalized article citation metrics: These metrics, such as the Field Citation Ratio, Field Weighted Citation Impact, and Relative Citation Ratio, are metrics about an article. The normalization process corrects two main problems with the JIF. The normalized citation metric corrects for 1) the publication date; and, 2) disciplinary differences in the rate of publishing and citation.
Article usage metrics: Article downloads are another research metric, or indicator, that has shown potential relevance for gauging use and later citations. The benefit of using downloads is that they accrue in real-time, as users download the article for use. Particularly for recently published articles (less than 12 months since publication), article downloads may be the most salient indicator of interest and potential use.

When discontinuing use of the Journal Impact Factor is not immediately feasible, we recommend the following:

Contextualize the JIF: When using the JIF, present the JIF value along with the relevant research area, ranking in the assigned research area, and quartile ranking or percentile in the assigned research area. Some journals are assigned to multiple research areas; in that case, choose the research area most relevant to the evaluation task.
- For example: The 2017 Journal Impact Factor for Virulence is 4.0, which puts the journal at a ranking of 9 of 93, or the 91^st percentile, for journals in the research area “Infectious Diseases”.
Supplement the JIF: To provide a more complete picture of scholarly impact, use normalized article citation metrics and article usage metrics along with the contextualized JIF.

When discontinuing use of the JIF is possible, use article metrics.

In some fields, the JIF is simply not available or does not apply. Article metrics and qualitative reviews are better options for evaluating individual products and scholars. In addition to the field normalized citation scores and other article metrics described above, evaluators should consult with librarians to identify other approaches to gathering evidence of research impact. Qualitative approaches, testimonies from known experts in the field or comments from community stakeholders should not be overlooked.

Next Steps

Thank you for taking the time to review this information. We invite you to reach out to the IUPUI Research Metrics Services team with any questions, concerns, or to discuss further.

Download a PDF of this page to share with your colleagues.

Footnotes

[i] Larivière, V., & Sugimoto, C. (2019). The Journal Impact Factor: A brief history, critique, and discussion of adverse effects. In Springer Handbook of Science and Technology Indicators. Retrieved from https://arxiv.org/abs/1801.08992

[ii] Callaway, E. (2016). Beat it, impact factor! Publishing elite turns against controversial metric. Nature News, 535(7611), 210. https://doi.org/10.1038/nature.2016.20224

[iii] Fang, F. C., & Casadevall, A. (2011). Retracted science and the retraction index. https://iai.asm.org/content/iai/79/10/3855.full.pdf

[iv] Larivière, V., Haustein, S., & Mongeon, P. (2015). The Oligopoly of Academic Publishers in the Digital Era. PLOS ONE, 10(6), e0127502. https://doi.org/10.1371/journal.pone.0127502

[v] Johnson, R., Watkinson, A., & Wabe, M. (2018). The STM Report: An overview of scientific and scholarly publishing. International Association of Scientific, Technical and Medical Publishers. https://www.stm-assoc.org/2018_10_04_STM_Report_2018.pdf