Defining, Measuring, and Rewarding Scholarly Impact: Mind the Level of Analysis

We address the grossly incorrect inferences that result from using journal impact factor (JIF) as a proxy to assess individual researcher and article scholarly impact. This invalid practice occurs because of confusion about the definition and measurement of impact at different levels of analysis. Specifically, JIF is a journal-level measure of impact, computed by aggregating citations of individual articles (i.e., upward effect), and is therefore inappropriate when measuring impact at lower levels of analysis, such as that of individual researchers, or of individual articles published in a particular journal (i.e., downward effect). We illustrate the severity of the errors that occur when using JIF to evaluate individual scholarly impact, and advocate for an immediate moratorium on the exclusive use of JIF and other journal-level (i.e., higher level of analysis) measures when assessing the impact of individual researchers and individual articles (i.e., lower level of analysis). Given the importance and interest in assessing the scholarly impact of researchers and articles, we delineate level-appropriate and readily available measures. We discuss implications for the careers of researchers and educators, the administration and future of business schools, and provide recommendations regarding the assessment of scholarly impact.

Clearly, the evaluation of individual articles and individual researchers based on JIF is far from a mere labeling exercise. For researchers, evaluating their impact based on JIF and JIF-influenced metrics affects critical career outcomes including securing a tenure-track job, enjoying a teaching reduction to devote more time to research, obtaining additional funding (e.g., summer support, research accounts, cash bonuses), receiving a positive or negative P&T review decision, and attaining a chaired position (Abritis, McCook, & Watch, 2017; Edwards & Roy, 2017). For business schools, using JIF and JIF-influenced metrics to classify articles published by their faculty influences important outcomes such as business school rankings, fundraising, media attention, faculty recruitment efforts, and student enrollment (Aguinis, Cummings, Ramani, & Cummings, 2020; Morgeson & Nahrgang, 2008; Ryazanova, McNamara, & Aguinis, 2017).

Our article therefore is about JIF, and the grossly incorrect inferential leap that occurs when JIF is used to assess the impact of individual articles published in that journal, and the impact of the individual researchers who have published articles in that journal. This mistaken practice is due to confusion about the definition and measurement of impact at different levels of analysis. Specifically, JIF is a journal-level measure of impact computed by aggregating citations of individual articles (i.e., upward effect), and is therefore not appropriate for measuring impact at lower levels of analysis, such as that of individual researchers and of individual articles published in a particular journal (i.e., downward effect). Accordingly, we advocate for an immediate moratorium on the exclusive use of JIF and other journal-level (i.e., higher level of analysis) measures to assess the impact of individual researchers and individual articles (i.e., lower level of analysis). Furthermore, we propose that this moratorium apply to other journal-level measures also incorrectly used to assess the impact of individual researchers and individual articles, such as: (a) Scimago Journal Rank (SJR; based on Scopus data, it counts citations in a given year to publications in the previous three-year publication window, weighing citations such that they are assigned a greater or lesser value based on the SJR of the journal giving the citation); (b) Source Normalized Impact per Paper (SNIP; based on Scopus data, it measures contextual citation impact by weighing citations based on the total number of citations in a subject field); (c) Article Influence Score (AIS; based on WoS data, it measures the average number of citations received by a journal’s articles in the first five years after publication, and weighs citation by the quality of the journal providing the citation, normalized as a fraction of all articles in all publications); and (d) the newly released Journal Citation Indicator (JCI; based on WoS data, it is a field-normalized metric representing the average category-normalized citation impact for papers published in the prior three-year period).

Click to read the full article