Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Journal ranking should depend on the level of aggregation

Journal ranking should depend on the level of aggregation Journal ranking is becoming more important in assessing the quality of academic research. Several indices have been suggested for this purpose, typically on the basis of a citation graph between the journals. We follow an axiomatic approach and find an impossibility theorem: any self-consistent ranking method, which satisfies a natural monotonicity property, should depend on the level of aggregation. Our result presents a trade-off between two axiomatic properties and reveals a dilemma of aggregation. Keywords: Journal ranking; citations; axiomatic approach; impossibility MSC class: 91A80, 91B14 JEL classification number : C44, D71 1 Introduction The measurement of the quality and quantity of academic research plays an increasing role in the evaluation of researchers and research proposals. This paper will focus on a E-mail: csato.laszlo@sztaki.mta.hu “I maintain that in each particular natural science there is only as much true science as there is mathematics.” (Source: Smith, J. T.: David Hilbert’s 1930 Radio Address – German and English. https://www.maa.org/book/export/html/326610) arXiv:1904.06300v4 [cs.DL] 2 Sep 2019 particular field of scientometrics, that is, journal ranking. Furthermore, since a number of bibliometric indices have been suggested to assess intellectual influence, and now a plethora of ranking methods are available to measure the performance of journals and scholars (Palacios-Huerta and Volij, 2014), we follow an axiomatic approach because the introduction of some reasonable axioms or conditions is able to narrow the set of appropriate methods, to reveal their crucial properties, and to allow for their comparison. An important contribution of similar analyses can be an axiomatic characterisation, meaning that a set of properties uniquely determine a preference vector. For example, Palacios-Huerta and Volij (2004) give a characterisation of the invariant method, while Demange (2014) provides a characterisation of the handicap method, both of them used to rank journals. Results for citation indices are probably even more abundant, including characterisations of the ℎ -index (Kongo, 2014; Marchant, 2009; Miroiu, 2013; Quesada, 2010, 2011a,b; Woeginger, 2008b), the 𝑔 -index (Woeginger, 2008a; Quesada, 2011a; Adachi and Kongo, 2015), the Euclidean index (Perry and Reny, 2016), or a class of step-based indices (Chambers and Miller, 2014), among others. de la Vega and Volij (2018) characterise scholar rankings admitting a measure representation. There are also axiomatic comparisons of bibliometric indices (Bouyssou and Marchant, 2014, 2016). However, the above works seldom uncover the inevitable trade-offs between different natural requirements, an aim which can be achieved mainly by impossibility theorems. Similar results are well-established in social choice theory since Arrow’s impossibility the- orem (Arrow, 1951) and the Gibbard-Satterthwaite theorem (Gibbard, 1973; Satterthwaite, 1975; Duggan and Schwartz, 2000) but not so widely used in scientometrics. We provide an impossibility result in journal ranking. In particular, it will be proved that two axioms, invariance to aggregation and self-consistency, cannot be satisfied simultaneously even on a substantially restricted domain of citation graphs. Invariance to aggregation means that the ranking of two journals is not influenced by the level of aggregation among the remaining journals, while self-consistency, introduced by Chebotarev and Shamis (1997), is a kind of monotonicity property, responsible for some impossibility theorems in ranking from paired comparisons (Csat´ o, 2019a,b). The paper is organised as follows. Our setting and notations are introduced in Section 2. Section 3 motivates and defines the two axioms, which turn out to be incompatible in Section 4. Section 5 summarises the main findings and concludes. 2 The journal ranking problem A journal ranking problem consists of a group of journals and their respective citation records (Palacios-Huerta and Volij, 2014). Let 𝑁 = {𝐽 , 𝐽 , . . . , 𝐽 }, 𝑛 ∈ N be a non-empty 1 2 𝑛 𝑛 ×𝑛 finite set of journals and 𝐶 = [𝑐 ] ∈ R be a |𝑁 |×|𝑁 | nonnegative citation matrix 𝑖𝑗 for 𝑁 . The entry 𝑐 can be directly the number of citations that journal 𝐽 received 𝑖𝑗 𝑖 from journal 𝐽 , or any reasonable transformation of this value, for example, by using exponentially decreasing weights for older citations. The pair (𝑁, 𝐶 ) is called a journal ranking problem. The set of journal ranking problems with 𝑛 journals (|𝑁 | = 𝑛 ) is denoted by 𝒥 . The aim is to aggregate the opinions given in the citation matrix into a single judgement. 𝑛 𝑛 Formally, a scoring procedure 𝑓 is a 𝒥 → R function that takes a journal ranking problem (𝑁, 𝐶 ) and returns a rating 𝑓 (𝑁, 𝐶 ) for each journal 𝐽 ∈ 𝑁 , representing this judgement. 𝑖 𝑖 A scoring method immediately induces a ranking ⪰ for the journals of 𝑁 (a transitive and complete weak order on the set of 𝑁 ): 𝑓 (𝑁, 𝐶 ) ≥ 𝑓 (𝑁, 𝐶 ) means that journal 𝐽 is 𝑖 𝑗 𝑖 2 ranked weakly above 𝐽 , denoted by 𝐽 ⪰ 𝐽 . The symmetric and asymmetric parts of ⪰ 𝑗 𝑖 𝑗 are denoted by ∼ and ≻, respectively: 𝐽 ∼ 𝐽 if both 𝐽 ⪰ 𝐽 and 𝐽 ⪯ 𝐽 hold, while 𝑖 𝑗 𝑖 𝑗 𝑖 𝑗 𝐽 ≻ 𝐽 if 𝐽 ⪰ 𝐽 holds but 𝐽 ⪯ 𝐽 does not hold. 𝑖 𝑗 𝑖 𝑗 𝑖 𝑗 A journal ranking problem (𝑁, 𝐶 ) has the symmetric matches matrix 𝑀 = 𝐶 + 𝐶 = 𝑛 ×𝑛 [𝑚 ] ∈ R such that 𝑚 is the number of the citations between the journals 𝐽 and 𝑖𝑗 𝑖𝑗 𝑖 𝐽 in both directions, which can be called the number of matches between them in the terminology of sports (K´ oczy and Strobel, 2010; Csat´ o, 2015). It is sometimes convenient to consider not a general problem, arising from complicated networks of citations, but only a simpler one. ∑︀ ∑︀ A journal ranking problem (𝑁, 𝐶 ) ∈ 𝒥 is called balanced if 𝑚 = 𝑚 𝑖𝑘 𝑗𝑘 𝑋 ∈𝑁 𝑋 ∈𝑁 𝑘 𝑘 for all 𝐽 , 𝐽 ∈ 𝑁 . The set of balanced journal ranking problems is denoted by 𝒥 . In a 𝑖 𝑗 𝐵 balanced journal ranking problem, all journals have the same number of matches. A journal ranking problem (𝑁, 𝐶 ) ∈ 𝒥 is called unweighted if 𝑚 ∈ {0; 1} for all 𝑖𝑗 𝐽 , 𝐽 ∈ 𝑁 . The set of unweighted journal ranking problems is denoted by 𝒥 . In an 𝑖 𝑗 𝑈 unweighted journal ranking problem, either there is no citations, or there exists only one citation between any pair of journals. A journal ranking problem (𝑁, 𝐶 ) ∈ 𝒥 is called loopless if 𝑐 = 0 for all 𝐽 ∈ 𝑁 . 𝑖𝑖 𝑖 The set of unweighted journal ranking problems is denoted by 𝒥 . In a loopless problem, self-citations are disregarded. The subsets of balanced, unweighted, and loopless journal ranking problems restrict the matches matrix 𝑀 . A journal ranking problem (𝑁, 𝐶 ) ∈ 𝒥 is called extremal if |𝑐 | ∈ {0; 𝑚 /2; 𝑚 } 𝑖𝑗 𝑖𝑗 𝑖𝑗 for all 𝐽 , 𝐽 ∈ 𝑁 . The set of extremal journal ranking problems is denoted by 𝒥 . In 𝑖 𝑗 𝐸 an extremal journal ranking problem, only three cases are allowed in the comparison of journals 𝐽 and 𝐽 : there are citations only for 𝐽 or 𝐽 , or they are tied with respect to 𝑖 𝑗 𝑖 𝑗 mutual citations. Any intersection of these special classes can be considered, too. While a given citation matrix 𝐶 will seldom lead to a balanced, unweighted, loopless, or extremal journal ranking problem in practice, they can still be relevant for applications due to the possible transformation of citations. For example, it may make sense to remove self-citations from matrix 𝐶 , and consider only three types of paired comparisons in the derived matrix 𝐶 : ∙ 𝑐^ = 0 if 𝑐 = 0 and 𝑐 = 0; 𝑖𝑗 𝑖𝑗 𝑗𝑖 ∙ 𝑐^ = 0 if 𝑐 > 0 and 𝑐 < 𝑐 /2; 𝑖𝑗 𝑗𝑖 𝑖𝑗 𝑗𝑖 ∙ 𝑐^ = 0.5 if 𝑐 > 0 and 𝑐 /2 ≤ 𝑐 ≤ 2𝑐 ; 𝑖𝑗 𝑗𝑖 𝑗𝑖 𝑖𝑗 𝑗𝑖 ∙ 𝑐^ = 1 if 2𝑐 < 𝑐 . 𝑖𝑗 𝑗𝑖 𝑖𝑗 In other words, two journals are not compared (𝑐^ = 𝑐^ = 0) if they do not cite each other, 𝑖𝑗 𝑗𝑖 their paired comparison is tied (𝑐^ = 𝑐^ = 0.5) if their mutual citations are approximately 𝑖𝑗 𝑗𝑖 balanced – that is, 𝐽 does not refer to 𝐽 more than two times than 𝐽 refers to 𝐽 , and 𝑖 𝑗 𝑗 𝑖 vice versa –, and 𝐽 is maximally better than 𝐽 (𝑐^ = 1 and 𝑐^ = 0) if 𝐽 cites 𝐽 more 𝑖 𝑗 𝑖𝑗 𝑗𝑖 𝑗 𝑖 (︁ )︁ than two times than 𝐽 cites 𝐽 . Then the resulting journal ranking problem 𝑁, 𝐶 ∈ 𝒥 𝑖 𝑗 is unweighted, loopless, and extremal. 3 3 Axioms of journal ranking In this section two properties, a natural axiom of aggregation and a variant of monotonicity, are introduced. 3.1 Invariance to aggregation The first condition aims to regulate the ranking if two journals are aggregated into one. Axiom 1. Invariance to aggregation (𝐼𝐴 ): Let (𝑁, 𝐶 ) ∈ 𝒥 be a journal ranking problem 𝑖 ∪𝑗 𝑖 ∪𝑗 𝑛 −1 and 𝐽 , 𝐽 ∈ 𝑁 be two different journals. Journal ranking problem (𝑁 , 𝐶 ) ∈ 𝒥 is 𝑖 𝑗 [︁ ]︁ 𝑖 ∪𝑗 𝑖 ∪𝑗 𝑖 ∪𝑗 (𝑛 −1)×(𝑛 −1) given by 𝑁 = (𝑁 ∖{ 𝐽 , 𝐽 })∪ 𝐽 and 𝐶 = 𝑐 ∈ R such that 𝑖 𝑗 𝑖 ∪𝑗 𝑘ℓ 𝑖 ∪𝑗 ∙ 𝑐 = 𝑐 if {𝐽 , 𝐽 }∩{𝐽 , 𝐽 } = ∅; 𝑘ℓ 𝑘 ℓ 𝑖 𝑗 𝑘ℓ 𝑖 ∪𝑗 ∙ 𝑐 = 𝑐 + 𝑐 for all 𝐽 ∈ 𝑁 ∖{ 𝐽 , 𝐽 }; 𝑘𝑖 𝑘𝑗 𝑘 𝑖 𝑗 𝑘 (𝑖 ∪𝑗 ) 𝑖 ∪𝑗 ∙ 𝑐 = 𝑐 + 𝑐 for all 𝐽 ∈ 𝑁 ∖{ 𝐽 , 𝐽 }. 𝑖ℓ 𝑗ℓ ℓ 𝑖 𝑗 (𝑖 ∪𝑗 )ℓ 𝑛 𝑛 Scoring procedure 𝑓 : 𝒥 → R is called invariant to aggregation if 𝑓 (𝑁, 𝐶 ) ≥ 𝑓 (𝑁, 𝐶 ) 𝑘 ℓ 𝑖 ∪𝑗 𝑖 ∪𝑗 𝑖 ∪𝑗 𝑖 ∪𝑗 implies 𝑓 (𝑁 , 𝐶 ) ≥ 𝑓 (𝑁 , 𝐶 ) for all 𝐽 , 𝐽 ∈ 𝑁 ∖{ 𝐽 , 𝐽 }. 𝑘 ℓ 𝑘 ℓ 𝑖 𝑗 The idea behind invariance to aggregation is that any journal ranking problem can be transformed into a reduced problem by defining the union 𝐽 of journals 𝐽 and 𝐽 as 𝑖 ∪𝑗 𝑖 𝑗 follows: all citations between them are deleted, while any citations by/to these journals are summed up for the “aggregated” journal 𝐽 . This transformation is required to preserve 𝑖 ∪𝑗 the order of the journals not affected by the aggregation. Such an aggregation makes sense, for example, if one is interested only in the ranking of journals from a given field (e.g. economics journals) when journals from other disciplines can be considered as one entity. Invariance to aggregation is somewhat related to the consistency axiom of Palacios- Huerta and Volij (2004), which is also based on the notion of the reduced problem. However, our property probably takes the information from the missing journal in a more straightforward way into consideration. Invariance to aggregation has some connections to the famous independence of irrelevant alternatives (𝐼𝐼𝐴 ) condition, too, which is used, for example, in Arrow’s impossibility theorem (Arrow, 1951). Both axioms require an important aspect of the problem, the citations between two journals and the individual preferences between two alternatives, respectively, to remain fixed. However, there is a crucial difference: the set of alternatives (corresponding to journals) is allowed to change in the case of 𝐼𝐴 , while the preferences (corresponding to citations) are allowed to change in the case of 𝐼𝐼𝐴 . 3.2 Self-consistency This axiom, originally introduced in Chebotarev and Shamis (1997) to operators used for aggregating preferences, may require a longer explanation. First, some reasonable conditions are formulated for the ranking derived from any journal ranking problem. In particular, journal 𝐽 is judged better than journal 𝐽 if one 𝑖 𝑗 of the following holds: D1) 𝐽 has more favourable citation records against the same journals; 4 D2) 𝐽 has more favourable citation records against journals with the same quality; D3) 𝐽 has the same citation records against higher quality journals; D4) 𝐽 has more favourable citation records against higher quality journals. In addition, journals 𝐽 and 𝐽 should get the same rank if one of the following holds: 𝑖 𝑗 D5) they have the same citation records against the same journals; D6) they have the same citation records against journals with the same quality. Principles D2-D4 and D6 can be applied only after measuring the quality of the journals. The name of the property, self-consistency, refers to the fact that this is provided by the scoring procedure itself. The meaning of the requirements above is illustrated by an example. Figure 1: The journal ranking problem of Example 3.1 𝐽 𝐽 1 2 𝐽 𝐽 3 4 4 4 4 4 Example 3.1. Consider the journal ranking problem (𝑁, 𝐶 ) ∈ 𝒥 ∩𝒥 ∩𝒥 ∩𝒥 with 𝐵 𝑈 𝐿 𝐸 the following citation matrix: ⎡ ⎤ 0 1 1 0 ⎢ ⎥ 0 0 0 1 ⎢ ⎥ 𝐶 = ⎢ ⎥ . ⎣ ⎦ 0 0 0 1 0 0 0 0 This is shown in Figure 1 where a directed edge from node 𝐽 to 𝐽 indicates that journal 𝑖 𝑗 𝐽 has received a citation from journal 𝐽 . 𝑖 𝑗 Self-consistency has the following implications for the journal ranking problem presented in Example 3.1: ∙ 𝐽 ∼ 𝐽 due to rule D5. 2 3 ∙ 𝐽 ≻ 𝐽 because of rule D1 as 𝑐 > 𝑐 and 𝑐 > 𝑐 . 1 4 12 42 13 43 ∙ Assume for contradiction that 𝐽 ⪯ 𝐽 . Then 𝑐 > 𝑐 and 𝐽 ⪰ 𝐽 , as well as 1 2 12 21 2 1 𝑐 = 𝑐 and 𝐽 ∼ 𝐽 ⪰ 𝐽 ≻ 𝐽 , so rule D4 leads to 𝐽 ≻ 𝐽 , which is impossible. 13 24 3 2 1 4 1 2 Consequently, 𝐽 ≻ (𝐽 ∼ 𝐽 ). 1 2 3 ∙ Assume for contradiction that 𝐽 ⪯ 𝐽 . Then 𝑐 > 𝑐 and 𝐽 ≻ 𝐽 , as well as 2 4 21 43 1 3 𝑐 > 𝑐 and 𝐽 ⪰ 𝐽 ∼ 𝐽 , so rule D4 leads to 𝐽 ≻ 𝐽 , which is impossible. 24 43 4 2 3 2 4 Consequently, (𝐽 ∼ 𝐽 ) ≻ 𝐽 . 2 3 4 5 To conclude, self-consistency demands the ranking to be 𝐽 ≻ (𝐽 ∼ 𝐽 ) ≻ 𝐽 in Ex- 1 2 3 4 ample 3.1. It is clear that self-consistency does not guarantee the uniqueness of the ranking in general (Csat´ o, 2019a). Now we turn to the mathematical formulation of this axiom. Definition 3.1. Competitor set: Let (𝑁, 𝐶 ) ∈ 𝒥 be an unweighted journal ranking problem. The competitor set of journal 𝐽 is 𝑆 = {𝐽 : 𝑚 = 1}. 𝑖 𝑖 𝑗 𝑖𝑗 Journals in the competitor set 𝑆 are called the competitors of 𝐽 . Note that |𝑆 | = |𝑆 | 𝑖 𝑖 𝑖 𝑗 for all 𝐽 , 𝐽 ∈ 𝑁 if and only if the ranking problem is balanced. 𝑖 𝑗 The competitor set is defined only for unweighted journal ranking problem but self- consistency may have implications for any pair of journals which have the same number of matches. The generalisation is based on a decomposition of journal ranking problems. ′ 𝑛 Definition 3.2. Sum of journal ranking problems: Let (𝑁, 𝐶 ), (𝑁, 𝐶 ) ∈ 𝒥 be two journal ranking problems with the same set of journals 𝑁 . The sum of these journal ′ 𝑛 ranking problems is the journal ranking problem (𝑁, 𝐶 + 𝐶 ) ∈ 𝒥 . The sum of journal ranking problems has a number of reasonable interpretations. For instance, they can reflect the citations from different years, or by authors from different countries. According to Definition 3.2, any journal ranking problem can be derived as the sum of unweighted journal ranking problems. However, it might have a number of possible decompositions. (𝑝 ) 𝑛 Notation 3.1. Let (𝑁, 𝐶 ) ∈ 𝒥 be an unweighted journal ranking problem. The (𝑝 ) (𝑝 ) competitor set of journal 𝐽 is 𝑆 . Let 𝐽 , 𝐽 ∈ 𝑁 be two different journals and 𝑔 : 𝑖 𝑖 𝑗 (𝑝 ) (𝑝 ) 𝑆 ↔ 𝑆 be a one-to-one correspondence between the competitors of 𝐽 and 𝐽 . Then 𝑖 𝑗 𝑖 𝑗 (𝑝 ) (𝑝 ) (𝑝 ) (𝑝 ) g : {𝑘 : 𝐽 ∈ 𝑆 } ↔ {ℓ : 𝐽 ∈ 𝑆 } is given by 𝐽 = 𝑔 (𝐽 ). (𝑝 ) 𝑘 ℓ 𝑘 𝑖 𝑗 g (𝑘 ) Finally, we are able to introduce conditions D1-D6 with mathematical formulas. Axiom 2. Self-consistency (𝑆𝐶 ) (Chebotarev and Shamis, 1997): Scoring procedure 𝑓 : 𝑛 𝑛 𝒥 → R is called self-consistent if the following implication holds for any journal ranking problem (𝑁, 𝐶 ) ∈ 𝒥 and for any journals 𝐽 , 𝐽 ∈ 𝑁 : if there exists a decomposition of 𝑖 𝑗 the journal ranking problem (𝑁, 𝐶 ) into 𝑚 unweighted journal ranking problems – that ∑︀ (𝑝 ) (𝑝 ) 𝑛 is, 𝐶 = 𝐶 and (𝑁, 𝐶 ) ∈ 𝒥 is an unweighted journal ranking problem for all 𝑝 =1 𝑈 (𝑝 ) (𝑝 ) 𝑝 = 1, 2, . . . , 𝑚 – together with the existence of a one-to-one mapping 𝑔 from 𝑆 onto (𝑝 ) (𝑝 ) (𝑝 ) 𝑆 such that 𝑐 ≥ 𝑐 and 𝑓 (𝑁, 𝐶 ) ≥ 𝑓 (𝑁, 𝐶 ) for all 𝑝 = 1, 2, . . . , 𝑚 and (𝑝 ) (𝑝 ) 𝑘 𝑗 𝑖𝑘 g (𝑘 ) 𝑗 g (𝑘 ) (𝑝 ) (𝑝 ) (𝑝 ) 𝐽 ∈ 𝑆 , then 𝑓 (𝑁, 𝐶 ) ≥ 𝑓 (𝑁, 𝐶 ). Furthermore, 𝑓 (𝑁, 𝐶 ) > 𝑓 (𝑁, 𝐶 ) if 𝑐 > 𝑐 𝑘 𝑖 𝑗 𝑖 𝑗 𝑖 (𝑝 ) 𝑖𝑘 𝑗 g (𝑘 ) (𝑝 ) or 𝑓 (𝑁, 𝐶 ) > 𝑓 (𝑝 ) (𝑁, 𝐶 ) for at least one 1 ≤ 𝑝 ≤ 𝑚 and 𝐽 ∈ 𝑆 . 𝑘 𝑘 g (𝑘 ) 𝑖 In a nutshell, self-consistency implies that if journal 𝐽 does not show worse performance than journal 𝐽 on the basis of the citation matrix, then it is not ranked lower, in addition, it is ranked strictly higher when it becomes clearly better. Chebotarev and Shamis (1997) consider only loopless journal ranking problems but the extension of self-consistency is trivial as presented above. Chebotarev and Shamis (1998, Theorem 5) gives a necessary and sufficient condition for self-consistent scoring procedures, while Chebotarev and Shamis (1998, Table 2) presents some scoring procedures that satisfy this requirement. See also Csat´ o (2019a) for an extensive discussion of self-consistency. 6 4 The incompatibility of the two axioms In the following, it will be proved that no scoring procedure can meet axioms 𝐼𝐴 and 𝑆𝐶 . Figure 2: The journal ranking problems of Example 4.1 3∪4 3∪4 (b) Journal ranking problem (𝑁 ,𝐶 ) (a) Journal ranking problem (𝑁,𝐶 ) 𝐽 𝐽 𝐽 𝐽 1 2 1 2 3∪4 𝐽 𝐽 3 4 4 4 4 4 3∪4 3∪4 3 3 3 3 Example 4.1. Let (𝑁, 𝐶 ) ∈ 𝒥 ∩𝒥 ∩𝒥 ∩𝒥 and (𝑁 , 𝐶 ) ∈ 𝒥 ∪𝒥 ∩𝒥 ∩𝒥 𝐵 𝑈 𝐿 𝐸 𝐵 𝑈 𝐿 𝐸 be the journal ranking problems with the citation matrices ⎡ ⎤ ⎡ ⎤ 0 0.5 0.5 0 0 0.5 0.5 ⎢ ⎥ 0.5 0 0 0.5 ⎢ ⎥ ⎢ ⎥ 3∪4 𝐶 = ⎢ ⎥ and 𝐶 = ⎣ 0.5 0 0.5 ⎦ , respectively. ⎣ ⎦ 0.5 0 0 1 0.5 0.5 0 0 0.5 0 0 3∪4 3∪4 Journal ranking problem (𝑁 , 𝐶 ) is obtained by uniting journals 3 and 4. This is shown in Figure 2 where a directed edge from node 𝐽 to 𝐽 indicates that 𝑖 𝑗 journal 𝐽 has received a citation from journal 𝐽 , and an undirected edge between the 𝑖 𝑗 nodes means that the two journals are tied by mutual citations. Theorem 4.1. There exists no scoring procedure that is invariant to aggregation and self-consistent. Proof. The contradiction of the two properties can be proved by Example 4.1. Take first the journal ranking problem (𝑁, 𝐶 ), which has the competitor sets 𝑆 = 𝑆 = {𝐽 , 𝐽 } 1 4 2 3 and 𝑆 = 𝑆 = {𝐽 , 𝐽 }. Assume for contradiction the existence of a scoring procedure 2 3 1 4 𝑛 𝑛 𝑓 : 𝒥 → R satisfying invariance to aggregation and self-consistency. Self-consistency has several implications for the scoring procedure 𝑓 as follows: a) Consider the (identity) one-to-one correspondence 𝑔 : 𝑆 ↔ 𝑆 such that 14 1 4 𝑔 (𝐽 ) = 𝐽 and 𝑔 (𝐽 ) = 𝐽 . Then 𝑔 satisfies condition D1 of 𝑆𝐶 due to 14 2 2 14 3 3 14 𝑐 = 𝑐 = 0.5 and 0.5 = 𝑐 > 𝑐 = 0, thus 𝑓 (𝑁, 𝐶 ) > 𝑓 (𝑁, 𝐶 ). 12 42 13 43 1 4 b) Consider the (identity) one-to-one correspondence 𝑔 : 𝑆 ↔ 𝑆 such that 32 3 2 𝑔 (𝐽 ) = 𝐽 and 𝑔 (𝐽 ) = 𝐽 . Then 𝑔 satisfies condition D1 of 𝑆𝐶 due to 32 1 1 32 4 4 32 𝑐 = 𝑐 = 0.5 and 1 = 𝑐 > 𝑐 = 0.5, thus 𝑓 (𝑁, 𝐶 ) > 𝑓 (𝑁, 𝐶 ). 31 31 34 24 3 2 c) Suppose that 𝑓 (𝑁, 𝐶 ) ≥ 𝑓 (𝑁, 𝐶 ), which implies 𝑓 (𝑁, 𝐶 ) > 𝑓 (𝑁, 𝐶 ) according 2 1 3 4 to the inequalities derived in a) and b). Consider the one-to-one correspondence 𝑔 : 𝑆 ↔ 𝑆 such that 𝑔 (𝐽 ) = 𝐽 and 𝑔 (𝐽 ) = 𝐽 . Then 𝑔 satisfies 12 1 2 12 2 1 12 3 4 12 condition D3 of 𝑆𝐶 due to 𝑐 = 𝑐 = 0.5 and 𝑐 = 𝑐 = 0.5, thus 𝑓 (𝑁, 𝐶 ) > 12 21 13 24 1 𝑓 (𝑁, 𝐶 ), a contradiction. 7 Therefore 𝑓 (𝑁, 𝐶 ) > 𝑓 (𝑁, 𝐶 ) should hold, when invariance to aggregation results 1 2 ′ ′ ′ ′ in 𝑓 (𝑁, 𝐶 ) > 𝑓 (𝑁, 𝐶 ). However, self-consistency leads to 𝑓 (𝑁, 𝐶 ) = 𝑓 (𝑁, 𝐶 ) in the 1 2 1 2 ′ ′ ′ journal ranking problem (𝑁, 𝐶 ) because of the one-to-one mapping 𝑔 : 𝑆 ↔ 𝑆 such 1 2 ′ ′ that 𝑔 (𝐽 ) = 𝐽 and 𝑔 (𝐽 ) = 𝐽 : the assumption of 𝑓 (𝑁, 𝐶 ) > 𝑓 (𝑁, 𝐶 ) implies 12 2 1 12 1 2 1 2 ′ ′ 𝑓 (𝑁, 𝐶 ) < 𝑓 (𝑁, 𝐶 ) due to condition D3 (the competitors of 𝐽 are more prestigious), 1 2 2 ′ ′ ′ ′ while 𝑓 (𝑁, 𝐶 ) < 𝑓 (𝑁, 𝐶 ) would result in 𝑓 (𝑁, 𝐶 ) < 𝑓 (𝑁, 𝐶 ) due to condition D3 1 2 1 2 (the competitors of 𝐽 are more prestigious) again. Hence a scoring procedure cannot meet 𝐼𝐴 and 𝑆𝐶 at the same time. Since Example 4.1 contains balanced, unweighted, loopless, and extremal journal ranking problems, there is few hope to avoid the impossibility of Theorem 4.1 by plausible domain restrictions. Remark 4.1. 𝐼𝐴 and 𝑆𝐶 are logically independent axioms as there exist scoring proced- ures that satisfy one of the two properties: the least squares method is self-consistent (Chebotarev and Shamis, 1998, Theorem 5), and the flat scoring procedure, which gives 𝑓 (𝑁, 𝐶 ) = 0 for all 𝑖 ∈ 𝑁 and (𝑁, 𝐶 ) ∈ 𝒥 , is invariant to aggregation. 5 Conclusions We have presented an impossibility theorem in journal ranking: a reasonable method cannot be invariant to the aggregation of journals, even in the case of a substantially restricted domain of citation graphs. An intuitive explanation is that invariance to aggregation is a local property (it modifies only the citations directly affecting the journals to be united), while self-consistency considers the global structure of the citations as it depends on the quality of the journals. The clash between local and global axioms can also be observed in other fields, such as game theory. In addition, the impossibility result clearly shows that invariance to aggregation is a rather strong requirement, similarly to its peer independence of irrelevant alternatives (Malawski and Zhou, 1994). Nevertheless, according to our finding, the choice of the set of journals to be compared is an important aspect of every empirical study which aims to measure intellectual influence. It is clear that the axiomatic analysis discussed here has a number of limitations as it is able to consider indices from only one point of view (Gl¨ anzel and Moed, 2013). For example, the citation graph is assumed to be known, that is, the issue of choosing an adequate time window is neglected. In addition, this paper has not addressed several important problems of scientometrics such as the comparability of distant research areas, or the proper treatment of different types of publications. To summarise, the derivation of similar impossibility results may contribute to a better understanding of the inevitable trade-offs between various properties, and it means a natural subject of further studies besides axiomatic characterisations. Acknowledgements We are grateful to Gy¨ orgy Moln´ ar and D´ ora Gr´ eta Petr´ oczy for inspiration. Two anonymous reviewers provided valuable comments and suggestions on an earlier draft. The research was supported by OTKA grant K 111797 and by the MTA Premium Postdoctoral Research Program. 8 References Adachi, T. and Kongo, T. (2015). Further axiomatizations of Egghe’s 𝑔 -index. Journal of Informetrics, 9(4):839–844. Arrow, K. J. (1951). Social choice and individual values. Wiley, New York. Bouyssou, D. and Marchant, T. (2014). An axiomatic approach to bibliometric rankings and indices. Journal of Informetrics, 8(3):449–477. Bouyssou, D. and Marchant, T. (2016). Ranking authors using fractional counting of citations: An axiomatic approach. Journal of Informetrics, 10(1):183–199. Chambers, C. P. and Miller, A. D. (2014). Scholarly influence. Journal of Economic Theory, 151:571–583. Chebotarev, P. and Shamis, E. (1997). Constructing an objective function for aggregating incomplete preferences. In Tangian, A. and Gruber, J., editors, Constructing Scalar- Valued Objective Functions, volume 453 of Lecture Notes in Economics and Mathematical Systems, pages 100–124. Springer, Berlin-Heidelberg. Chebotarev, P. Yu. and Shamis, E. (1998). Characterizations of scoring methods for preference aggregation. Annals of Operations Research, 80:299–332. Csat´ o, L. (2015). A graph interpretation of the least squares ranking method. Social Choice and Welfare, 44(1):51–69. Csat´ o, L. (2019a). An impossibility theorem for paired comparisons. Central European Journal of Operations Research, 27(2):497–514. Csat´ o, L. (2019b). Some impossibilities of ranking in generalized tournaments. International Game Theory Review, 21(01):1940002. de la Vega, C. L. and Volij, O. (2018). Ranking scholars: A measure representation. Journal of Informetrics, 12(2):510–517. Demange, G. (2014). A ranking method based on handicaps. Theoretical Economics, 9(3):915–942. Duggan, J. and Schwartz, T. (2000). Strategic manipulability without resoluteness or shared beliefs: Gibbard-Satterthwaite generalized. Social Choice and Welfare, 17(1):85–93. Gibbard, A. (1973). Manipulation of voting schemes: A general result. Econometrica, 41(4):587–601. Gl¨ anzel, W. and Moed, H. F. (2013). Opinion paper: thoughts and facts on bibliometric indicators. Scientometrics, 96(1):381–394. K´ oczy, L. A. and Strobel, M. (2010). The world cup of economics journals: A ranking by a tournament method. IEHAS Discussion Papers 1018, Institute of Economics, Hungarian Academy of Sciences. Kongo, T. (2014). An alternative axiomatization of the Hirsch index. Journal of Informet- rics, 8(1):252–258. 9 Malawski, M. and Zhou, L. (1994). A note on social choice theory without the Pareto principle. Social Choice and Welfare, 11(2):103–107. Marchant, T. (2009). An axiomatic characterization of the ranking based on the h-index and some other bibliometric rankings of authors. Scientometrics, 80(2):325–342. Miroiu, A. (2013). Axiomatizing the Hirsch index: Quantity and quality disjoined. Journal of Informetrics, 7(1):10–15. Palacios-Huerta, I. and Volij, O. (2004). The measurement of intellectual influence. Econometrica, 72(3):963–977. Palacios-Huerta, I. and Volij, O. (2014). Axiomatic measures of intellectual influence. International Journal of Industrial Organization, 34:85–90. Perry, M. and Reny, P. J. (2016). How to count citations if you must. American Economic Review, 106(9):2722–2741. Quesada, A. (2010). More axiomatics for the Hirsch index. Scientometrics, 82(2):413–418. Quesada, A. (2011a). Axiomatics for the Hirsch index and the Egghe index. Journal of Informetrics, 5(3):476–480. Quesada, A. (2011b). Further characterizations of the Hirsch index. Scientometrics, 87(1):107–114. Satterthwaite, M. A. (1975). Strategy-proofness and Arrow’s conditions: Existence and correspondence theorems for voting procedures and social welfare functions. Journal of Economic Theory, 10(2):187–217. Woeginger, G. J. (2008a). An axiomatic analysis of Egghe’s 𝑔 -index. Journal of Informetrics, 2(4):364–368. Woeginger, G. J. (2008b). An axiomatic characterization of the Hirsch-index. Mathematical Social Sciences, 56(2):224–232. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Economics arXiv (Cornell University)

Journal ranking should depend on the level of aggregation

Csató, László
Economics , Volume 2020 (1904) – Apr 8, 2019
Download PDF
10 pages

Loading next page...
 
/lp/arxiv-cornell-university/journal-ranking-should-depend-on-the-level-of-aggregation-00Z02CvB9F
ISSN
1751-1577
eISSN
ARCH-3349
DOI
10.1016/j.joi.2019.100975
Publisher site
See Article on Publisher Site

Abstract

Journal ranking is becoming more important in assessing the quality of academic research. Several indices have been suggested for this purpose, typically on the basis of a citation graph between the journals. We follow an axiomatic approach and find an impossibility theorem: any self-consistent ranking method, which satisfies a natural monotonicity property, should depend on the level of aggregation. Our result presents a trade-off between two axiomatic properties and reveals a dilemma of aggregation. Keywords: Journal ranking; citations; axiomatic approach; impossibility MSC class: 91A80, 91B14 JEL classification number : C44, D71 1 Introduction The measurement of the quality and quantity of academic research plays an increasing role in the evaluation of researchers and research proposals. This paper will focus on a E-mail: csato.laszlo@sztaki.mta.hu “I maintain that in each particular natural science there is only as much true science as there is mathematics.” (Source: Smith, J. T.: David Hilbert’s 1930 Radio Address – German and English. https://www.maa.org/book/export/html/326610) arXiv:1904.06300v4 [cs.DL] 2 Sep 2019 particular field of scientometrics, that is, journal ranking. Furthermore, since a number of bibliometric indices have been suggested to assess intellectual influence, and now a plethora of ranking methods are available to measure the performance of journals and scholars (Palacios-Huerta and Volij, 2014), we follow an axiomatic approach because the introduction of some reasonable axioms or conditions is able to narrow the set of appropriate methods, to reveal their crucial properties, and to allow for their comparison. An important contribution of similar analyses can be an axiomatic characterisation, meaning that a set of properties uniquely determine a preference vector. For example, Palacios-Huerta and Volij (2004) give a characterisation of the invariant method, while Demange (2014) provides a characterisation of the handicap method, both of them used to rank journals. Results for citation indices are probably even more abundant, including characterisations of the ℎ -index (Kongo, 2014; Marchant, 2009; Miroiu, 2013; Quesada, 2010, 2011a,b; Woeginger, 2008b), the 𝑔 -index (Woeginger, 2008a; Quesada, 2011a; Adachi and Kongo, 2015), the Euclidean index (Perry and Reny, 2016), or a class of step-based indices (Chambers and Miller, 2014), among others. de la Vega and Volij (2018) characterise scholar rankings admitting a measure representation. There are also axiomatic comparisons of bibliometric indices (Bouyssou and Marchant, 2014, 2016). However, the above works seldom uncover the inevitable trade-offs between different natural requirements, an aim which can be achieved mainly by impossibility theorems. Similar results are well-established in social choice theory since Arrow’s impossibility the- orem (Arrow, 1951) and the Gibbard-Satterthwaite theorem (Gibbard, 1973; Satterthwaite, 1975; Duggan and Schwartz, 2000) but not so widely used in scientometrics. We provide an impossibility result in journal ranking. In particular, it will be proved that two axioms, invariance to aggregation and self-consistency, cannot be satisfied simultaneously even on a substantially restricted domain of citation graphs. Invariance to aggregation means that the ranking of two journals is not influenced by the level of aggregation among the remaining journals, while self-consistency, introduced by Chebotarev and Shamis (1997), is a kind of monotonicity property, responsible for some impossibility theorems in ranking from paired comparisons (Csat´ o, 2019a,b). The paper is organised as follows. Our setting and notations are introduced in Section 2. Section 3 motivates and defines the two axioms, which turn out to be incompatible in Section 4. Section 5 summarises the main findings and concludes. 2 The journal ranking problem A journal ranking problem consists of a group of journals and their respective citation records (Palacios-Huerta and Volij, 2014). Let 𝑁 = {𝐽 , 𝐽 , . . . , 𝐽 }, 𝑛 ∈ N be a non-empty 1 2 𝑛 𝑛 ×𝑛 finite set of journals and 𝐶 = [𝑐 ] ∈ R be a |𝑁 |×|𝑁 | nonnegative citation matrix 𝑖𝑗 for 𝑁 . The entry 𝑐 can be directly the number of citations that journal 𝐽 received 𝑖𝑗 𝑖 from journal 𝐽 , or any reasonable transformation of this value, for example, by using exponentially decreasing weights for older citations. The pair (𝑁, 𝐶 ) is called a journal ranking problem. The set of journal ranking problems with 𝑛 journals (|𝑁 | = 𝑛 ) is denoted by 𝒥 . The aim is to aggregate the opinions given in the citation matrix into a single judgement. 𝑛 𝑛 Formally, a scoring procedure 𝑓 is a 𝒥 → R function that takes a journal ranking problem (𝑁, 𝐶 ) and returns a rating 𝑓 (𝑁, 𝐶 ) for each journal 𝐽 ∈ 𝑁 , representing this judgement. 𝑖 𝑖 A scoring method immediately induces a ranking ⪰ for the journals of 𝑁 (a transitive and complete weak order on the set of 𝑁 ): 𝑓 (𝑁, 𝐶 ) ≥ 𝑓 (𝑁, 𝐶 ) means that journal 𝐽 is 𝑖 𝑗 𝑖 2 ranked weakly above 𝐽 , denoted by 𝐽 ⪰ 𝐽 . The symmetric and asymmetric parts of ⪰ 𝑗 𝑖 𝑗 are denoted by ∼ and ≻, respectively: 𝐽 ∼ 𝐽 if both 𝐽 ⪰ 𝐽 and 𝐽 ⪯ 𝐽 hold, while 𝑖 𝑗 𝑖 𝑗 𝑖 𝑗 𝐽 ≻ 𝐽 if 𝐽 ⪰ 𝐽 holds but 𝐽 ⪯ 𝐽 does not hold. 𝑖 𝑗 𝑖 𝑗 𝑖 𝑗 A journal ranking problem (𝑁, 𝐶 ) has the symmetric matches matrix 𝑀 = 𝐶 + 𝐶 = 𝑛 ×𝑛 [𝑚 ] ∈ R such that 𝑚 is the number of the citations between the journals 𝐽 and 𝑖𝑗 𝑖𝑗 𝑖 𝐽 in both directions, which can be called the number of matches between them in the terminology of sports (K´ oczy and Strobel, 2010; Csat´ o, 2015). It is sometimes convenient to consider not a general problem, arising from complicated networks of citations, but only a simpler one. ∑︀ ∑︀ A journal ranking problem (𝑁, 𝐶 ) ∈ 𝒥 is called balanced if 𝑚 = 𝑚 𝑖𝑘 𝑗𝑘 𝑋 ∈𝑁 𝑋 ∈𝑁 𝑘 𝑘 for all 𝐽 , 𝐽 ∈ 𝑁 . The set of balanced journal ranking problems is denoted by 𝒥 . In a 𝑖 𝑗 𝐵 balanced journal ranking problem, all journals have the same number of matches. A journal ranking problem (𝑁, 𝐶 ) ∈ 𝒥 is called unweighted if 𝑚 ∈ {0; 1} for all 𝑖𝑗 𝐽 , 𝐽 ∈ 𝑁 . The set of unweighted journal ranking problems is denoted by 𝒥 . In an 𝑖 𝑗 𝑈 unweighted journal ranking problem, either there is no citations, or there exists only one citation between any pair of journals. A journal ranking problem (𝑁, 𝐶 ) ∈ 𝒥 is called loopless if 𝑐 = 0 for all 𝐽 ∈ 𝑁 . 𝑖𝑖 𝑖 The set of unweighted journal ranking problems is denoted by 𝒥 . In a loopless problem, self-citations are disregarded. The subsets of balanced, unweighted, and loopless journal ranking problems restrict the matches matrix 𝑀 . A journal ranking problem (𝑁, 𝐶 ) ∈ 𝒥 is called extremal if |𝑐 | ∈ {0; 𝑚 /2; 𝑚 } 𝑖𝑗 𝑖𝑗 𝑖𝑗 for all 𝐽 , 𝐽 ∈ 𝑁 . The set of extremal journal ranking problems is denoted by 𝒥 . In 𝑖 𝑗 𝐸 an extremal journal ranking problem, only three cases are allowed in the comparison of journals 𝐽 and 𝐽 : there are citations only for 𝐽 or 𝐽 , or they are tied with respect to 𝑖 𝑗 𝑖 𝑗 mutual citations. Any intersection of these special classes can be considered, too. While a given citation matrix 𝐶 will seldom lead to a balanced, unweighted, loopless, or extremal journal ranking problem in practice, they can still be relevant for applications due to the possible transformation of citations. For example, it may make sense to remove self-citations from matrix 𝐶 , and consider only three types of paired comparisons in the derived matrix 𝐶 : ∙ 𝑐^ = 0 if 𝑐 = 0 and 𝑐 = 0; 𝑖𝑗 𝑖𝑗 𝑗𝑖 ∙ 𝑐^ = 0 if 𝑐 > 0 and 𝑐 < 𝑐 /2; 𝑖𝑗 𝑗𝑖 𝑖𝑗 𝑗𝑖 ∙ 𝑐^ = 0.5 if 𝑐 > 0 and 𝑐 /2 ≤ 𝑐 ≤ 2𝑐 ; 𝑖𝑗 𝑗𝑖 𝑗𝑖 𝑖𝑗 𝑗𝑖 ∙ 𝑐^ = 1 if 2𝑐 < 𝑐 . 𝑖𝑗 𝑗𝑖 𝑖𝑗 In other words, two journals are not compared (𝑐^ = 𝑐^ = 0) if they do not cite each other, 𝑖𝑗 𝑗𝑖 their paired comparison is tied (𝑐^ = 𝑐^ = 0.5) if their mutual citations are approximately 𝑖𝑗 𝑗𝑖 balanced – that is, 𝐽 does not refer to 𝐽 more than two times than 𝐽 refers to 𝐽 , and 𝑖 𝑗 𝑗 𝑖 vice versa –, and 𝐽 is maximally better than 𝐽 (𝑐^ = 1 and 𝑐^ = 0) if 𝐽 cites 𝐽 more 𝑖 𝑗 𝑖𝑗 𝑗𝑖 𝑗 𝑖 (︁ )︁ than two times than 𝐽 cites 𝐽 . Then the resulting journal ranking problem 𝑁, 𝐶 ∈ 𝒥 𝑖 𝑗 is unweighted, loopless, and extremal. 3 3 Axioms of journal ranking In this section two properties, a natural axiom of aggregation and a variant of monotonicity, are introduced. 3.1 Invariance to aggregation The first condition aims to regulate the ranking if two journals are aggregated into one. Axiom 1. Invariance to aggregation (𝐼𝐴 ): Let (𝑁, 𝐶 ) ∈ 𝒥 be a journal ranking problem 𝑖 ∪𝑗 𝑖 ∪𝑗 𝑛 −1 and 𝐽 , 𝐽 ∈ 𝑁 be two different journals. Journal ranking problem (𝑁 , 𝐶 ) ∈ 𝒥 is 𝑖 𝑗 [︁ ]︁ 𝑖 ∪𝑗 𝑖 ∪𝑗 𝑖 ∪𝑗 (𝑛 −1)×(𝑛 −1) given by 𝑁 = (𝑁 ∖{ 𝐽 , 𝐽 })∪ 𝐽 and 𝐶 = 𝑐 ∈ R such that 𝑖 𝑗 𝑖 ∪𝑗 𝑘ℓ 𝑖 ∪𝑗 ∙ 𝑐 = 𝑐 if {𝐽 , 𝐽 }∩{𝐽 , 𝐽 } = ∅; 𝑘ℓ 𝑘 ℓ 𝑖 𝑗 𝑘ℓ 𝑖 ∪𝑗 ∙ 𝑐 = 𝑐 + 𝑐 for all 𝐽 ∈ 𝑁 ∖{ 𝐽 , 𝐽 }; 𝑘𝑖 𝑘𝑗 𝑘 𝑖 𝑗 𝑘 (𝑖 ∪𝑗 ) 𝑖 ∪𝑗 ∙ 𝑐 = 𝑐 + 𝑐 for all 𝐽 ∈ 𝑁 ∖{ 𝐽 , 𝐽 }. 𝑖ℓ 𝑗ℓ ℓ 𝑖 𝑗 (𝑖 ∪𝑗 )ℓ 𝑛 𝑛 Scoring procedure 𝑓 : 𝒥 → R is called invariant to aggregation if 𝑓 (𝑁, 𝐶 ) ≥ 𝑓 (𝑁, 𝐶 ) 𝑘 ℓ 𝑖 ∪𝑗 𝑖 ∪𝑗 𝑖 ∪𝑗 𝑖 ∪𝑗 implies 𝑓 (𝑁 , 𝐶 ) ≥ 𝑓 (𝑁 , 𝐶 ) for all 𝐽 , 𝐽 ∈ 𝑁 ∖{ 𝐽 , 𝐽 }. 𝑘 ℓ 𝑘 ℓ 𝑖 𝑗 The idea behind invariance to aggregation is that any journal ranking problem can be transformed into a reduced problem by defining the union 𝐽 of journals 𝐽 and 𝐽 as 𝑖 ∪𝑗 𝑖 𝑗 follows: all citations between them are deleted, while any citations by/to these journals are summed up for the “aggregated” journal 𝐽 . This transformation is required to preserve 𝑖 ∪𝑗 the order of the journals not affected by the aggregation. Such an aggregation makes sense, for example, if one is interested only in the ranking of journals from a given field (e.g. economics journals) when journals from other disciplines can be considered as one entity. Invariance to aggregation is somewhat related to the consistency axiom of Palacios- Huerta and Volij (2004), which is also based on the notion of the reduced problem. However, our property probably takes the information from the missing journal in a more straightforward way into consideration. Invariance to aggregation has some connections to the famous independence of irrelevant alternatives (𝐼𝐼𝐴 ) condition, too, which is used, for example, in Arrow’s impossibility theorem (Arrow, 1951). Both axioms require an important aspect of the problem, the citations between two journals and the individual preferences between two alternatives, respectively, to remain fixed. However, there is a crucial difference: the set of alternatives (corresponding to journals) is allowed to change in the case of 𝐼𝐴 , while the preferences (corresponding to citations) are allowed to change in the case of 𝐼𝐼𝐴 . 3.2 Self-consistency This axiom, originally introduced in Chebotarev and Shamis (1997) to operators used for aggregating preferences, may require a longer explanation. First, some reasonable conditions are formulated for the ranking derived from any journal ranking problem. In particular, journal 𝐽 is judged better than journal 𝐽 if one 𝑖 𝑗 of the following holds: D1) 𝐽 has more favourable citation records against the same journals; 4 D2) 𝐽 has more favourable citation records against journals with the same quality; D3) 𝐽 has the same citation records against higher quality journals; D4) 𝐽 has more favourable citation records against higher quality journals. In addition, journals 𝐽 and 𝐽 should get the same rank if one of the following holds: 𝑖 𝑗 D5) they have the same citation records against the same journals; D6) they have the same citation records against journals with the same quality. Principles D2-D4 and D6 can be applied only after measuring the quality of the journals. The name of the property, self-consistency, refers to the fact that this is provided by the scoring procedure itself. The meaning of the requirements above is illustrated by an example. Figure 1: The journal ranking problem of Example 3.1 𝐽 𝐽 1 2 𝐽 𝐽 3 4 4 4 4 4 Example 3.1. Consider the journal ranking problem (𝑁, 𝐶 ) ∈ 𝒥 ∩𝒥 ∩𝒥 ∩𝒥 with 𝐵 𝑈 𝐿 𝐸 the following citation matrix: ⎡ ⎤ 0 1 1 0 ⎢ ⎥ 0 0 0 1 ⎢ ⎥ 𝐶 = ⎢ ⎥ . ⎣ ⎦ 0 0 0 1 0 0 0 0 This is shown in Figure 1 where a directed edge from node 𝐽 to 𝐽 indicates that journal 𝑖 𝑗 𝐽 has received a citation from journal 𝐽 . 𝑖 𝑗 Self-consistency has the following implications for the journal ranking problem presented in Example 3.1: ∙ 𝐽 ∼ 𝐽 due to rule D5. 2 3 ∙ 𝐽 ≻ 𝐽 because of rule D1 as 𝑐 > 𝑐 and 𝑐 > 𝑐 . 1 4 12 42 13 43 ∙ Assume for contradiction that 𝐽 ⪯ 𝐽 . Then 𝑐 > 𝑐 and 𝐽 ⪰ 𝐽 , as well as 1 2 12 21 2 1 𝑐 = 𝑐 and 𝐽 ∼ 𝐽 ⪰ 𝐽 ≻ 𝐽 , so rule D4 leads to 𝐽 ≻ 𝐽 , which is impossible. 13 24 3 2 1 4 1 2 Consequently, 𝐽 ≻ (𝐽 ∼ 𝐽 ). 1 2 3 ∙ Assume for contradiction that 𝐽 ⪯ 𝐽 . Then 𝑐 > 𝑐 and 𝐽 ≻ 𝐽 , as well as 2 4 21 43 1 3 𝑐 > 𝑐 and 𝐽 ⪰ 𝐽 ∼ 𝐽 , so rule D4 leads to 𝐽 ≻ 𝐽 , which is impossible. 24 43 4 2 3 2 4 Consequently, (𝐽 ∼ 𝐽 ) ≻ 𝐽 . 2 3 4 5 To conclude, self-consistency demands the ranking to be 𝐽 ≻ (𝐽 ∼ 𝐽 ) ≻ 𝐽 in Ex- 1 2 3 4 ample 3.1. It is clear that self-consistency does not guarantee the uniqueness of the ranking in general (Csat´ o, 2019a). Now we turn to the mathematical formulation of this axiom. Definition 3.1. Competitor set: Let (𝑁, 𝐶 ) ∈ 𝒥 be an unweighted journal ranking problem. The competitor set of journal 𝐽 is 𝑆 = {𝐽 : 𝑚 = 1}. 𝑖 𝑖 𝑗 𝑖𝑗 Journals in the competitor set 𝑆 are called the competitors of 𝐽 . Note that |𝑆 | = |𝑆 | 𝑖 𝑖 𝑖 𝑗 for all 𝐽 , 𝐽 ∈ 𝑁 if and only if the ranking problem is balanced. 𝑖 𝑗 The competitor set is defined only for unweighted journal ranking problem but self- consistency may have implications for any pair of journals which have the same number of matches. The generalisation is based on a decomposition of journal ranking problems. ′ 𝑛 Definition 3.2. Sum of journal ranking problems: Let (𝑁, 𝐶 ), (𝑁, 𝐶 ) ∈ 𝒥 be two journal ranking problems with the same set of journals 𝑁 . The sum of these journal ′ 𝑛 ranking problems is the journal ranking problem (𝑁, 𝐶 + 𝐶 ) ∈ 𝒥 . The sum of journal ranking problems has a number of reasonable interpretations. For instance, they can reflect the citations from different years, or by authors from different countries. According to Definition 3.2, any journal ranking problem can be derived as the sum of unweighted journal ranking problems. However, it might have a number of possible decompositions. (𝑝 ) 𝑛 Notation 3.1. Let (𝑁, 𝐶 ) ∈ 𝒥 be an unweighted journal ranking problem. The (𝑝 ) (𝑝 ) competitor set of journal 𝐽 is 𝑆 . Let 𝐽 , 𝐽 ∈ 𝑁 be two different journals and 𝑔 : 𝑖 𝑖 𝑗 (𝑝 ) (𝑝 ) 𝑆 ↔ 𝑆 be a one-to-one correspondence between the competitors of 𝐽 and 𝐽 . Then 𝑖 𝑗 𝑖 𝑗 (𝑝 ) (𝑝 ) (𝑝 ) (𝑝 ) g : {𝑘 : 𝐽 ∈ 𝑆 } ↔ {ℓ : 𝐽 ∈ 𝑆 } is given by 𝐽 = 𝑔 (𝐽 ). (𝑝 ) 𝑘 ℓ 𝑘 𝑖 𝑗 g (𝑘 ) Finally, we are able to introduce conditions D1-D6 with mathematical formulas. Axiom 2. Self-consistency (𝑆𝐶 ) (Chebotarev and Shamis, 1997): Scoring procedure 𝑓 : 𝑛 𝑛 𝒥 → R is called self-consistent if the following implication holds for any journal ranking problem (𝑁, 𝐶 ) ∈ 𝒥 and for any journals 𝐽 , 𝐽 ∈ 𝑁 : if there exists a decomposition of 𝑖 𝑗 the journal ranking problem (𝑁, 𝐶 ) into 𝑚 unweighted journal ranking problems – that ∑︀ (𝑝 ) (𝑝 ) 𝑛 is, 𝐶 = 𝐶 and (𝑁, 𝐶 ) ∈ 𝒥 is an unweighted journal ranking problem for all 𝑝 =1 𝑈 (𝑝 ) (𝑝 ) 𝑝 = 1, 2, . . . , 𝑚 – together with the existence of a one-to-one mapping 𝑔 from 𝑆 onto (𝑝 ) (𝑝 ) (𝑝 ) 𝑆 such that 𝑐 ≥ 𝑐 and 𝑓 (𝑁, 𝐶 ) ≥ 𝑓 (𝑁, 𝐶 ) for all 𝑝 = 1, 2, . . . , 𝑚 and (𝑝 ) (𝑝 ) 𝑘 𝑗 𝑖𝑘 g (𝑘 ) 𝑗 g (𝑘 ) (𝑝 ) (𝑝 ) (𝑝 ) 𝐽 ∈ 𝑆 , then 𝑓 (𝑁, 𝐶 ) ≥ 𝑓 (𝑁, 𝐶 ). Furthermore, 𝑓 (𝑁, 𝐶 ) > 𝑓 (𝑁, 𝐶 ) if 𝑐 > 𝑐 𝑘 𝑖 𝑗 𝑖 𝑗 𝑖 (𝑝 ) 𝑖𝑘 𝑗 g (𝑘 ) (𝑝 ) or 𝑓 (𝑁, 𝐶 ) > 𝑓 (𝑝 ) (𝑁, 𝐶 ) for at least one 1 ≤ 𝑝 ≤ 𝑚 and 𝐽 ∈ 𝑆 . 𝑘 𝑘 g (𝑘 ) 𝑖 In a nutshell, self-consistency implies that if journal 𝐽 does not show worse performance than journal 𝐽 on the basis of the citation matrix, then it is not ranked lower, in addition, it is ranked strictly higher when it becomes clearly better. Chebotarev and Shamis (1997) consider only loopless journal ranking problems but the extension of self-consistency is trivial as presented above. Chebotarev and Shamis (1998, Theorem 5) gives a necessary and sufficient condition for self-consistent scoring procedures, while Chebotarev and Shamis (1998, Table 2) presents some scoring procedures that satisfy this requirement. See also Csat´ o (2019a) for an extensive discussion of self-consistency. 6 4 The incompatibility of the two axioms In the following, it will be proved that no scoring procedure can meet axioms 𝐼𝐴 and 𝑆𝐶 . Figure 2: The journal ranking problems of Example 4.1 3∪4 3∪4 (b) Journal ranking problem (𝑁 ,𝐶 ) (a) Journal ranking problem (𝑁,𝐶 ) 𝐽 𝐽 𝐽 𝐽 1 2 1 2 3∪4 𝐽 𝐽 3 4 4 4 4 4 3∪4 3∪4 3 3 3 3 Example 4.1. Let (𝑁, 𝐶 ) ∈ 𝒥 ∩𝒥 ∩𝒥 ∩𝒥 and (𝑁 , 𝐶 ) ∈ 𝒥 ∪𝒥 ∩𝒥 ∩𝒥 𝐵 𝑈 𝐿 𝐸 𝐵 𝑈 𝐿 𝐸 be the journal ranking problems with the citation matrices ⎡ ⎤ ⎡ ⎤ 0 0.5 0.5 0 0 0.5 0.5 ⎢ ⎥ 0.5 0 0 0.5 ⎢ ⎥ ⎢ ⎥ 3∪4 𝐶 = ⎢ ⎥ and 𝐶 = ⎣ 0.5 0 0.5 ⎦ , respectively. ⎣ ⎦ 0.5 0 0 1 0.5 0.5 0 0 0.5 0 0 3∪4 3∪4 Journal ranking problem (𝑁 , 𝐶 ) is obtained by uniting journals 3 and 4. This is shown in Figure 2 where a directed edge from node 𝐽 to 𝐽 indicates that 𝑖 𝑗 journal 𝐽 has received a citation from journal 𝐽 , and an undirected edge between the 𝑖 𝑗 nodes means that the two journals are tied by mutual citations. Theorem 4.1. There exists no scoring procedure that is invariant to aggregation and self-consistent. Proof. The contradiction of the two properties can be proved by Example 4.1. Take first the journal ranking problem (𝑁, 𝐶 ), which has the competitor sets 𝑆 = 𝑆 = {𝐽 , 𝐽 } 1 4 2 3 and 𝑆 = 𝑆 = {𝐽 , 𝐽 }. Assume for contradiction the existence of a scoring procedure 2 3 1 4 𝑛 𝑛 𝑓 : 𝒥 → R satisfying invariance to aggregation and self-consistency. Self-consistency has several implications for the scoring procedure 𝑓 as follows: a) Consider the (identity) one-to-one correspondence 𝑔 : 𝑆 ↔ 𝑆 such that 14 1 4 𝑔 (𝐽 ) = 𝐽 and 𝑔 (𝐽 ) = 𝐽 . Then 𝑔 satisfies condition D1 of 𝑆𝐶 due to 14 2 2 14 3 3 14 𝑐 = 𝑐 = 0.5 and 0.5 = 𝑐 > 𝑐 = 0, thus 𝑓 (𝑁, 𝐶 ) > 𝑓 (𝑁, 𝐶 ). 12 42 13 43 1 4 b) Consider the (identity) one-to-one correspondence 𝑔 : 𝑆 ↔ 𝑆 such that 32 3 2 𝑔 (𝐽 ) = 𝐽 and 𝑔 (𝐽 ) = 𝐽 . Then 𝑔 satisfies condition D1 of 𝑆𝐶 due to 32 1 1 32 4 4 32 𝑐 = 𝑐 = 0.5 and 1 = 𝑐 > 𝑐 = 0.5, thus 𝑓 (𝑁, 𝐶 ) > 𝑓 (𝑁, 𝐶 ). 31 31 34 24 3 2 c) Suppose that 𝑓 (𝑁, 𝐶 ) ≥ 𝑓 (𝑁, 𝐶 ), which implies 𝑓 (𝑁, 𝐶 ) > 𝑓 (𝑁, 𝐶 ) according 2 1 3 4 to the inequalities derived in a) and b). Consider the one-to-one correspondence 𝑔 : 𝑆 ↔ 𝑆 such that 𝑔 (𝐽 ) = 𝐽 and 𝑔 (𝐽 ) = 𝐽 . Then 𝑔 satisfies 12 1 2 12 2 1 12 3 4 12 condition D3 of 𝑆𝐶 due to 𝑐 = 𝑐 = 0.5 and 𝑐 = 𝑐 = 0.5, thus 𝑓 (𝑁, 𝐶 ) > 12 21 13 24 1 𝑓 (𝑁, 𝐶 ), a contradiction. 7 Therefore 𝑓 (𝑁, 𝐶 ) > 𝑓 (𝑁, 𝐶 ) should hold, when invariance to aggregation results 1 2 ′ ′ ′ ′ in 𝑓 (𝑁, 𝐶 ) > 𝑓 (𝑁, 𝐶 ). However, self-consistency leads to 𝑓 (𝑁, 𝐶 ) = 𝑓 (𝑁, 𝐶 ) in the 1 2 1 2 ′ ′ ′ journal ranking problem (𝑁, 𝐶 ) because of the one-to-one mapping 𝑔 : 𝑆 ↔ 𝑆 such 1 2 ′ ′ that 𝑔 (𝐽 ) = 𝐽 and 𝑔 (𝐽 ) = 𝐽 : the assumption of 𝑓 (𝑁, 𝐶 ) > 𝑓 (𝑁, 𝐶 ) implies 12 2 1 12 1 2 1 2 ′ ′ 𝑓 (𝑁, 𝐶 ) < 𝑓 (𝑁, 𝐶 ) due to condition D3 (the competitors of 𝐽 are more prestigious), 1 2 2 ′ ′ ′ ′ while 𝑓 (𝑁, 𝐶 ) < 𝑓 (𝑁, 𝐶 ) would result in 𝑓 (𝑁, 𝐶 ) < 𝑓 (𝑁, 𝐶 ) due to condition D3 1 2 1 2 (the competitors of 𝐽 are more prestigious) again. Hence a scoring procedure cannot meet 𝐼𝐴 and 𝑆𝐶 at the same time. Since Example 4.1 contains balanced, unweighted, loopless, and extremal journal ranking problems, there is few hope to avoid the impossibility of Theorem 4.1 by plausible domain restrictions. Remark 4.1. 𝐼𝐴 and 𝑆𝐶 are logically independent axioms as there exist scoring proced- ures that satisfy one of the two properties: the least squares method is self-consistent (Chebotarev and Shamis, 1998, Theorem 5), and the flat scoring procedure, which gives 𝑓 (𝑁, 𝐶 ) = 0 for all 𝑖 ∈ 𝑁 and (𝑁, 𝐶 ) ∈ 𝒥 , is invariant to aggregation. 5 Conclusions We have presented an impossibility theorem in journal ranking: a reasonable method cannot be invariant to the aggregation of journals, even in the case of a substantially restricted domain of citation graphs. An intuitive explanation is that invariance to aggregation is a local property (it modifies only the citations directly affecting the journals to be united), while self-consistency considers the global structure of the citations as it depends on the quality of the journals. The clash between local and global axioms can also be observed in other fields, such as game theory. In addition, the impossibility result clearly shows that invariance to aggregation is a rather strong requirement, similarly to its peer independence of irrelevant alternatives (Malawski and Zhou, 1994). Nevertheless, according to our finding, the choice of the set of journals to be compared is an important aspect of every empirical study which aims to measure intellectual influence. It is clear that the axiomatic analysis discussed here has a number of limitations as it is able to consider indices from only one point of view (Gl¨ anzel and Moed, 2013). For example, the citation graph is assumed to be known, that is, the issue of choosing an adequate time window is neglected. In addition, this paper has not addressed several important problems of scientometrics such as the comparability of distant research areas, or the proper treatment of different types of publications. To summarise, the derivation of similar impossibility results may contribute to a better understanding of the inevitable trade-offs between various properties, and it means a natural subject of further studies besides axiomatic characterisations. Acknowledgements We are grateful to Gy¨ orgy Moln´ ar and D´ ora Gr´ eta Petr´ oczy for inspiration. Two anonymous reviewers provided valuable comments and suggestions on an earlier draft. The research was supported by OTKA grant K 111797 and by the MTA Premium Postdoctoral Research Program. 8 References Adachi, T. and Kongo, T. (2015). Further axiomatizations of Egghe’s 𝑔 -index. Journal of Informetrics, 9(4):839–844. Arrow, K. J. (1951). Social choice and individual values. Wiley, New York. Bouyssou, D. and Marchant, T. (2014). An axiomatic approach to bibliometric rankings and indices. Journal of Informetrics, 8(3):449–477. Bouyssou, D. and Marchant, T. (2016). Ranking authors using fractional counting of citations: An axiomatic approach. Journal of Informetrics, 10(1):183–199. Chambers, C. P. and Miller, A. D. (2014). Scholarly influence. Journal of Economic Theory, 151:571–583. Chebotarev, P. and Shamis, E. (1997). Constructing an objective function for aggregating incomplete preferences. In Tangian, A. and Gruber, J., editors, Constructing Scalar- Valued Objective Functions, volume 453 of Lecture Notes in Economics and Mathematical Systems, pages 100–124. Springer, Berlin-Heidelberg. Chebotarev, P. Yu. and Shamis, E. (1998). Characterizations of scoring methods for preference aggregation. Annals of Operations Research, 80:299–332. Csat´ o, L. (2015). A graph interpretation of the least squares ranking method. Social Choice and Welfare, 44(1):51–69. Csat´ o, L. (2019a). An impossibility theorem for paired comparisons. Central European Journal of Operations Research, 27(2):497–514. Csat´ o, L. (2019b). Some impossibilities of ranking in generalized tournaments. International Game Theory Review, 21(01):1940002. de la Vega, C. L. and Volij, O. (2018). Ranking scholars: A measure representation. Journal of Informetrics, 12(2):510–517. Demange, G. (2014). A ranking method based on handicaps. Theoretical Economics, 9(3):915–942. Duggan, J. and Schwartz, T. (2000). Strategic manipulability without resoluteness or shared beliefs: Gibbard-Satterthwaite generalized. Social Choice and Welfare, 17(1):85–93. Gibbard, A. (1973). Manipulation of voting schemes: A general result. Econometrica, 41(4):587–601. Gl¨ anzel, W. and Moed, H. F. (2013). Opinion paper: thoughts and facts on bibliometric indicators. Scientometrics, 96(1):381–394. K´ oczy, L. A. and Strobel, M. (2010). The world cup of economics journals: A ranking by a tournament method. IEHAS Discussion Papers 1018, Institute of Economics, Hungarian Academy of Sciences. Kongo, T. (2014). An alternative axiomatization of the Hirsch index. Journal of Informet- rics, 8(1):252–258. 9 Malawski, M. and Zhou, L. (1994). A note on social choice theory without the Pareto principle. Social Choice and Welfare, 11(2):103–107. Marchant, T. (2009). An axiomatic characterization of the ranking based on the h-index and some other bibliometric rankings of authors. Scientometrics, 80(2):325–342. Miroiu, A. (2013). Axiomatizing the Hirsch index: Quantity and quality disjoined. Journal of Informetrics, 7(1):10–15. Palacios-Huerta, I. and Volij, O. (2004). The measurement of intellectual influence. Econometrica, 72(3):963–977. Palacios-Huerta, I. and Volij, O. (2014). Axiomatic measures of intellectual influence. International Journal of Industrial Organization, 34:85–90. Perry, M. and Reny, P. J. (2016). How to count citations if you must. American Economic Review, 106(9):2722–2741. Quesada, A. (2010). More axiomatics for the Hirsch index. Scientometrics, 82(2):413–418. Quesada, A. (2011a). Axiomatics for the Hirsch index and the Egghe index. Journal of Informetrics, 5(3):476–480. Quesada, A. (2011b). Further characterizations of the Hirsch index. Scientometrics, 87(1):107–114. Satterthwaite, M. A. (1975). Strategy-proofness and Arrow’s conditions: Existence and correspondence theorems for voting procedures and social welfare functions. Journal of Economic Theory, 10(2):187–217. Woeginger, G. J. (2008a). An axiomatic analysis of Egghe’s 𝑔 -index. Journal of Informetrics, 2(4):364–368. Woeginger, G. J. (2008b). An axiomatic characterization of the Hirsch-index. Mathematical Social Sciences, 56(2):224–232.

Journal

Economics – arXiv (Cornell University)

Published: Apr 8, 2019

References

  • Further axiomatizations of Egghe's g-index
    Adachi