ISSN 2707-0476 (Online)

University Library at a New Stage of Social Communications Development. Conference Proceedings, 2019, No IV

Scientific Library, Belarusian National Technical University (Minsk, Republic of Belarus),

ON THE POSSIBILITIES OF EVALUATING

PROPERTIES OF SCIENTIFIC DOCUMENTS

ON THE BASIS OF THEIR CITATIONS

COUNT (OR AGAIN: WHAT PROPERTY

IS REFLECTED BY CITATIONS COUNT

PAR EXCELLENCE, AFTER ALL?).

The objective is consideration of the recent works on the problem of the phenomenon reflected by citations count of scientific documents with a slant to the papers analyzing the citations count possibilities to reflect quality of cited research. The method: authoritative definitions of quality were under analysis and interpretation of corresponding research publications in the context of the results the mentioned analysis was undertaken. The results: citations count does not reflect quality of cited research as there are not causal relations between quality and citedness. The conclusions: since in practice the results of the citations count for documents value assessment and of peer assessment of their quality may mostly coincide, this makes it possible still to use citations count as a proxy indicator of quality. As for the “negative” citations, they seem to refer to cited items of high value, but of low quality.

Despite the confidence of a number of authors, including the author of this article, that citations count reflects the value of cited works, still a number of authors believe or take for granted that the reflected property is quality. “Even today, citation indicators are sometimes presented as measures of scientific quality”, state D.W. Aksnes, L. Langfeldt & P. Wouters (2019, p. 4). I would even say: rather frequently than sometimes. In 2018 I published a special paper that was aimed to refute the recent “citations count reflects quality” views (Lazarev, 2018), and I feel this paper already needs to be replenished! The most recent evidence of the prevalence of the “citations count reflects quality” opinion is the conference paper by M. Teplitskiy, E. Duede, M. Menietti, & K. Lakhani (2019) specifically aimed at the empirical refutation of this statement: the statement was treated as universally recognized, and its refutation – as a sensation.

To my mind, one might say that citation reflects the quality of the cited document only as about a conclusion from the normative theory of citation (mostly rejected by the scientific community), but we cannot mindlessly repeat “citation reflects quality” without looking into the vocabulary definitions of “quality” and without considering the essential characteristics of citing! However, this statement is repeated again and again – just because “it is accepted”, and also because it was contained in many works recognized as classical – as for example, the paper by S. Cole & J. R. Cole (1967). These papers apparently do not receive critical consideration, because no attention is paid to such wordings, for example, as “The number of citations is taken to represent the relative scientific significance or ‘quality’ of papers” (Cole & Cole, 1973, p. 21), though significance and quality are different properties (cf. the definitions of ‘quality’ stated below with the definitions of “significance” at https://www.lexico.com/-en/definition/significance)!

The prevalence of the “citations count reflects quality” viewpoint requires further consideration of the properties reflected by citations count involving the musings on the degree of mediation of such a reflection, as well as on the reflection of a certain property par excellence.

Different definitions of quality were analyzed and the results of the analysis were compared with widespread arguments supporting the “citations count reflects quality” idea and with the essence of citedness featured in the first part of this paper. Special attention was paid in this context to the most recent paper by D.W. Aksnes et al. (2019) as their authors distinguish several “dimensions” of quality, casting a rather unexpected light on a rather familiar understanding of this property.

Quality may be in all the cases defined as “degree of conformance to a standard” (Gove, 1993, p. 1858) (including speculative “standards” that might exist only in the evaluator’s mind and not be formulated in a documentary format) or to requirements (Sistemy menedzhmenta…, 2015, p. 15). In the international standard ISO 5127:2017(en) “Information and documentation – Foundation and vocabulary” it is straightforwardly stated that quality is the “entirety of features and characteristics <…> of a product or service <…> that bear on its ability to satisfy stated or implied needs”. (Mind, that “implied” means not necessarily formulated in a documentary form!) Also, in my paper (Lazarev, 1996, p. 274), having grounded on the definitions of (Radlov, 1904, p. 127) and of (Encyclopedia Britannica, 1969, p. 915), I maintained that “quality” is cognized rather abstractly, not in the process of (or in relation to) the use or satisfaction of the concrete needs, strongly depending on a cognizing subject (Radlov, 1904, p. 127) and with the aid of some ideal standards” (Encyclopedia Britannica, 1969, p. 915).

If somebody thinks that too old definitions were used in the cited paper (Lazarev, 1996, p. 274), let us immediately consult just the current one. Here it is: “Quality is a judgment of how excellent something or someone is” (https://www.lexico.com/en/definition/significant). So, it is seen from the definitions that, by its very nature, quality of a scientific document or a collection of documents is a property that is quantitatively characterized by expert evaluation (peer reviewing). At the same time, as it is was shown in Part 1, by its very nature, citations count is primarily a “measure” of the use of scientific documents which, in turn, indirectly reflects their value. Citations count has no causal relationships with quality (Bornmann & Haunschild, 2017; Ricker, 2017; Lazarev, 2018) – it is so simple!^¹ Acknowledgment or credit “contributions by others” “covers only part of the dynamics” <of citing>^² (Aksnes et al., 2019, p. 5) and, even when they do, they are just motivations of citing, not its reason, while quality is just the consequence of acknowledgment or credit!

Some experts believe, however, that “a peer evaluation may involve assessments of factors besides scientific quality” (Aksnes et al., 2019, p. 7). But what other factors could be “besides scientific quality” if conformance to the requirements (which is assessed) is quality by its definition?! No matter which requirements are formulated, –conformance to the requirements (or “stated or implied needs”) is quality.

As related to exactly scientific documents, very popular, e.g., is the following definition: “‘Quality’ is a property of the publication and the research described in it. It describes how well the research has been done, whether it is free from obvious ‘error’ <…>, how original the conclusions are, and so on” (Martin & Irvine, 1983, p. 70)^³. It is clear that the mentioned “constituents” of quality are not reflected by the citations of publications as “a “mistaken” publication can have a large impact by stimulating further research” (Aksnes et al., 2019, p. 10) – everything is exactly the opposite, – while, a well-done paper may just be irrelevant to the research of a potential citer and, accordingly, not being cited at all!

In the recent paper (Aksnes et al., 2019), the following “elements” of quality are mentioned: “originality, significance, rigor, impact, vitality, and sustainability” (Aksnes et al., 2019, p. 7). Also, on page 8 of the same paper the existence of the following “dimensions” of quality is asserted: “plausibility, <…> originality, <…> scientific value, <…> societal value”.

Significance, impact, value and quality are apparently different standalone properties: one can verify this statement by a simple comparison of their definitions cited in Parts 1 and 2 of the present paper (except of “impact”, an overview of the definitions of which was presented in (Lazarev, 2019). Of course, in accordance with a general definition of quality, any other properties can be “substituted” as requirements, but will this not “blur” all the semantic boundaries, the intuitive clarity of assessments, the clearness of their perception?! And in any case, the semantic differences between value comprehended through use and quality comprehended through a priori comparison with the “standard” are too great.

Based on their review of the literature, D.W. Aksnes et al. (2019) tried to consider the possibilities of using citation counts to evaluate listed “dimensions of research quality”.

Their conclusions are the following: as for the plausibility, “it seems unlikely that citations can be seen as valid indicators of the solidity of the publications” (Aksnes et al., 2019, p. 9); as for originality, “no simple relationship between originality or novelty and citations” was found (Aksnes et al., 2019, p. 9); as for “societal value and relevance” – to their mind, it is poorly reflected by citation counts (Aksnes et al., 2019, p. 10-11). And as far as for scientific value, again, it is absolutely different and standalone property that is difficult to imagine being included in the list of “requirements” of quality… As D. W. Aksnes et al. (2019) believe, “scientific value and significance <which are different properties, according to their very definitions! – V.S.L.> are dimensions of the quality concept to which some citations may most directly relate. This is commonly argued as follows. When a scientist refers to a paper, it has been useful or relevant <different notions – V.S.L.> in some way for the present research or for the writing of the publication. Thus, frequently cited articles may be assumed to have been more useful than publications that are hardly cited or not at all, and possibly be more useful and thus important in their own right <…>. This means that the number of citations may be considered as a measure of the article’s usefulness, impact, or influence <which are also different properties, according to their very definitions! Again, see the paper V.S. Lazarev (2019) – V.S.L.> on other research” (p. 10).

What is the subject matter of this passage? Is it value, significance, usefulness, relevance, impact or performance?! Having “cleared” the terminological and semantic confusion of this fragment, we come to seeing the authors’ involuntary recognition that the citation reflects directly the use and – through it – indirectly reflects the value of cited documents! But this – I repeat it once again – is a completely different property. As for the other “dimensions” of quality, D.W. Aksnes et al. (2019) confirm themselves that the citations count is in principle unsuitable for their evaluation.

Yet, a number of times correlation was revealed between the number of citations to the collections of documents and the results of their peer reviewing (Virgo, 1977; Lawani & Bayer, 1983; Rinia, van Leeuwen, van Vuren, & van Raan, 1998 and many more). Due to this correlation, we can assume that the quality of scientific documents might still be also evaluated by the citedness level. However, such an evaluation is purely probabilistic, not causal. Citations do not confirm quality of cited works; the portion of citations may just coincide with the portion of corresponding expert evaluations. Documents value is reflected by citations count par exellence, while the “reflection” of quality is just a matter of statistics. This becomes possible due to the often practical coincidence of these properties. However, the results obtained by J. Nicolaisen (2002) demonstrate deviations in the corresponding distribution supposedly caused by “negative citation”. If the cause is really this, then these deviations point to very interesting documents that are valuable, but not qualitative: when analyzing works that then receive “negative references”, the citing author finds new, additional, developing arguments to present his view of the problem, being “provoked” by the publication that later received a “negative reference” from him. This “provocative stimulation” determines the value of the “negatively” cited works, the value that is confirmed by its citedness figures. But as for the quality of such works, its assessment will be quite different. Hence, “negative” citations seem to refer to items of high value, but of low quality.

So, the presence of the studies which results show a good correlation between the results of the analysis of documents citedness and documents expert evaluation does not indicate the adequacy of the method of citation analysis to assess the quality, but demonstrate a certain coincidence of the assessments of the properties of “value” and “quality” in practice and, apparently, about proximity (but not identity!) of the essences of value and quality. Since in most of the studies that I know such correlation is invariably confirmed, the citation analysis can be successfully applied to assess the quality of a large number of scientific documents in cases where it is technically unacceptable to conduct an expert evaluation. It is only necessary to understand that it actually reflects – due to cause-and-effect relationships – not quality, but value. However, in practice these properties differ from each other relatively rarely (it is quite confident to talk about their practical mismatch only in cases of “negative citations”).

In the present paper, I did not consider the concept of impact (which is an unnecessary concept as applied to citations count, as I believe and as I tried to demonstrated in (Lazarev, 2019), and the problem of the reliability of the evaluation of use of documents by citations count as compared with the reliability of the same evaluation by assessing readers’ activity in search, queries, bookmarking documents (which may be another way to assess documents value). It is because the first problem is reflected in (Lazarev, 2019), while the second one – in (Lazarev, 2017, p. 8-10).

So, I have considered the most recent work on the problem of the phenomenon reflected by the documents citedness which was based on highlighting the various “dimensions” of the quality of cited documents and on attempts to analyze the possible associations between the citedness of research works and the various “dimensions” of their quality (Aksnes et al., 2019) in the context of analysis of both quality definitions and of some other papers devoted to assessment quality by citations count. This led to the confirmation of the following statement: citation counts reflect documents value par excellence and do not have any causal relations with their quality. However, in practice the results of the citations count for documents value assessment and of peer assessment of their quality can mostly coincide. This makes it possible to use citations count as a proxy indicator of quality documents. (As for the “negative” citations, they seem to refer to items of high value, but of low quality.)

Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019). Citations, citation indicators, and research quality: an overview of basic concepts and theories. SAGE Open, January-March, 1-17. doi: 10.1177/2158244019829575

Bornmann, L., & Haunschild, R. (2017). Does evaluative scientometrics lose its main focus on scientific quality by the new orientation towards societal impact? Scientometrics, 2(2), 937-943. doi: 10.1007/s11192-016-2200-2

Cole, J. R., & Cole, S. (1973). Social stratification in science. Chicago, IL: The University of Chicago Press.

Cole, S., & Cole, J. R. (1967). Scientific output and recognition: a study in the operation of the reward system in science. American Sociological Review, 32(3), 377-390.

Gove, Ph. B. (Ed.-in-Chief). (1993). Webster’s Third New International Dictionary of the English Language. Unabridged. Cologne: Köneman.

Lawani, S. M., & Bayer, A. E. (1983). Validity of citation criterion for assessing of scientific publication: new evidence with peer assessing. Journal of American Society for Information Science, 34(1), 59-66. doi: 10.1002/asi.4630340109

Lazarev, V. S. (1996). On chaos in bibliometric terminology. Scientometrics, 35(2), 271-277. doi: 10.1007/BF02018485

Lazarev, V. S. (2018). Is it right to consider the level of citations to scientific papers as the indicator of their quality? Scientometrics: methodology, tools, practical application, 88-103. Retrieved from https://rep.bntu.by/handle/data/37342 (in Russian, with English abstract)

Lazarev, V. S. (2019). Insufficient definitions or a vaguely grasped notion? On definitions of “impact”. Scholarly Research and Information, 2(1), 63-78. doi: 10.24108/2658-3143-2019-2-1-63-78 (in Russian, with English abstract)

Martin, B. R., & Irvine, J. (1983). Assessing basic research: Some partial indicators of scientific progress in radio astronomy. Research Policy, 12(2), 61-90.

Nicolaisen, J. (2002). The J-shaped distribution of citedness. Journal of Documentation, 58(4), 383-395. doi: 10.1108/00220410210431118

Radlov, L. (Ed.). (1904). Filosofskiy slovar: logika, psikhologiya, etika, estetika i istoriya filosofii. St.-Petersburg: Brokgauz - Efron. (in Russian)

Ricker, M. (2017). Letter to the Editor: About the quality and impact of scientific articles. Scientometrics, 111(3), 1851-1855. doi: 10.1007/s11192-017-2374-2

Rinia, E. J., van Leeuwen, T. N., van Vuren, H. G., & van Raan, A. F. J. (1998). Comparative analysis of a set of bibliometric indicators and central peer review criteria: Evaluation of condensed matter physics in the Netherlands. Research Policy, 27, 95-107. doi: 10.1016/S0048-7333(98)00026-2

Tahamtan, I., &·Bornmann L. (2019). What do citation counts measure? An updated review of studies on citations in scientific documents published between 2006 and 2018. Scientometrics, 121(3), 1635-1684. doi: 10.1007/s11192-019-03243-4

Teplitskiy, M., Duede, E., Menietti, M., & Lakhani, K. (2019). What does citations measure: evidence from citers. Proceedings of the 17^th Conference of the International Society for Scientometrics and Informetrics, September 2-5, 2019, Rome, Vol. 2, pp. 2440-2441. Retrieved from http://issi-society.org/publications/issi-conference-proceedings/

Thornley, C., Watkinson, A., Nicholas, D., Volentine, R., Jamali, H. R., Herman, E., … Tenopir, C. (2015). The role of trust and authority in the citation behaviour of researchers. Information Research, 20(3). Retrieved from http://InformationR.net/ir/20-3/paper677.html

Virgo, J. A. (1977). A statistical procedure for evaluating the importance of scientific paper. The Library Quarterly, 47(4), 415-430. doi: 10.1086/620723

Наукова бібліотека, Білоруський національний технічний університет (Мінськ, Республіка Білорусь), e-mail vslazarev@bntu.by, ORCID 0000-0003-0387-4515

ПРО МОЖЛИВОСТІ ОЦІНКИ ВЛАСТИВОСТЕЙ

НАУКОВИХ ДОКУМЕНТІВ НА ОСНОВІ ЇХ

ЦИТОВАНОСТІ (АБО ЗНОВУ ПРО ТЕ,

ЯКА Ж, НАРЕШТІ, ВЛАСТИВІСТЬ

ВІДОБРАЖУЄТЬСЯ В ЦИТОВАНОСТІ

PAR EXCELLENCE?). ЧАСТИНА 2: ЯКІСТЬ?

Метою є розгляд свіжих робіт з проблеми феномена, що відображується кількістю цитувань наукових документів, з ухилом на статті, котрі аналізують можливості кількості цитувань для відображення якості цитованого дослідження. Методи. Були проаналізовані авторитетні визначення якості; в контексті проведеного аналізу було здійснено інтерпретацію відповідних наукових публікацій. Як результат було встановлено, що кількість цитувань не відображує якості цитованого дослідження, оскільки причинно-наслідкові зв'язки між якістю й цитованістю відсутні. Висновки. Через те, що на практиці результати підрахунку цитувань для оцінки цінності документів та експертної оцінки їх якості можуть в основному збігатися, можливо, як і раніше, використовувати підрахунок цитувань в якості непрямого, що заміщує (proxy), показника якості. Що стосується "негативних" цитувань, то вони, мабуть, відносяться до цитованих документів високої цінності, але низької якості.

Ключові слова: кількість цитувань; властивості; якість; цінність

1 Apropos, in the paper (Thornley et al., 2015) cited by the review by I. Tahamtan &·L. Bornmann (2019), it is noted, that “the <citers’> responses that could be grouped under normative reasons account for 19.18%, so these are in the minority” – and this looks as one more empirical blow to the popular normative citation theory on which the idea of the “quality” of the cited document as a property reflected by its citedness is based. Also, the paper by C. Thornley et al. (2015) stated that “high levels of citation are an indicator of trust and authority, which is at least a necessary, if not sufficient, condition for impact, but are too varied in meaning to be used as a sole measurement for quality” (italicized by me – V.S.L.).

2 It is implied here that, “according to Merton’s view, the norms of science oblige researchers to cite the work upon which they draw, and in this way acknowledge or credit contributions by others” (Aksnes et al., 2019, p. 5).

https://doi.org/10.15802/unilib/2019_187406