Which Are the Most Influential Cited References in Information?

Abstract

This bibliometric study presents the most influential cited references for papers published in the journal Information by using reference publication year spectroscopy (RPYS). A total of 30,960 references cited in 996 papers in the journal Information, published between 2012 and 2019, were analyzed in this study. In total, 29 peaks with 48 peak papers are presented and discussed. The most influential cited references are related to set theory and machine learning which is consistent with the scope of the journal. A single peak paper was published in the journal Information. Overall, authors publishing in the journal Information have drawn from many different sources (e.g., journal papers, books, book chapters, and conference proceedings).

1. Introduction

The MDPI journal Information covers a broad range of subject areas: Information theory and methodology (e.g., coding theory, information-theoretic security, quantum information, and philosophy/ethics of information), information systems (e.g., knowledge management, social media/social networks, and big data/cloud computing), information processes (e.g., digital signal processing, data mining, and information extraction), information applications, (e.g., man–machine interface, information in society/social development, and business process management), and information/communication technology (e.g., communication systems/networks, wireless sensor networks, and mobile communication services), see also [1]. The tenth anniversary of a journal is also an opportunity to look back at the papers published so far and which sources (i.e., cited references) were most influential. This study employs reference publication year spectroscopy (RPYS) [2] to achieve this goal.

In the first step of an RPYS analysis, a publication set is gathered. Such a publication set can comprise a scientific field (e.g., Higgs boson research [3], climate change [4], dentistry, and neurosciences [5,6], health equity [7], or density functional theory [8,9,10]), a journal (e.g., the journal Ecological Economics [11] and the journal FEMS Microbiology Letters [12]), or the oeuvre of a researcher (e.g., oeuvre of Eugene Garfield [13]). Reference publication year spectroscopy analysis focuses on the references cited in publications and, therefore, employs a backward view from the published papers, in contrast to a times cited analysis which employs a forward view. In the second step, the number of cited references (NCRs) for each of the reference publication years (RPYs) are plotted in a spectrogram. In the third step, the peaks in the spectrogram are inspected to find the most frequently cited references in addition to ordering the cited references by their NCR values. The peaks point to cited references which have been referenced very frequently in the initial publication set. By inspecting the earliest peaks, the historical roots of the initial publication set can be determined. The meaning of the cited references should be interpreted by an expert in the field.

2. Materials and Methods

Two bibliographic databases were used for the study: Web of Science (WoS, provided by Clarivate Analytics) and Scopus (provided by Elsevier). The MDPI journal Information is covered in WoS only since 2015 via the emerging sources citation index (ESCI). Scopus indexes the journal since 2012. However, the structure of the cited references is more systematic in WoS than in Scopus. Therefore, Scopus data were used for the time frame 2012–2014, and WoS data were used thereafter. The papers published in the MDPI journal Information were downloaded with cited references information on 29 October 29 2019.

We used the CRExplorer [14,15,16] (see also crexplorer.net) to convert the Scopus download in the WoS format to import the full publication set in CRExplorer. Nine hundred and six publications with 30,960 cited references were imported from the WoS download; 649 cited references without RPY were discarded. Ninety publications with 3551 cited references were imported from the Scopus download. The earliest cited references date back to the 17th century with René Descartes (1637, “Discourse on the Method”), but this reference was cited only once and, therefore, removed (see below). CRExplorer offers a feature to disambiguate cited references. This feature was used to merge equivalent cited references considering volume and page number with the Levenshtein threshold of 0.75. Afterwards, cited references occurring only once were removed to sharpen the spectrogram. Additionally, the cited references in peak years and the cited references occurring at least ten times were merged manually. This procedure led to a set of 2044 cited references within the time frame 1859–2019.

The NCR and the five-year median deviation of the NCR (years x, x − 1, x − 2, x + 1, and x + 2) were plotted. For the identification of peak years (RPYs with higher NCR values than the neighboring years), both curves were inspected. Cited references which were mainly responsible for the peak are referred to as peak papers in the following. The specific level of NCR values differed by publication set, citing year, and cited year. The NCR values should not be compared across RPYs.

3. Results

Figure 1 shows an overview of the RPYS spectrogram. The NCR values are shown as a bar chart with on overlay of the five-year median deviation as a blue line.

In the following, a selection of cited references is discussed which is mainly responsible for the observed peaks in the spectrogram. The interested reader can obtain the full list of 2044 cited references at the Supplementary Materials.

In the following, the RPYS results are divided into two parts: (i) the time frame 1858–1989 and (ii) the time frame 1990–2019.

3.1. RPYS Analysis for the Time Frame 1858–1990

The RPYS spectrogram for the time frame 1858–1990 is shown in Figure 2. Twenty-one peaks, some of them very small, can be observed in this time frame: 1859, 1902, 1928, 1935, 1937, 1943, 1946, 1948, 1950, 1954, 1957, 1959/1960, 1965, 1967, 1970, 1972, 1975, 1980, 1984, 1986, and 1989.

A selection of peak papers which are mainly responsible for the observed peaks in the spectrogram in Figure 2 is shown in

Table 1. Selection of peak papers which were mainly responsible for the observed peaks in the spectrogram in Figure 2.

CR Number	RPY	Cited Reference	NCRs
CR1	1859	Darwin, C., 1859, On the Origin of Species	4
CR2	1902	Muirhead, R.F., 1902, Proc. Edinburgh Math. Soc., V21, P144, DOI: 10.1017/S001309150003460X	2
CR3	1928	Hartley, R.V.L., 1928, Bell Syst. Tech. J., V7, P535, DOI: 10.1002/j.1538-7305.1928.tb01236.x	4
CR4	1928	Bohr, N., 1928, Nature, V121, P580	3
CR5	1935	Einstein, A., 1935, Phys. Rev., V47, P0777, DOI: 10.1103/PhysRev.47.777	3
CR6	1937	Turing, A.M., 1937, Proc. London Math. Soc., V42, P230	5
CR7	1943	McCulloch, W.S., 1943, Bull. Math. Biophys., V5, P115, DOI: 10.1007/BF02478259	2
CR8	1946	Ryle, G., 1946, Symposium: Why are the calculuses of logic and arithmetic applicable to reality? In Proceedings of the Logic and Reality, Symposia Read at the Joint Session of the Aristotelian Society and the Mind Association, Manchester, P20	2
CR9	1948	Shannon, C.E., 1948, Bell Syst. Tech. J., V27, P623, DOI: 10.1002/j.1538-7305.1948.tb00917.x	16
CR10	1950	Turing, A.M., 1950, Mind, V49, P433, DOI 10.1093/MIND/LIX.236.433	6
CR11	1957	Everett, H., 1957, Rev. Mod. Phys., V29, P454	4
CR12	1959	Popper, K.R., 1959, The Logic of Scientific Discovery	3
CR13	1960	Cohen, J., 1960, Educ. Psychol. Measurement, V20, P37, DOI: 10.1177/001316446002000104	3
CR14	1965	Zadeh L.A., 1965, Information and Control, V8, P338, DOI: 10.1016/S0019-9958(65)90241-X	38
CR15	1967	MacQueen, J., 1967, Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, P281	6
CR16	1970	Kuhn, T.S., 1970, The Structure of Scientific Revolutions	4
CR17	1970	Bellman, R.E., 1970, Management Science, V17, P141, DOI: 10.1287/MNSC.17.4.B141	3
CR18	1972	Anderson, P.W., 1972, Science, V177, P393, DOI: 10.1126/science.177.4047.393	3
CR19	1975	Zadeh, L.A., 1975, Information Sciences, V8, P199, DOI: 10.1016/0020-0255(75)90036-5	14
CR20	1980	Saaty, T., 1980, The Analytic Hierarchy Process	7
CR21	1980	Porter, M.F., 1980, Program: Electr. Lib. Inform. Sys., V14, P130, DOI: 10.1108/eb046814	5
CR22	1986	Atanassov, K.T., 1986, Fuzzy Sets Syst., V20, P87, DOI: 10.1016/S0165-0114(86)80034-3	32
CR23	1989	Atanassov, K.T., 1989, Fuzzy Sets Syst., V31, P343, DOI: 10.1016/0165-0114(89)90205-4	13

CR: Cited reference, RPY: Reference publication year, NCRs: Number of cited references.

and shortly discussed below. The earliest peak, and the only peak in the 19th century, was due to four references to Darwin’s classic that revolutionized natural science (in particular biology) and altered our understanding of the world (CR1).

In 1902, the Muirhead mean was introduced in CR2 which is the only cited reference below the first peak in the 20th century. The peak in 1928 was composed of two cited references (CR3 and CR4). In CR3, Hartley developed a quantitative measure of “information” which is based on physical, in contrast to psychological, considerations. In CR4, Bohr contributed general remarks to the discussion on the quantum postulate and the development of the atomic theory with the hope to harmonize the different views on the topic. The peak in 1935 was only due to the fact of three references to CR5 which has become known as the Einstein–Podolsky–Rosen (EPR) paradox. The authors (as most of their colleagues at the time) struggled with the quantum-mechanical description of physical reality.

The peak in 1937 was mainly due to the fact of Turing’s contribution (CR6) to the topic of computable numbers (i.e., real numbers which have expressions that are calculable by finite means) with an application to the Hilbertian Entscheidungsproblem (“decision problem”). The small peaks in 1943 and 1946 were composed of a single cited reference each, CR7 and CR8. In CR7, McCulloch and Pitts discussed a logical calculus of the ideas immanent in nervous activity. In CR8, Ryle, Lewy, and Popper suggested that a statement can be logically true (in the case of mathematical inference) but also falsifiable as a statement about the world (in the case of event interpretation and inference). In CR9, Shannon initiated classical information theory. The cited reference CR9 and another less frequently cited reference (therefore not included in Table 1) are the only cited references in the peak year 1948. Although the NCR value of 16 in Table 1 might seem low, CR9 is the sixth most cited reference in this RPYS analysis. In CR10, Turing proposed what is known today as the Turing test.

In CR11, Everett introduced what is known today as the many-worlds interpretation of quantum theory. The peak in 1959/1960 was mainly composed of two cited references, CR12 and CR13. The cited reference CR12 is the famous book about the philosophy of science by Popper. In CR13, Cohen proposed a coefficient of agreement for nominal scales. The peak in 1965 was mainly due to the fact of CR14, in which Zadeh proposed fuzzy sets (sets with a gradual assignment of the membership of elements in a set). The most cited reference below the peak in 1967 was CR15. In this conference paper, MacQueen proposed the clustering algorithm k-means.

The peak in 1970 was mainly due to the presence of two quite different cited references, CR16 and CR17. The cited reference CR16 is Kuhn’s famous book about the history of science with the distinction between normal science and scientific revolutions. In CR17, Bellman’s and Zahdeh’s contributed to the topic of fuzzy decision making. The most cited reference below the peak in 1972 was CR18. Here, Anderson discussed the broken symmetry and the nature of the hierarchical structure of science providing some examples thereof. The peak in 1975 was mainly due to the Zahdeh’s proposal of the concept of linguistic variables (CR19). The most cited references below the peak in and around 1980 were Saaty’s introduction of the analytic hierarchy process as an effective tool for dealing with complex decision making (CR20) and Porter’s proposal of an algorithm for automated suffix stripping (CR21).

The remaining two peaks in the time frame 1858–1990 were mainly due to the fact of Atanassov’s works on intuitionistic fuzzy sets. In CR22, Atanassov defined the concept of intuitionistic fuzzy sets. In CR23, Atanassov generalized the notion of intuitionistic fuzzy sets in the spirit of ordinary interval-valued fuzzy sets. The rather small peaks in 1954 and 1984 were composed of 5 and 6, respectively, different cited references which are cited only twice each. It would be arbitrary to pick one or a few of them. Also, it would not be useful to discuss all of these eleven cited references here.

3.2. RPYS Analysis for the Time Frame 1990–2019

The RPYS spectrogram for the time frame 1990–2019 is shown in Figure 3. Eight peaks, some of them very broad, can be observed in the spectrogram: 1994/1995, 1996–1999, 2000–2002, 2003–2005, 2006/2007, 2010–2012, 2013–2015, and 2016/2017. A selection of peak papers which were mainly responsible for the observed peaks in the spectrogram in Figure 3 is shown in

Table 2. Selection of peak papers which is mainly responsible for the observed peaks in the spectrogram in Figure 3.

CR Number	RPY	Cited Reference	NCRs
CR24	1994	Bengio, Y., 1994, IEEE Trans. Neural Networks, V5, P157, DOI: 10.1109/72.279181	6
CR25	1995	Kennedy, J., 1995, IEEE ICNN, VOL 4, P1942, DOI: 10.1109/ICNN.1995.488968	8
CR26	1995	Cortes C., 1995, Machine Learning, V20, P273, DOI: 10.1007/BF00994018	6
CR27	1996	Herrera F., 1996, Fuzzy Sets Systems, V78, P73, DOI: 10.1016/0165-0114(95)00107-7	7
CR28	1997	Hochreiter S., 1997, Neural Comput., V9, P1735, DOI: 10.1162/neco.1997.9.8.1735	19
CR29	1998	Smarandache F., 1998, Neutrosophy: Neutrosophic Probability, Set and Logic	12
CR30	1999	Smarandache F., 1999, A Unifying Field in Logics: Neutrosophy Logic	14
CR31	2001	Breiman L., 2001, Machine Learning, V45, P5, DOI: 10.1023/A:1010933404324	16
CR32	2001	Lafferty J., 2001, Proc. 18th Int. Conf. Mach. Learn., P282	9
CR33	2003	Blei D.M., 2003, J Mach. Learn. Res., V3, P993	9
CR34	2004	Lowe D.G., 2004, Int. J. Comput. Vision, V60, P91, DOI: 10.1023/B:VISI.0000029664.99615.94	9
CR35	2004	Wang Z., 2004, IEEE Trans. Image Proc, V13, P600, DOI: 10.1109/TIP.2003.819861	8
CR36	2004	Xu Z.S., 2004, Information Sciences, V168, P171, DOI: 10.1016/j.ins.2004.02.003	8
CR37	2005	Wang H., 2005, Interval neutrosophic sets and logic: Theory and applications in computing	11
CR38	2006	Demšar J., 2006, J. Mach. Learn. Res., V7, P1	7
CR39	2006	Bishop C., 2006, Pattern Recognition and Machine Learning	6
CR40	2010	Wang H., 2010, Multispace Multistructure, V4, P410	18
CR41	2011	Collobert R., 2011, J. Mach. Learn. Res., V12, P2493	11
CR42	2012	Logan R., 2012, Information, V3, P68, DOI: 10.3390/info3010068	9
CR43	2013	Mikolov T., 2013, Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, P3111	20
CR44	2013	Ye J., 2013, Int. J. Gen. Syst., V42, P386, DOI: 10.1080/03081079.2012.761609	12
CR45	2014	Pennington J., 2014, Proc. Conf. Emp. Meth. Nat. Lang. Proc. (EMNLP), P1532, DOI: 10.3115/V1/D14-1162	11
CR46	2014	Ye J., 2014, J. Intell. Fuzzy Syst., V26, P2459, DOI: 10.3233/IFS-130916	11
CR47	2015	LeCun Y., 2015, Nature, V521, P436, DOI: 10.1038/nature14539	10
CR48	2017	Krizhevsky A., 2017, Commun. ACM, V60, P84, DOI: 10.1145/3065386	10

CR: Cited reference, RPY: Reference publication year, NCRs: Number of cited references.

and shortly discussed below.

The first and smallest peak of the time frame 1990–2019 was mainly composed of three cited references which contributed to machine learning (CR24, CR25, and CR26). In CR24, Bengio, Simard, and Frasconi showed why gradient-based learning algorithms face an increasingly difficult problem in the case of long-term dependencies. In CR25, Kennedy and Eberhart introduced a concept for the optimization of non-linear functions using a particle swarm methodology. In CR26, Cortes and Vapnik proposed the popular support vector network model for machine learning.

The peak across RPYs 1996–1999 was mainly composed of four cited references (i.e., CR27, CR28, CR29, and CR30). In CR27, Herrera, Herrera-Viedma, and Verdegay present a consensus model in group decision making under linguistic assessments. In CR28, Hochreiter and Schmidhuber introduced a novel and efficient gradient-based method called long short-term memory. In CR29 and CR30, Smarandache introduced neutrosophic sets as generalization of fuzzy sets, intuitionistic fuzzy sets, and interval-valued intuitionistic fuzzy sets.

Mainly, three cited references (i.e., CR31, CR32, and CR33) were responsible for the peak across RPYs 2001–2003. In CR31, Breiman introduced the popular random forest methodology. In CR32, Lafferty, McCallum, and Pereira proposed the conditional random field model. In CR33, Blei, Ng, and Jordan described the latent Dirichlet allocation (LDA), a generative statistical model in natural language processing.

The cited references which were mainly responsible for the peak 2004–2006 were CR34, CR35, CR36, CR37, CR38, and CR39. In CR34, Lowe presented a method for extracting scale- and rotation-invariant features from images that can be used to perform reliable matching between different views of an object or scene. In CR35, Wang, Bovik, Sheikh, and Simoncelli introduced an alternative complementary framework for image quality assessment based on the degradation of structural information. In CR36, Xu proposed the concept of an uncertain linguistic variable. In CR37, Wang, Smarandache, Zhang, and Sunderraman introduced interval neutrosophic sets. In CR38, Demšar proposed the sign test. In CR39, Bishop introduced the cluster validity index which is capable of providing a quality measurement for the goodness of a clustering result for a data set.

With the peak 2010–2012, we enter the decade in which Information has been publishing so far. This peak was mainly composed of three cited references (i.e., CR40, CR41, and CR42), one of them being published in Information (CR42). In CR40, Wang, Smarandache, Zhang, and Sunderraman introduced single-valued neutrosophic sets. In CR41, Collobert and coworkers proposed a unified neural network architecture and learning algorithm for natural language processing. In CR42, Logan reviews the historic development of the concept of information including the relationship of Shannon information and entropy.

The peak 2013–2015 was mainly composed of five cited references (i.e., CR43, CR44, CR45, CR47, and CR47). Mikolov and co-workers proposed the popular Doc2vec methodology in CR43. In CR44, Ye presented the correlation and correlation coefficient of single-valued neutrosophic sets based on the extension of the correlation of intuitionistic fuzzy sets. In CR45, Pennington, Socher, and Manning proposed the GloVe method (global vectors for word representation). In CR46, Ye proposed a multi-criteria decision-making method using aggregation operators for simplified neutrosophic sets. In CR47, LeCun, Bengio, and Hinton provided a review about deep learning.

Finally, many cited references contributed to the peak 2016/2017. The most cited reference was CR48. Here, Krizhevsky, Sutskever, and Hinton presented large, deep convolutional neural networks to classify high-resolution images.

4. Discussion and Conclusions

Compared to other RPYS analyses of publication sets of a similar size (see for example Reference [17]), individual cited references of the current study have rather low NCR values. Instead of exhibiting large NCR values, more important cited references were found. This seems to reflect a rather broad basis on which the authors publishing in the journal Information have drawn on.

Overall, authors publishing in the journal Information have drawn from many different sources (e.g., journal papers, books, book chapters, and conference proceedings). Most publications the authors from Information have referenced are related to set theory and machine learning. This can be seen from the comparably high NCR values of the corresponding cited references. The majority of the most frequently cited references belong to the field of set theory. Publications dealing with natural sciences (e.g., quantum mechanics and evolutionary biology) and philosophical or historical aspects of science appear too but are cited much less often in papers published in the journal Information. The observed bias of cited references fits well with the journal’s scope of publishing papers related to information theory and methodology, philosophy/ethics of information, information systems, information processes, information applications, and information/communication technology.

Only a single peak paper mentioned in this study (i.e., CR42) was published in the journal Information itself. Several other articles published in Information appeared in the RPYS analysis as rather lowly cited references at this point. Considering the citation reports in Web of Science and Scopus, the journal Information receives many more citations from papers published in other venues.

In summary, one can conclude that the studies published in Information rely on a broad scientific basis with a thematic focus on set theory and machine learning.