SentenceLDA- and ConNetClus-Based Heterogeneous Academic Network Analysis for Publication Ranking

Abstract

Scientific papers published in journals or conferences, also considered academic publications, are the manifestation of scientific research achievements. Lots of scientific papers published in digital form bring new challenges for academic evaluation and information retrieval. Therefore, research on the ranking method of scientific papers is significant for the management and evaluation of academic resources. In this paper, we first identify internal and external factors for evaluating scientific papers and propose a publication ranking method based on an analysis of a heterogeneous academic network. We use four types of metadata (i.e., author, venue (journal or conference), topic, and title) as vertexes for creating the network; in there, the topics are trained by the SentenceLDA algorithm with the metadata of the abstract. We then use the Gibbs sampling method to create a heterogeneous academic network and apply the ConNetClus algorithm to calculate the probability value of publication ranking. To evaluate the significance of the method proposed in this paper, we compare the ranking results with BM25, PageRank, etc., and homogeneous networks in MAP and NDCG. As shown in our evaluation results, the performance of the method we propose in this paper is better than other baselines for ranking publications.

1. Introduction

Scientific papers, as the carrier of knowledge dissemination, play the role of “standing on the shoulders of giants” [1] With the development of Internet technology, the increase in scientific papers being published in the digital form not only helps avoid resource waste caused by the printing process but also promotes the speed and scope of knowledge dissemination. However, the rapid development trend of digital papers has introduced new challenges for scientific resources management. One estimate puts the count at 1.8 million articles published each year in about 28,000 journals [2] Digitally published articles foster more opportunities for information retrieval but make it harder to find the appropriate resource. Therefore, more and more people have focused on the methodology of evaluating scientific papers to improve the efficiency of information retrieval, eliminate the influence of non-relevant results, and satisfy the user’s information needs. Fortunately, research on a publication ranking algorithm is one of the effective ways to solve this problem.

In fact, many studies have focused on methods of scientific publication ranking. Citation counts [3] are the most popular method (that is, the ranking result is determined by the count of citations). This method assumes that the more citations a paper has, the higher its influence tends to be; thus, the paper is ranked higher. In addition, there is one publication ranking method based on the impact factors [4] of the venue or a paper published in a journal or a conference. If the venue has a higher impact factor, the paper published in it will also have a high influence. Besides that, the author’s reputation will also determine the ranking result; that is, a paper written by well-known scholars in the field is often ranked higher than others [5] Currently, the methods mentioned above are widely used in academic paper retrieval institutions or search engines as the basic or core algorithm for scientific paper searching. However, the above methods also have limitations when used for evaluating publications. For example, self-citation may confuse the result of citation counts, which affects the evaluation of author authority. Furthermore, the recognition of the venue affects the evaluation results of scientific papers.

Fortunately, more and more methods (e.g., machine learning, text mining, network and graph, etc.) are used in bibliometrics for ranking. PageRank [6] algorithms rely on graph theory, which promotes the research of ranking and evaluating scientific papers. However, most methods only adopt a single type of network structure and neglect the influence of different types of vertexes for ranking. The single type of network structure may include a publication network by citation relationship, keyword network by co-occurrence relationship, author network by co-author relationship, etc.

Therefore, in this paper, we propose a publication ranking method based on heterogeneous academic network analysis. The metadata that affect the evaluation of scientific papers are regarded as the vertexes in the heterogeneous network (i.e., title, topic, keyword, author, venue, abstract and full text, etc.). We then use the SentenceLDA algorithm for training topics by the metadata of the abstract, and the Gibbs sampling method is proposed for creating the heterogeneous network (also known as TVKA). Finally, we apply the ConNetClus algorithm to calculate the probability value of publication ranking.

As one contribution of this paper, we propose a method for creating a heterogeneous academic network, which is topic-sensitive with four types of vertexes. The other contribution is that we use the ConNetClus algorithm to calculate the probability value in the heterogeneous network as the publication ranking result, which we compare with the classical ranking algorithm. As the result shows, the method we use in this paper is better than other baselines.

2. Literature Review

2.1. Publication Ranking

The current rapid access to digital publications can greatly accelerate research, but the information overload also challenges researchers, especially junior researchers, in finding appropriate citations for their research projects. In previous papers, to deal with this problem, bibliometric methods have focused on ranking publications by citation analysis [7] along with graph mining [8] A common and easy method based on citation frequency or citation counts has been studied for a long time. Nevertheless, there is a basic assumption in previous methods, which is easy and straightforward: if paper1 cites paper2, then paper1 and paper2 are related. This assumption is oversimplified because it treats all citations equally, regardless of sentiment, reason, topic, or motivation. Cronin [9] reviewed a variety of perspectives on citations and argued that citations have multiple articulations in that they inform their understanding of the socio-cultural, cognitive, and textual aspects of scientific communication. Besides that, Egghe and Rousseau [10] discussed the influence of the author’s authority on citation analysis from the view of the author’s point in their paper. In recent years, there are still many scholars working in this field. Abrishami and Aliakbary [11] proposed a novel method for predicting long-term citations of a paper based on the number of its citations in the first few years after publication. Small et al. [12] from a new interpretation of citation counts, found that early citations made soon after publication are more hedged than later citations.

With the development of network and graph theory, the bibliometric method has also been improved in theory. Larson [13] first presented the concept of bibliometric methods based on the WWW (World Wide Web) and has explicitly introduced the network theory into bibliometrics since 1996. The HITS (hyperlink induced topic search) algorithm was first proposed to calculate the weight of network vertexes [14] and the PageRank algorithm proposed by Google is the most widely used algorithm to solve the problem of web page ranking in the network [6].

However, PageRank is insensitive toward topics and queries. Haveliwala [15] proposed a topic-sensitive PageRank algorithm, computing a set of PageRank vectors using a set of representative topics. After that, more and more researchers began using PageRank or improved the PageRank algorithm in the field of bibliometrics for measuring publication and author importance. Qiao et al. [16] combined the characteristics of the citation network in their paper and proposed an alternative method for evaluating the value of a paper based on an improved PageRank algorithm. Reference [17] proposed an alternative hybrid algorithm based on an optimized normalization technique and content-based approach to overcome the setbacks of PageRank. Since the algorithm of the topic model (Latent Dirichlet Allocation, LDA) [18] was proposed, scholars have tried to combine this algorithm with the PageRank method to avoid PageRank’s deficiencies toward the topic. To receive paper recommendations, Tao et al. [19] calculated the probability distribution according to the LDA of papers and used the PageRank algorithm to reorder the options in the reference network to obtain the final recommendation results. Zhang et al. [20] proposed a pipeline model, named collective topical PageRank, to evaluate the topic-dependent impact of scientific papers.

2.2. Factors for Evaluating Scientific Publications

The metadata of scientific papers (i.e., title, author, abstract, keyword, full text, reference, venue, topic, and publication time) affected the evaluation result of scientific publications [21]. In this paper, we divide these metadata into two categories: internal and external factors.

The external factors refer to the objective elements other than the content of the paper; these factors decide the ranking results in a scientific paper’s evaluation. Although the external factors do not directly determine the article’s academic influence, they can provide an indirect evaluation reference basis for them. There are seven categories of external factors divided from the research on citation incentives, whereas four factors are related to the evaluation results of academic influence (i.e., venue impact, author authority, citation, and co-author factors). Regarding the venue impact factor, there is a strong common perception that high-ranking journals publish “better” or “more important” science. The assumption is that high-ranking journals are able to be highly selective and publish only the most important, novel, and best-supported scientific discoveries [22] In terms of the author authority factor, it is generally believed that papers published by authors with high popularity in specific disciplines or those who came from famous institutions will be ranked higher. “h-index” is used to quantify the research output of individual scientists [23] which is increasingly important to calculate and is used to precisely interpret the h-index along with the rapid evolution of bibliographic databases [24] Regarding the citation factor, it is generally considered that if a paper is cited more times, it often means its academic level is higher and influence is greater. Citation counts are the highest-weighted factor in Google Scholar’s ranking algorithm. Highly cited articles are found significantly more often in higher positions than articles that are cited less often [25] In terms of the co-author factor, scientific research cooperation is a scientific activity carried out by researchers to achieve their common purpose of producing new scientific knowledge. Reference [26] examined the association of author bibliographic coupling strength and citation exchange in 18 subject areas.

Internal factors refer to the correlation between the content quality of scientific papers and their academic influence. The content quality of a published paper is the most important factor in evaluating its academic level. However, it is very difficult to quantitatively analyze the content of papers. More and more researchers are focusing on abstract or full-text content analysis plus machine-learning algorithms to evaluate the contributions of a paper. Most traditional content analysis is implemented using a keyword-matching process; thus, it cannot consider the semantic contexts of items [27] As full-text publication data provide crucial content for capturing the different methods of data use, Reference [28] employed a content-analysis method to a multidisciplinary full-text corpus to gain insights into dataset mentions and citations. Reference [29] extracted data from many full-text publications and represented the results by a probability distribution over a set of predefined topics to give a new method to enhance scholarly networks.

2.3. Heterogeneous Network

The heterogeneous network was originally used in communication and belongs to a type of network. That network is composed of computers, systems, and network devices. In most cases, different protocols run on different applications or functions. The origin of heterogeneous networks can be traced back to the BARWAN (Bay Area Research Wireless Access Network) project initiated by the University of California in 1995. In [30] R.H. Katz, the leader of this project, integrated different types of networks that overlap each other to build a heterogeneous network to meet different types of business needs. It can be seen that heterogeneous networks are mostly used to study wireless communication technology. At present, many pieces of research have been established in the frontier direction of data mining, namely, heterogeneous information networks. This method of peeling off other objects and only caring about the relationship between the research objects simplifies ideas well, but it causes a significant loss of information. In today’s complex network, there are many kinds and types of connections between objects. At this time, introducing the heterogeneous information network can mine the potential value of data without losing the relationship between objects.

The heterogeneous network can be used to implement a collaborative filtering recommendation algorithm, search-ranking algorithm, abnormal data detection, and so on. Zhang et al. [31] introduced user and item attribute information and integrated the scoring matrix into the heterogeneous information network, which effectively improved the recommendation accuracy. Wang et al. [32] considered using the heterogeneous features of a 3D model that interrelated co-clustering. This method used limited semantic information—which comes from multi-channel information—to build a heterogeneous semantic network, and then converted the heterogeneous semantic network to the semantic features of a 3D model. Zhang et al. [33] proposed an improved meta path-based collaborative filtering algorithm for weighted heterogeneous information networks. The method was evaluated with the extended MovieLens dataset, and experimental results showed that our approach outperformed several traditional algorithms and made the result of recommendation more accurate. Mu et al. [34] used complex heterogeneous information network simulation experiments to detect abnormal data so that the algorithm had a high anti-interference ability. Reference [35] proposed a method of combining meta-path selection and user preference Top-k correlation query on heterogeneous information networks to measure the correlation between different types of objects.

There are many different types of vertexes in a heterogeneous network, some of which are more important. Such vertexes often affect the structure and function of the network to a greater extent. In recent years, more and more people have been paying attention to the ranking method of vertex importance, not only because of the theoretical research significance of this method but also because of its strong practical application value [36] Among the ranking results-based algorithms, most of the current ranking functions are defined based on homogeneous networks, such as PageRank and HITS, and there are also some ranking algorithms based on heterogeneous networks, such as dual-type rankClus [37] multi-type netClus [38] and other algorithms [39] In sorting methods such as heterogeneous networks, the relationship between different types of objects and objects of the same type is important. By moving from a single homogeneous network level to the analysis of multiple types of heterogeneous network objects, the analysis of sorting algorithms is also transformed from a specific webpage to an object-oriented level of operation. However, the netClus algorithm cannot analyze references between target objects, and the network used must be a star network, which has limitations, so it leads to the optimization method that we use, and at the same time, introduces the calling relationship between homogeneous networks papers. There are increasingly more types of objects in the network, and research using a pure homogeneous network can no longer express the relationship between different types of objects. Therefore, a heterogeneous network can better reflect relationships and provide a large number of opportunities for mining research. Heterogeneous network analysis is essential to studying the relationship between different types of objects.

3. Methodology

3.1. SentenceLDA for Training Topics based on the Abstract

LDA [18] is a generative probabilistic model for collecting discrete data, such as text corpora. LDA is a three-level hierarchical Bayesian model in which each item of a collection is modeled as a finite mixture over an underlying set of topics. The limitations of LDA stem from the use of the Dirichlet distribution to model the variability among the topic proportions. One assumption is the interchangeability among words, which ignores the coherence of contents. In fact, the sentences in a text are coherent, so there is a logical relationship between sentences. We also hope that only a limited number of potential topics will appear after each sentence is processed. Therefore, we use SentenceLDA [40] as an extension of LDA, which aims to overcome this limitation by incorporating the structure of the text in the generative and inference processes.

An abstract is a brief summary of a research article, thesis, review, conference proceeding, or any in-depth analysis of a particular subject and is often used to help the reader quickly ascertain the paper’s purpose. The abstract, as one of the metadata of the publication, is the embodiment of the content of the paper, regarded as the internal factor for evaluating publications.

Considering the content of an abstract is text data, we must transform it into smaller granularity topics. This means that the content presented in the abstract will be displayed in the form of topic distributions. On the other hand, abstracts are composed of sentences, and there is a logical relationship among those sentences, so we use SentenceLDA for training topics based on the abstract to avoid the limitations of LDA.

Therefore, abstract $A$ can be expressed as the distributions of topics:

$$A=\left\{topi{c}_1,topi{c}_2,\dots \dots topi{c}_n\right\}$$

(1)

where $topi{c}_i$ represents a topic trained from the topic model, and n is the number of topics. The probability of each topic is $scor{e}_i$. This means each topic ($topi{c}_i$) has its own weight score ($scor{e}_i$) and ${{\displaystyle \sum }}_{i=1}^{n}scor{e}_i$= 1.

$$A=\{<topi{c}_1,scor{e}_1>,<topi{c}_2,scor{e}_2>,\dots <topi{c}_n,scor{e}_n>\}$$

(2)

$$scor{e}_i=P\left(w,z|\alpha ,\beta \right)=P\left(w|\alpha ,\beta \right)P\left(z|\alpha \right)$$

(3)

where w is the word in the sentence, z is the subject in the sentence, and α and β are priori parameters. According to Reference [40], we can retrieve the result:

$$P\left(z_s=k|{\overrightarrow{z}}_{\neg s},\overrightarrow{w}\right)=\left(n_{m,\neg s}^{\left(k\right)}+\alpha \right)\times \frac{{\displaystyle {\displaystyle \prod }_{w\in s}}\left(n_{k,\neg s}^{(w)}+\beta \right)\dots \left(n_{k,\neg s}^{(w)}+\beta +\left(n_{k,s}^{(w)}-1\right)\right)}{\left({{\displaystyle \sum }}_{w\in V}\left(n_{k,\neg s}^{(w)}+\beta \right)\right)\dots \left({{\displaystyle \sum }}_{w\in V}{n}_{k,\neg s}^{(w)}+\beta +\left(n_{k,s}^{(w)}-1\right)\right)}$$

(4)

where k is the number of topics to calculate the full conditional, and $\overrightarrow{w_d}=\left\{\overrightarrow{w_{d\neg s}},\overrightarrow{w_{\neg s}}\right\}$, $\overrightarrow{z}=\left\{\overrightarrow{z_{d\neg s}},\overrightarrow{z_{\neg s}}\right\}$, and $\overrightarrow{w_s}$, and $\overrightarrow{z_s}$ denote the words and topic, respectively, of sentence. w is a term that can occur many times in a sentence, $n_{k,s}^{\left(w\right)}$ denotes w’s frequency in sentence s given that the sentence s belongs to topic k, and $N_{k,s}^{\left(w\right)}$ denotes how many words of sentence s belong to topic t. $n_{m,\neg s}^{\left(w\right)}$ denotes the number of times topic k has been observed with a sentence from document d, excluding the sentence sampled.

3.2. Construction of the Heterogeneous Academic Network

The network is generally composed of vertexes and edges. Vertexes represent the objects in the network, and edges represent the relationship between objects. In order to explain the construction of heterogeneous academic networks, the notations and their interpretations are shown in

Table 1. The notations and their interpretations.

Notation	Interpretation
$G=\left(V,E\right)$	the homogeneous network
V	the vertex set, $V=\left\{v_1,v_2,\cdots v_N\right\}$, N is the total number of vertexes
E	the edge set, $E=\left\{e_1,e_2,\cdots e_M\right\}$, M is the total number of edges
$G=V,E,C$	the heterogeneous network
C	the vertex-type set, $C=\left\{c_1,c_2,\cdots c_S\right\}$, S is the total number of vertex types
$v_i{}^{c_s}$	the type of vertex $v_i$ is $c_s$
$e_{ij}=\left(v_i{}^{c_s},v_j{}^{c_s}\prime \right)$	the edge between $v_i$ and $v_j$, the type of vertex $v_i$ is $c_s$…and $v_j$ is $c_{s\prime }$

.

For the homogeneous network, $G=\left(V,E\right)$ is composed of the set of vertexes and set of edges, where V represents the vertex set, $V=\left\{v_1,v_2,\cdots v_N\right\}$, E represents the edge set, and $E=\left\{e_1,e_2,\cdots e_M\right\}$. Each edge represents the connection between two vertexes,$e_{ij}=<v_i,v_j>$, where N represents the total number of vertexes and M represents the total number of edges. In this paper, we put the focus on the heterogeneous network, $G=<V,E,C>$, where C represents the vertex-type set, $C=\left\{c_1,c_2,\cdots c_S\right\}$, S represents the total number of vertex types. If S = 1, there is only one type of vertex in the network, which belongs to the homogeneous network. If S > 1, there are two or more vertex-types, and the network is a heterogeneous network.

In the heterogeneous network, for each vertex, $v_i{}^{c_s}\in V$, and its type is $c_s\in C$. For each edge, if $e_{ij}=\left(v_i{}^{c_s},v_j{}^{c_s}\right)\in E$, which means the vertex $v_i{}^{c_s}$ is connected to the vertex $v_j{}^{c_s}$, and these two vertexes have the same vertex type, $c_s$. If $e_{i^{\prime }{j}^{\prime }}=\left(v_{i^{\prime }}{}^{c_s},v_{j^{\prime }}{}^{c_{s\prime }}\right)\in E$, which means the vertex $v_{i^{\prime }}{}^{c_s}$ is connected to the vertex $v_{j^{\prime }}{}^{c_{s\prime }}$, because the vertex types of $v_{i^{\prime }}{}^{c_s}$ and $v_{j^{\prime }}{}^{c_{s\prime }}$ are $c_s$ and $c_{s\prime }$ respectively. $v_{i^{\prime }}{}^{c_s}$ and $v_{j^{\prime }}{}^{c_{s\prime }}$ have the different vertex type. For a heterogeneous network G, we can disassemble it into several homogeneous sub-networks, $G=\left\{G^{c_1},G^{c_2},\dots ,G^{c_s}\right\}$. The number of sub-networks disassembled from the heterogeneous network is determined by the number of vertex types. Therefore, the heterogeneous network dividing into a collection of sub-networks:

$$V\left(G\right)=V\left(G^{c_1}\right)\cup V\left(G^{c_2}\right)\cup V\left(G^{c_3}\right)\cup \dots \cup V\left(G^{c_S}\right)$$

$$E\left(G\right)=E\left(G^{c_1}\right)\cup E\left(G^{c_2}\right)\cup E\left(G^{c_3}\right)\cup \dots \cup E\left(G^{c_S}\right)$$

(5)

$$C\left(G\right)=C\left(G^{c_1}\right)\cup C\left(G^{c_2}\right)\cup C\left(G^{c_3}\right)\cup \dots \cup C\left(G^{c_S}\right)$$

As a simple example, in the heterogeneous network (shown in Figure 1), we assumed there are two edges, $e_{ij}=\left(v_i{}^{c_s\prime },v_j{}^{c_s}\right)$ and $e_{ik}=\left(v_i{}^{c_{s\prime }},v_k{}^{c_s,}\right)$; three vertexes,$v_i{}^{c_{s\prime }}$, $v_j{}^{c_s}$ and $v_k{}^{c_{s\prime }}$; and two vertex types, $c_s$ and $c_{s\prime }$. We can obtain two sub-networks from this heterogeneous network, $G^{c_s}$ and $G^{c_{s\prime }}$. Where $V\left(G\right)=V\left(G^{c_s}\right)\cup V\left(G^{c_{s\prime }}\right)$, and $V\left(G^{c_s}\right)=\{v_i{}^{c_{s\prime }}$, $v_j{}^{c_s}\}$, $V\left(G^{c_s}\prime \right)=\{v_i{}^{c_{s\prime }}$, $v_k{}^{c_{s\prime }}\}$. $E\left(G\right)=E\left(G^{c_s}\right)\cup E\left(G^{c_{s\prime }}\right)$, and $E\left(G^{c_s}\right)=\left\{e_{ij}\right\}$, $E\left(G^{c_{s\prime }}\right)=\left\{e_{ik}\right\}$. $C\left(G\right)$=$C\left(G^{c_s}\right)\cup C\left(G^{c_{s\prime }}\right)$, and $C\left(G^{c_s}\right)$ = {$c_s$}, $C\left(G^{c_{s\prime }}\right)$ = {$c_{s\prime }$}.

In order to better explain this question, Figure 2 shows the process of a heterogeneous network disassembling into several homogeneous sub-networks. The heterogeneous network has four types of vertexes, so we can obtain four sub-networks according to the relationships of edges and vertexes.

In this paper, the reason for designing the heterogeneous academic network is based on users’ requirements for academic information. This means users are not only eager to obtain the ranking of papers but also want to understand the influence of related topics, the authority of authors, the impact factor of venues, as well as the highly relevant topics with search terms.

Considering the internal and external factors of scientific publication evaluation, we construct a heterogeneous academic network. The external factors include venue impact, author authority, citation relationship, and co-author. The internal factors include the title, abstract, and keyword. Because there are certain dependencies between these metadata, and there is a one-to-one relationship between title and abstract, we convert the metadata into topics for analysis in processing.

Therefore, the heterogeneous academic network in this paper has four types of vertexes (i.e., publication (title), venue, author, and topic). The types of edges include publication to publication (created by a citation relationship), author to author (created by a co-author relationship), topic to topic (created by a co-occurrence relationship), publication to venue (the paper is published in the venue), publication to author (the paper is created by an author), and publication to topic (the paper includes a topic). The list is shown in

Table 2. Types of vertex and edge.

No.	Vertex1 Type	Vertex2 Type	Edge
1	Publication	Publication	Citation relationship
2	Author	Author	Co-author relationship
3	Topic	Topic	Co-occurrence relationship
4	Publication	Venue	Paper published in venue
5	Publication	Author	Paper created by author
6	Publication	Topic	Paper includes topic

. Considering that this paper aims to propose an algorithm for publication ranking, the vertex in the network is mainly about publications. The co-author relationship and keyword co-occurrence relationship are ignored in the latter part of this paper.

We also take an example in Figure 3 to illustrate the heterogeneous academic network. Author $a_1$ writes two papers, $d_1$ and $d_2$. Paper $d_2$ is cited by paper $d_1$. The topic $t_1$ is included in $d_1$, and topic $t_2$ is included in $d_2$. $d_1$ and $d_2$ are published in venue $c_1$ and $c_2$, respectively. Paper $d_3$ is written by author $a_2$ with the topic $t_3$ and published in venue $c_2$. Author $a_3$ and author $a_4$ jointly complete paper $d_4$, which includes the topics $t_3$ and $t_4$. Paper $d_4$ is published in venue $c_3$. Paper $d_3$ is cited by papers $d_2$ and $d_4$. Paper $d_5$ is created by author $a_4$, and published in venue $c_4$. By considering the correlation of the heterogeneous network, this computing structure offsets the limitations of the homogeneous network.

When constructing the heterogeneous academic network, we introduced the Gibbs sampling model to represent the topic relevance. When different authors work together to complete a paper, they may contribute to different topics of the paper. At the same time, the topic of this paper also determines which journals the paper can be submitted to and which topics are relevant.

In this model, the generation of topics is determined by the words. Therefore, we can extract an author from all authors and determine the topic generated by that author, that is, the implicit distribution of the topic. Then, we extract a topic from the probability distributions of all words and obtain the topic of venues and keywords from the distributions of all journals and keywords.

The overall process can be divided into the following steps:

① For each topic $z$, randomly extract the probabilistic $\varphi$ for generating words for the topic of each document; the probabilistic of venue $\phi$ is generated, along with the probabilistic $\sigma$ for generating keywords for the topic. The three probability distributions are based on Dirichlet distributions with prior parameters $\beta$, $\mu$, and $\gamma$, respectively.

② For each word $w_{di}$ in document $d$:

Extract author $x_{di}$ from author $a_d$:

• A topic $z_{di}$ that follows the multinomial distribution ${\theta }_{x_{di}}$ is generated by the author $x_{di}$ ($\theta ~\mathrm{Dirichlet}\left(\alpha \right)$ follows a prior distribution);
• Word $w_{di}~\mathrm{multinomial}\left({\varphi }_{z_{di}}\right)$;
• Venue $c_{di}~\mathrm{multinomial}\left({\phi }_{z_{di}}\right)$;
• Keywords $r_{di}~\mathrm{multinomial}\left({\sigma }_{z_{di}}\right)$.

In the calculation, the Gibbs sample in the LDA method is to be used. First, we calculate the probability that the author generates a topic $\theta$, the topic generates a word $\varphi$, the topic generates a keyword $\sigma$, and the topic generates a venue $\phi$; the i-th word $w_{di}$, venue $c_{di}$, and keyword $r_{di}$ in paper d are attached to topic $z_{di}$ and author $x_{di}$. The calculated joint probability calculation formula is as follows:

$$P\left(x,z,w,c,r|\theta ,\varphi ,\phi ,\sigma ,\alpha \right)={\displaystyle \prod }_{d=1}^D{\displaystyle \prod }_{i=1}^{N_d}\frac{1}{A_d}\times {\displaystyle \prod }_{x=1}^A{\displaystyle \prod }_{z=1}^K({\theta }_{xz}{}^{m_{xz}}{\displaystyle \prod }_{v=1}^V{\varphi }_{zv}{}^{n_{zv}}{\displaystyle \prod }_{c=1}^C{\phi }_{zc}{}^{n_{zc}}{\displaystyle \prod }_{r=1}^R{\sigma }_{zr}{}^{n_{zr}})$$

(6)

where $m_{xz}$ represents the frequency of the topic $z$ that was sampled based on author $x$ distribution,$n_{zv}$ represents the frequency of the word $w_v$ that was sampled based on the topic $z$ distribution, $n_{zt}$ represents the frequency of the venue (journal or conference) $c$ that was sampled based on the topic z distribution, and $n_{zr}$ represents the frequency of the keyword r. The Dirichlet prior distribution of the $\theta$, $\varphi$, $\phi ,\mathrm{and}\sigma$ distributions are considered in calculation, and we can obtain the probability $P\left(x,z,w,c,r|\theta ,\varphi ,\phi ,\sigma ,\alpha \right)$.

After the unified modeling of the heterogeneous network, we can use the calculation results as the retrieval feedback of the author, venue, keyword, and publication. Under the premise of a certain topic, the probability that a paper d generates a word w can be expressed as follows:

$$P\left(w|d,\theta ,\varphi \right)={\displaystyle \sum }_{k=1}^{K}P\left(w|z_k,{\varphi }_{zk}\right)P\left(z_k|d,{\theta }_d\right)$$

(7)

The language model is used to represent the correlation between the document and query word, which is represented by the probability that the document generates the query word:

$$P(d|q)\propto P(q|d)P(d)$$

(8)

For a query $q$, it is usually assumed that the words contained in it all exist independently:

$$P(q|d)={\displaystyle \prod }_{w_i\in q}P(w_i|d)$$

(9)

Among them, $w_i$ represents the $i$-th word in the query word and $P\left(w_i|d\right)$ represents the probability that document $d$ generates word $w_i$.

In the same way, P(q|a), P(q|c), and P(q|r) can obtain the probability under the relevant topic. Variable a represents all papers published by the author, thereby calculating the probability of the author for the query word; variable c represents the journal or conference in which the paper was published, thereby calculating the probability of the journal or conference for the query word; and variable r represents the corresponding keyword related to the paper, thereby calculating the probability of the relevant keyword for the query word.

3.3. Ranking Algorithm for Heterogeneous Academic Networks

The NetClus algorithm is a ranking algorithm suitable for star heterogeneous networks; it divides different types of vertexes into attribute and target objects and uses the association between different types of vertexes in the clustering ranking process. In a star heterogeneous academic network, papers are defined as target objects, and authors, venues, and keywords are defined as attribute objects. During this operation, the NetClus algorithm uses the relationship between papers and authors, the relationship between papers and keywords, and the relationship between papers and venues but ignores the citation relationship between papers and papers. The citation relationship between papers is equally important for heterogeneous academic networks. Therefore, this paper hopes to use the relationship between papers and papers and the relationship between papers and the attribute types of authors, keywords, and venues. Therefore, the ConNetClus algorithm optimized by the NetClus algorithm is adopted.

A sample data relationship in this paper Is shown in Figure 4. Suppose there are four papers (D) as the target type, and there is a citation relationship between the papers; meanwhile, the papers have a link relationship with the attribute types of author (A), keyword (T), and venue (C), respectively.

The ConNetClus algorithm framework is as follows:

① The target object is initially divided into $K$ clusters, and the initial network clustering is generated according to the partition, that is ${\left\{G_k^0\right\}}_{k=1}^K$. The superscript 0 represents the 0th round, and $G_k^0$ represents the target object included in the initialization cluster $k$.

② Calculate a ranking-based probabilistic generative model for each target object under each cluster, ${\left\{P\left(x|G_k^t\right)\right\}}_{k=1}^K$.

③ Calculate the posterior probability $P\left(G_k^t|x\right)$ of each target object under each cluster and adjust the cluster assignment result according to the evaluation of the posterior probability.

④ Repeat steps ② and ③ until all clustering results no longer change significantly. That is, ${\left\{G_k^t\right\}}_{k=1}^K={\left\{G_k^{t-1}\right\}}_{k=1}^K$.

⑤ After the results are obtained for the target objects, the probability distributions of different types of attribute objects are obtained in each cluster so as to obtain the sorting results.

Generating probability model of target and attribute objects.

To simplify different types of objects in heterogeneous networks, different types of objects are decomposed, and then the generation behavior of the paper of the target object is modeled. If the probability of the authors, journals, and keywords of the paper being accessed is high, or the probability of the papers cited in the paper being accessed is high, it can be considered that the paper is highly likely to be accessed. The idea of decomposing heterogeneous networks is that for a heterogeneous network G, the access probabilities of objects of different attribute types are independent of each other. For example, author $a_i$, $p\left(a_i|G\right)$ can be decomposed into two parts: $p\left(a_i|G\right)=p\left(A|G\right)\times p\left(a_i|A,G\right)$, where $p\left(A|G\right)$ indicates the type of access author and $p\left(a_i|A,G\right)$ represents the probability of facing all authors but accessing object $a_i$. Therefore, for an attribute object $x$ and its type $T_x$, the probability of accessing $x$ in network $G$ is defined as follows:

$$p\left(x|G\right)=p\left(T_x|G\right)\times p\left(x|T_x,G\right)$$

(10)

Among them, $p\left(T_x|G\right)$ represents the probability that the type $T_x$ is accessed in the network, and $p\left(x|T_x,G\right)$ represents the probability that the object $x$ is accessed in this type. The accessed probabilities of any two attribute objects are independent. Therefore, the joint probability is defined as follows:

$$p\left(x_i,x_j|T_x,G\right)=p\left(x_i|T_x,G\right)\times p\left(x_j|T_x,G\right)$$

(11)

Among them, $x_i,x_j\in T_x$.

Then, we build a generative model of the target object. Taking the scientific research network as an example, the multiple authors who have completed the paper $d_i$, the journals that the paper is published in, and the keywords contained in the paper will all affect the generation probability of the target object paper $d_i$. Therefore, the probability of generating a paper $d_i$ under the $G_k$ cluster in the network G is defined as follows:

$$p^t\left(d_t|G_k\right)=\frac{{\displaystyle \prod _{x\in N_G(d_j)\cap x\notin P}{p}^t{(x|G_k)}^{w(d_i,x)}}{\displaystyle \prod _{x\in N_G(d_j)\cap x\notin P}{p}^{t-1}(x|G_k)}}{{\displaystyle \sum _{d_j\in p}{p}^t\left(d_j|G_k\right)}}$$

(12)

where $N_G\left(d_t\right)$ represents the set of attribute objects associated with the target object paper $d_t$; $w\left(d_t,x\right)$ represents whether the paper $d_t$ is related to the attribute object $x$; $p^{t-1}\left(x|G\right)$ represents the generation probability of a paper citing other papers, using the results of the previous round; ${\displaystyle {\displaystyle \prod }_{x\in N_G(d_j)\cap x\notin P}}{p}^t{(x|G_k)}^{w(d_i,x)}$ represents the product of the generation probabilities of all attribute objects related to the target object; and ${\displaystyle {\displaystyle \prod }_{x\in N_G(d_j)\cap x\notin P}}{p}^{t-1}\left(x|G_k\right)$ represents the product of all the generation probabilities of the target object papers in the previous round.

Because the generation probability of the attribute object may be 0, a parameter is referenced to make it smooth. The formula is as follows:

$$P_s(x|T_x,G_k)=\left(1-{\lambda }_s\right)P(x|T_x,G_k)+{\lambda }_{s}P(x|T_x,G)$$

(13)

Among them, $P(x|T_x,G_k)$ represents the probability that the attribute object $x$ is accessed in the cluster. ${\lambda }_{s}P(x|T_x,G)$ represents the access probability of attribute object $x$ in heterogeneous network G. Because the generation probability of the target object as the center point will not be 0, no parameters are added for smoothing.

The posterior probability of the target object.

It is obtained by the Bayesian formula. This probability is the probability that each target object belongs to cluster k:

$$p^{\left(t\right)}\left(k\right)=\frac{{\displaystyle {\displaystyle \sum }_{i=1}^{\left|D\right|}}{p}^{\left(t-1\right)}(k|d_i)}{\left|D\right|}$$

(14)

Among them, $p^{\left(t\right)}\left(k\right)$ represents the probability of cluster k in network G and $p^{\left(t-1\right)}(k|d_i)$ represents the last round of calculation of the probability that the paper $d_t$ belongs to cluster k.

In each round of iteration, each target object can calculate the posterior probability of each dimension; then, we can compose a K-dimensional vector that can be represented as each target object, that is, $v\left(d\right)=(p(1|d),p(2|d),\dots ,p(K|d))$. The center point of the K-dimensional vector of the cluster can be obtained according to the average value of the K-dimensional vector of the papers in this cluster. We can calculate the distance of each paper from the center point using the cosine similarity and then use this as an adjustment. We stop the iteration when the target object in the cluster no longer changes. Finally, we have the result of comprehensive sorting.

After the clustering and ranking of ConNetClus, the clustered classifiers and sorting results of each type of vertex are obtained. Then, we use the word similarity of Wikipedia [41] to calculate the similarity between the clustered classifiers and words under the topic calculated.

For a paper $d$, consider that authority $u\left[d\right]$ of the paper’s vertex is the sum of the products of the similarity ${\alpha }_n\left(d\right)$ under each cluster word and authority $u_n\left(d\right)$ under each category:

$$u\left[d\right]={\alpha }_1\left(d\right),u_1\left(d\right)+{\alpha }_2\left(d\right),u_2\left(d\right)+\cdots +{\alpha }_n\left(d\right),u_n\left(d\right)$$

(15)

Similarly, the authority of authors, conferences, and keywords is also calculated by this formula.

Then, we can apply the calculated authority of the object under the vertex type to the retrieval task of the object. This section intends to use the combined method of multiplication. We used the addition method, but the experimental results were not as good as the product. For a query word, the final score of a paper $d$ is the product of relevance $P\left(q|d\right)$ and authority $u\left[d\right]$:

$$U[d]=P(q|d)\times u[d]$$

(16)

Similarly, the final scores for authors, venues, and keywords are:

$$\begin{gathered} U\left[a\right]=P\left(q|a\right)\times u\left[a\right] \\ U\left[c\right]=P\left(q|c\right)\times u\left[c\right] \\ U\left[r\right]=P\left(q|r\right)\times u\left[r\right] \end{gathered}$$

(17)

If the user gives more than one query word, the proportion probability between the query words must be given; it is not a simple comparison between query words. Suppose $\left\{x_1,x_2,\dots ,x_n\right\}$ query keywords are given; then, we set the weight value to decrease from the first keyword to the nth keyword, with the weight $k_{median}=\frac{1}{n}$. The weights before and after are incremented and decremented, respectively. Because there are no more than 10 query keywords in general, the increasing and decreasing trends are increased or decreased by 0.01, respectively. Therefore, at this time, the final score $U_d$ of a paper is:

$$U(d)=k_1\times P(q_1|d)\times u[d]+k_2\times P(q_2|d)\times u[d]+\dots +k_n\times P(q_n|d)\times u[d]$$

(18)

Similarly, the final scores for authors, conferences, and keywords are also derived from this.

4. Experiments

4.1. Data Description

In this paper, we use the dataset of high-energy physics-based theoretical papers published on the SNAP website [42] The dataset includes a paper’s number, title, author, venue, abstract, and other metadata. There are 27,770 publications with 35,702 authors published in 118 venues (journals or conferences) with 200 keywords. The citation relationships of publication-to-publication are 352,807 edges. The publications are written by 35,702 authors; therefore, the edges of publication-to-author are 40,327. Besides that, the edges of publication-to-venue are 20,956. There are 55,540 edges of publication-to-keyword.

As shown in Figure 5, we visualized an example of sampling data with the software Gephi to illustrate the heterogeneous academic network. There are 105 publications as paper vertexes (D), 25 authors as author vertexes (A), 19 venues as venue vertexes (C), 38 keywords as topic vertexes (T), and 170 edges between the vertexes. In the figure, the larger vertexes with a darker color indicate that the higher its penetration, the more times it is connected.

In this paper, we first use Python 3.7 as the application environment when processing the dataset. The topic model SentinceLDA is implemented in Python language, which comes from a semantic-matching calculation tool in Baidu’s open source project Familia [43] in 2017.

Based on the SentinceLDA model, we can obtain the probability distributions of the abstract data for different topics, in which we define the number of topics as 100, and 2 words of each topic are considered. These selected topics are used to create the heterogeneous academic network, and then we calculate the probability distributions of each vertex in the heterogeneous academic network under different topics. After that, we select 50 topics and 5 words corresponding to each topic. Finally, the ConNetClus algorithm is used to cluster and sort the vertexes in the network. The probability value calculated by ConNetClus is used for ranking different types of vertexes.

4.2. Data Processing

For the dataset used in this paper, we must preprocess the text data, especially for the abstract and title, to effectively carry out the data analysis. The preprocessing includes word segmentation, removing stop words, utilizing WordNet, and so on.

Firstly, for the metadata in the dataset, the authors, venues, abstracts, and keywords are extracted. The data extracted by Java programming code are stored in the MySQL database. Then, we remove the stop words because there are many meaningless semantic words in the dataset. These words occupy storage space and reduce efficiency in the retrieval process. In this paper, we use the stop words in nltk corpus.

In a text paragraph, a word may appear several times in different parts of speech, such as a verb, noun, and adjective. The topic model may treat the word as different words and calculate them repeatedly. To solve this problem, WordNet is needed for text processing. WordNet is a huge English vocabulary database. A word may have many expressions, such as nouns, verbs, adjectives, and so on. This method can associate synonyms with each other through concept and semantic relationships. For example, the words ‘environment’ and ‘environmental’ have similar meanings. If they are not processed, the calculation of topic model will be redundant. Therefore, WordNet, a natural language processing tool, is used for processing.

WordNet can not only eliminate similar words in font, but also merge the same words in semantics. This paper uses the jar file of “edu.princeton.cs.algs4” from Princeton to solve the WordNet problem.

4.3. Evaluation Methods

Two indicators were used in this paper to measure algorithm performance: mean average precision (MAP) and normalized discounted cumulative gain (nDCG) [44].

MAP: Average precision is the average accuracy of each related document in a single topic. As an indicator, MAP reflects the performance of the system for all relevant documents. If no relevant documents are returned, the default accuracy is 0.

Generally, multiple query statements are used to calculate the performance, and we calculate the average value of these performances, where Q represents the total number of queries q.

$$MAP=\frac{{{\displaystyle \sum }}_{q=1}^{Q}AveP\left(q\right)}{Q}$$

(19)

NDCG: In the indicator of MAP, there are only two statuses of documents: relevant or irrelevant. In NDCG, documents can be evaluated at different levels according to their relevance. In the process of evaluation, the results with a high correlation degree are more important than those with a low correlation degree.

NDCG estimates the cumulative relevance gain a user receives by examining retrieval results up to a given rank on the list. In this research, we use the importance score, −1, 0, 1, 2, as the relevance label to calculate NDCG scores.

$$DC{G}_p={\displaystyle \sum }_{i=1}^p\frac{2^{re{l}_i}-1}{{\log }_2(i+1)}$$

(20)

where rel_i refers to the relevance level of the document i, which is divided into four values: 2 means very relevant, 1 means relevant, 0 means irrelevant, and -1 means noise file.

IDCG is the maximum value of DCG.

$$IDC{G}_p={\displaystyle \sum }_{i=1}^{\left|REL\right|}\frac{2^{re{l}_i}-1}{{\log }_2(i+1)}$$

(21)

where |REL| refers to taking the first p documents, and the sorting order is from large to small according to the relevance. Because the query results are different from each other, the length of p is not same, which also has a great impact on the calculation results of DCG.

$$nDC{G}_p=\frac{DC{G}_p}{IDC{G}_p}$$

(22)

4.4. Evaluation Results

To verify the influence of the heterogeneous academic network analysis for the academic paper ranking algorithm, this paper first compares the method proposed in this paper with the classical algorithm.

PLSA is the probabilistic latent semantic analysis model. Its main goal is to reduce the dimension and map the document from the sparse, high-dimensional word space to low-dimensional vector space. This method can create a model by a single vertex as publication, author, journal, or keyword, separately, and cannot build a heterogeneous network. Therefore, PLSA was compared with the method proposed in this paper (SenLDA + ConNetClus) to verify if there are advantages to heterogeneous network analysis.

BM25 is a classical language model method of information retrieval, which is realized by calculating the correlation between network objects and query keywords. BM25 + ConNetClus is also compared to test if the SenLDA + ConNetClus method is better than the language model for publication ranking.

PageRank is a global page ranking algorithm based on the intuition that links from important pages are more significant than links from other pages. This method has been widely used in citation analysis for evaluating the importance of vertexes in the network; however, it is independent of topics. Therefore, LDA + PageRank, as used in this paper, is tested to prove that the SenLDA + ConNetClus method is more significant in heterogeneous academic network computing and analysis.

The ground truth in this paper is from the manual annotation of physics students and gives the ranking results for papers, venues, keywords, and authors in a specific topic. Considering that users may pay more attention to the top results in information retrieval, the labeling process takes 5 results as the standard. Then, using the search intersection of the algorithm proposed in this paper, BM25 and hep-th system, the search words are selected for experimental comparison. The evaluation results in MAP and NDCG are shown in

Table 3. Experimental results of MAP.

Method	Object	MAP@10	@50	@100	Method	Object	MAP@10	@50	@100
BM25	Publication	0.176	0.114	0.108	BM25 + ConNetClus	Publication	0.246	0.124	0.118
	Venue	0.163	0.102	0.094		Venue	0.228	0.119	0.104
	Keyword	0.191	0.135	0.114		Keyword	0.269	0.1149	0.132
	Author	0.225	0.159	0.143		Author	0.319	0.204	0.183
	AVE.	0.1888	0.1275	0.1148		AVE.	0.2655	0.149	0.1343
LDA + PageRank	Publication	0.234	0.172	0.171	SenLDA + ConNetClus	Publication	0.315	0.274	0.246
						Venue	0.293	0.257	0.225
						Keyword	0.321	0.279	0.245
PLSA	Publication	0.109	0.094	0.082		Author	0.356	0.3	0.271
						AVE.	0.3213	0.2775	0.2468

and

Table 4. Experimental results of NDCG.

Method	Object	NDCG@10	@50	@100	Method	Object	NDCG@10	@50	@100
BM25	Publication	0.067	0.083	0.097	BM25 + ConNetClus	Publication	0.068	0.083	0.096
	Venue	0.041	0.062	0.071		Venue	0.052	0.071	0.087
	Keyword	0.083	0.104	0.11		Keyword	0.09	0.111	0.129
	Author	0.088	0.103	0.119		Author	0.091	0.1	0.138
	AVE.	0.0698	0.088	0.0993		AVE.	0.0753	0.0913	0.1125
LDA + PageRank	Publication	0.088	0.097	0.118	SenLDA + ConNetClus	Publication	0.098	0.138	0.153
						Venue	0.088	0.127	0.149
PLSA	Publication	0.071	0.086	0.1		Keyword	0.102	0.146	0.171
						Author	0.131	0.17	0.192
						AVE.	0.1048	0.1453	0.1663

.

From the experimental results of MAP shown in Table 3 and Figure 6, it can be found that the higher the MAP value, the more similar the ranking result is close to the ground truth. It can be clearly seen in Figure 5 that the senLDA + ConNetClus method proposed in this paper obtains better calculation results, and its MAP value is higher than other algorithms, in MAP@10, @50, and @100.

The NDCG evaluation index considers the influence of the relative importance of different contents in the search results. A higher NDCG means that the more important results will be listed at the top of search results. From the experimental results of NDCG shown in Table 4 and Figure 7, we found that the NDCG evaluation results of the method proposed in this paper are better than other comparison methods in NDCG@10, @50, and @100.

Based on the above experimental results, the method for publication ranking proposed in this paper, which considers the relevance of sentences in the process of topic training by sentenceLDA, is compared with the traditional language model analysis. In addition, this method is constructed by four types of vertexes (i.e., publication, venue, author, and topic) as the heterogeneous academic network, which was contrasted with the homogeneous network for calculating and evaluating. After that, we used a heterogeneous network analysis method named ConNetClus to calculate the probability of each vertex and the ranking result. Finally, this method achieved better results than the publication ranking algorithm. Additionally, this algorithm not only realizes the ranking questions but also supports the discovery of authors, venues, and topics with high authority and relevance.

5. Conclusions

With the development of electronic scientific publications, research on the evaluation and ranking of publications has become very important for managing academic resources. The factors affecting the evaluation of publications include internal and external factors, which are also consistent with the goal of academic evaluation. Academic evaluation not only lies in the evaluation of academic papers but also includes the analysis of venues, authors, and topics. In this paper, we realized the research objects of the topic model, network construction, and weight calculation in network analysis and explored these objects and methods with the analysis of a heterogeneous academic network. As seen in the experiments, we found that the method proposed in this paper is better than the comparative methods in MAP and NDCG.

However, there are still some limitations to this paper. For example, we only selected the four most important metadata as the vertexes in the heterogeneous academic network. However, there are more metadata affecting the evaluation results of scientific publications. In addition, the calculation of edge weights can better reflect the complexity and authenticity of social networks [45] but this paper only considered the vertex weight of heterogeneous academic networks and ignored edge weights. The reason for this comes from the limitation of the dataset. In this experiment, we only took the abstract as the content of the publication from the dataset, which lacked the support of the full-text data. In a further study, we may build a heterogeneous academic network as a complex weighted network [46] using full-text data analysis; in there, the edge weights can be calculated by a full-text analysis and topics model. During the experimental comparison, the selection of ground truth data was also limited by the dataset. More open datasets should be used for further study.