On Conditional Tsallis Entropy

Abstract

There is no generally accepted definition for conditional Tsallis entropy. The standard definition of (unconditional) Tsallis entropy depends on a parameter $\alpha$ that converges to the Shannon entropy as $\alpha$ approaches 1. In this paper, we describe three proposed definitions of conditional Tsallis entropy suggested in the literature—their properties are studied and their values, as a function of $\alpha$, are compared. We also consider another natural proposal for conditional Tsallis entropy and compare it with the existing ones. Lastly, we present an online tool to compute the four conditional Tsallis entropies, given the probability distributions and the value of the parameter $\alpha$.

1. Introduction

Tsallis entropy [1], (The name Tsallis entropy used in this paper, to identify the quantity presented in Equation (3), is not consensual in the community, given that before Tsallis presented it in 1988, and as he himself acknowledges, other authors had already introduced it [2,3,4].) a generalization of Shannon entropy [5,6], was extensively studied by Constantino Tsallis in 1988, and provides an alternative way of dealing with several characteristics of nonextensive physical systems, given that the information about the intrinsic fluctuations in the physical system can be characterized by the nonextensivity parameter $\alpha$. It can be applied to many scientific fields, such as physics [7], economics [8], computer science [9,10], and biology [11]. We refer the reader to Reference [12] for a more extensive bibliography on applications of Tsallis entropy. Furthermore, we refer the reader to Reference [13] for a survey on the most significant areas of application of the most usual entropy measures, including Shannon [6], Rényi [14], and Tsallis entropies [1,2,3,4].

It is known that, as the parameter $\alpha$ approaches 1, the Tsallis entropy corresponds to the Shannon entropy. Unlike for Shannon entropy, but similar to Rényi entropy (yet another generalization of Shannon entropy developed by Alfréd Rényi in 1961 [14], which also depends on a parameter $\alpha$ and converges to Shannon entropy when $\alpha$ approaches 1), there is no commonly accepted definition for the conditional Tsallis entropy: several versions have been proposed and used in the literature [15,16]. In this work, we revisit the notion of conditional Tsallis entropy by studying some natural and desirable properties in the existing proposals (see for instance References [15,16]): when $\alpha \to 1$, the usual conditional Shannon entropy should be recovered, the conditional Tsallis entropy should not exceed the unconditional Tsallis entropy, and the conditional Tsallis entropy should have values between 0 and the maximum value of the unconditional version.

The use of entropies in different fields, especially in the field of information theory and its connection to communication, allowed the development of several useful information measures, such as mutual information, symmetry of information, and information distances. See, for example, References [17,18,19] for some recent work related to the aforementioned information measures.

Depending on the entropy measure used, all of these have been applied in many different areas of knowledge, such as physics [20], information theory [21,22], complexity theory [23,24,25], security [26,27,28], biology [29,30,31,32], finances [33], and medicine [34,35,36], among others. The conditional Tsallis entropy, as suggested in Reference [37], can be directly applied to information theory, especially coding theory. Furthermore, since Tsallis entropy can be applied in many areas (see, for example, Reference [12]), the study of conditional Tsallis entropies is quite promising. This paper analyzes several definitions of conditional Tsallis entropy, with the intent of providing the reader with a description of the properties that each approach satisfies.

Continuing from previous works [37,38], we introduce a new natural definition for conditional Tsallis entropy as a possible alternative to the existing ones. Our new proposal does not intend to be the ultimate version of conditional Tsallis entropy, but an alternative to the existing ones, with its own properties that, in settings, such as biomedical applications, might be useful for defining information distances or other significant measurements. None of the known definitions contain all of the desired properties for a conditional version. In particular, the one presented here (as it takes the maximum over the marginal distributions) does not converge to the Shannon entropy when $\alpha \to 1$—it behaves similar to a parameterized entropy, and is akin to the one proposed in Reference [38] as an alternative to Rényi’s conditional entropy, another generalization of Shannon entropy.

The paper is organized as follows. In the next section, we present the definitions necessary for the rest of the paper, namely Shannon entropy and Tsallis entropy. In Section 3, we provide several definitions for the conditional Tsallis entropy in both existing literature and our proposal. In Section 4, we establish several results, comparing the definitions presented previously. In Section 5, we explore some features of each variant for the conditional Tsallis entropy. Finally, in Section 6, we present the conclusions and future work.

2. Preliminaries

In the remainder of the paper, we use the standard notation for entropies and for probability distributions according to Reference [5]. For the sake of simplicity of notation, we use the notation log for the logarithm in base 2. We call the reader’s attention to the fact that, whenever we say that one entropy converges to another, it is always up to logarithmic factor that depends only on the choice of cardinality of the alphabet.

The Shannon entropy of X is the expectation of the surprise of an occurrence,

$$H\left(X\right)=-\sum _x\left(\mathrm{P}(X=x)log\mathrm{P}(X=x)\right).$$

(1)

The conditional Shannon entropy, $H\left(Y\right|X)$, is the expectation over x of the entropy of the distribution $\mathrm{P}\left(Y\right|X=x)$,

$$\begin{array}{ccc}\hfill H\left(Y\right|X)& =& {\mathrm{E}}_{x,y}\left[log\left(\frac{1}{\mathrm{P}(Y=y|X=x)}\right)\right].\hfill \end{array}$$

(2)

It is easy to derive the chain rule $H(X,Y)=H\left(X\right)+H\left(Y\right|X)$: to get the average information contained in $(X,Y)$, we may first get the average information contained in X, and add to it the average information of Y, given X.

The Tsallis entropy [1] was firstly introduced in [2,3] and is defined for a random variable X by:

$$T_{\alpha }\left(X\right)=\frac{1}{\alpha -1}\left(1-\sum _x\mathrm{P}{(X=x)}^{\alpha }\right),(\mathrm{for}\alpha >0,\alpha \ne 1).$$

(3)

It is straightforward to show that, when the parameter $\alpha$ converges to 1, the value of the entropy converges to the Shannon entropy.

3. Conditional Tsallis Entropy: Four Definitions

We consider three definitions for conditional Tsallis entropy that already exist in the literature and introduce a new proposal. All definitions consider a positive parameter $\alpha$.

Definition 1.

Let $Z=(X,Y)$ be a random vector. One can define the following variants of conditional Tsallis entropy:

1. 

Definition of $T_{\alpha }\left(Y\right|X)$ from Reference [15]

$$\begin{array}{ccc}\hfill T_{\alpha }\left(Y\right|X)& =& \sum _x\mathrm{P}{(X=x)}^{\alpha }{T}_{\alpha }\left(Y\right|x)\hfill \end{array}$$

(4)

$$\begin{array}{ccc}\hfill & =& \frac{1}{\alpha -1}\sum _x\mathrm{P}{(X=x)}^{\alpha }\left(1-\sum _y\mathrm{P}{(Y=y|X=x)}^{\alpha }\right).\hfill \end{array}$$

(5)

One can easily verify that $T_{\alpha }(X,Y)=T_{\alpha }\left(Y\right|X)+T_{\alpha }\left(X\right)$ and, therefore, it satisfies the chain rule.

2. 

Definition of $S_{\alpha }\left(Y\right|X)$ from [16] (Definition 2.8)

$$\begin{array}{ccc}\hfill S_{\alpha }\left(Y\right|X)& =& \sum _x\mathrm{P}(X=x)T_{\alpha }\left(Y\right|X=x)\hfill \end{array}$$

(6)

$$\begin{array}{ccc}& =& \sum _x\mathrm{P}(X=x)\frac{1}{\alpha -1}\left(1-\sum _y\mathrm{P}{(Y=y|X=x)}^{\alpha }\right)\hfill \end{array}$$

(7)

$$\begin{array}{ccc}& =& \frac{1}{\alpha -1}\sum _x\mathrm{P}(X=x)\left(1-\sum _y\mathrm{P}{(Y=y|X=x)}^{\alpha }\right).\hfill \end{array}$$

(8)

3. 

Definition of $S_{\alpha }^{\prime }\left(Y\right|X)$ from [16] (Definition 2.10)

$$S_{\alpha }^{\prime }\left(Y\right|X)=\frac{1}{\alpha -1}\left({\displaystyle \frac{{\displaystyle 1-\sum _{x,y}\mathrm{P}{(X=x,Y=y)}^{\alpha }}}{{\displaystyle \sum _x\mathrm{P}{(X=x)}^{\alpha }}}}\right).$$

(9)

The first definition presented proposes that the conditional Tsallis entropy should be weighed by the probability of sampling $X=x$ with parameter $\alpha$, while the second one proposes that one uniformly weighs only the probability of sampling $X=x$. Therefore, notice that for the first definition presented, the value of $\alpha$ largely affects the value of the conditional Tsallis entropy. The idea for the third proposal is to distribute evenly the influence of the parameter $\alpha$ by the entire joint distribution.

Next, we present another possible definition of the conditional Tsallis entropy. This definition is based on Definition III.6 of [38] and captures the intuitive notion of defining the conditional entropy, by taking the maximum over all possible marginal distributions. Note that this definition is analogous to an existing one for the Rényi entropy; however, as we will show later, this proposal does not satisfy some of the expected basic properties.

Definition 2

(Definition of $T_{\alpha }^{\prime }\left(Y\right|X)$).

$$T_{\alpha }^{\prime }\left(Y\right|X)=\frac{1}{\alpha -1}\underset{x}{max}\left(1-\sum _y\mathrm{P}{(Y=y|X=x)}^{\alpha }\right).$$

(10)

We opted to use different notations for the variants of the conditional Tsallis entropy in the last definition, to better distinguish them in the rest of the paper. In particular, we follow the same approach as in Reference [38].

The following expressions will be useful later.

Theorem 1.

Let $Z=(X,Y)$ be a random vector. The following identities are true:

$$\begin{array}{ccc}\hfill T_{\alpha }^{\prime }\left(Y\right|X)& =& \underset{x}{max}{T}_{\alpha }\left(Y\right|X=x)(for\alpha >1)\hfill \end{array}$$

(11)

$$\begin{array}{ccc}\hfill T_{\alpha }^{\prime }\left(Y\right|X)& =& \underset{x}{min}{T}_{\alpha }\left(Y\right|X=x)(for\alpha <1).\hfill \end{array}$$

(12)

4. Comparison of the Definitions

We now compare the above four definitions of the conditional Tsallis entropy by comparing whether or not the definition satisfies some common properties of an entropy measure. In the next theorem, we report two simple facts with straightforward proofs. We leave the details for the interested reader to check.

Theorem 2.

For any fixed joint probability distribution $\mathrm{P}(X,Y)$,

(i) 

$T_{\alpha }\left(Y\right|X)$, $S_{\alpha }\left(Y\right|X)$ and $S_{\alpha }^{\prime }\left(Y\right|X)$, as functions of α, are continuous and differentiable;

(ii) 

$T_{\alpha }^{\prime }\left(Y\right|X)$, as a function of α, is continuous for all $\alpha \ne 1$.

The following results provide the possible comparisons (in terms of values) between the proposed definitions. For the sake of organization, we split the comparison by types of entropy.

First we compare $T_{\alpha }\left(Y\right|X)$ with $S_{\alpha }\left(Y\right|X)$.

Theorem 3.

For all joint probability distributions $\mathrm{P}(X,Y)$ and for every $\alpha >0$,

$$\begin{array}{ccc}\hfill & & if\alpha <1:S_{\alpha }\left(Y\right|X)\le T_{\alpha }\left(Y\right|X)\hfill \end{array}$$

(13)

$$\begin{array}{ccc}\hfill & & if\alpha =1:S_{\alpha }\left(Y\right|X)=T_{\alpha }\left(Y\right|X)=H\left(Y\right|X)\hfill \end{array}$$

(14)

$$\begin{array}{ccc}\hfill & & if\alpha >1:S_{\alpha }\left(Y\right|X)\ge T_{\alpha }\left(Y\right|X).\hfill \end{array}$$

(15)

Proof. 

Consider first the case $\alpha <1$. In this case, we have that $P{(X=x)}^{\alpha }\ge P(X=x)$. Thus,

$$\begin{array}{ccc}\hfill P{(X=x)}^{\alpha }\times T_{\alpha }\left(Y\right|X=x)& \ge & P(X=x)\times T_{\alpha }\left(Y\right|X=x)\hfill \\ \hfill {\displaystyle \iff \sum _{x}P{(X=x)}^{\alpha }\times T_{\alpha }\left(Y\right|X=x)}& \ge & {\displaystyle \sum _{x}P(X=x)\times T_{\alpha }\left(Y\right|X=x)}\hfill \\ \hfill \iff T_{\alpha }\left(Y\right|X)& \ge & S_{\alpha }\left(Y\right|X).\hfill \end{array}$$

For the case $\alpha =1$, see the proof of Theorem 8.

The case $\alpha >1$ is similar to the previous one, but this time, the conclusion follows, since for $\alpha >1$, $P{(X=x)}^{\alpha }\le P(X=x)$. □

In the next theorem we provide the comparison between $T_{\alpha }^{\prime }\left(Y\right|X)$ and $S_{\alpha }\left(Y\right|X)$.

Theorem 4.

For all joint probability distributions $\mathrm{P}(X,Y)$ and for every $\alpha >0$,

$$\begin{array}{ccc}\hfill & & if\alpha \le 1:T_{\alpha }^{\prime }\left(Y\right|X)\le S_{\alpha }\left(Y\right|X)\hfill \end{array}$$

(16)

$$\begin{array}{ccc}\hfill & & if\alpha >1:T_{\alpha }^{\prime }\left(Y\right|X)\ge S_{\alpha }\left(Y\right|X).\hfill \end{array}$$

(17)

Proof. 

Consider first the case $\alpha <1$. In this case, we have that ${\displaystyle T_{\alpha }^{\prime }\left(Y\right|X)=\underset{x}{min}{T}_{\alpha }\left(Y\right|X=x)}$. So,

$$\begin{array}{ccc}\hfill S_{\alpha }\left(Y\right|X)& =& {\displaystyle \sum _x\left(\mathrm{P}(X=x)·T_{\alpha }\left(Y\right|X=x)\right)}\hfill \end{array}$$

(18)

$$\begin{array}{ccc}& \ge & {\displaystyle \sum _x(\mathrm{P}(X=x)·\left[\underset{x}{min}{T}_{\alpha }\left(Y\right|X=x)\right])}\hfill \end{array}$$

(19)

$$\begin{array}{ccc}& =& {\displaystyle \left(\underset{x}{min}{T}_{\alpha }\left(Y\right|X=x)\right)·\sum _x\mathrm{P}(X=x)}\hfill \end{array}$$

(20)

$$\begin{array}{ccc}& =& {\displaystyle \underset{x}{min}{T}_{\alpha }\left(Y\right|X=x)}\hfill \end{array}$$

(21)

$$\begin{array}{ccc}& =& T_{\alpha }^{\prime }\left(Y\right|X).\hfill \end{array}$$

(22)

The proof of the case $\alpha >1$ is similar to the previous one but this time, the conclusion follows from the fact that, for $\alpha >1$, $T_{\alpha }^{\prime }\left(Y\right|X)={max}_x{T}_{\alpha }\left(Y\right|X=x)$. □

As a consequence of the two previous results and the definitions, we can derive the relation between $T_{\alpha }^{\prime }$ and $T_{\alpha }$.

Corollary 1.

For all joint probability distributions $\mathrm{P}(X,Y)$ and for every $\alpha >0$,

$$\begin{array}{ccc}\hfill & & if\alpha \le 1:T_{\alpha }^{\prime }\left(Y\right|X)\le T_{\alpha }\left(Y\right|X)\hfill \end{array}$$

(23)

$$\begin{array}{ccc}\hfill & & if\alpha >1:T_{\alpha }^{\prime }\left(Y\right|X)\ge T_{\alpha }\left(Y\right|X).\hfill \end{array}$$

(24)

The proof follows directly from Theorems 3 and 4. Now, we derive the relation between $S_{\alpha }^{\prime }\left(Y\right|X)$ and $T_{\alpha }^{\prime }\left(Y\right|X)$.

Theorem 5.

For all joint probability distributions $\mathrm{P}(X,Y)$ and for every $\alpha >0$,

$$\begin{array}{ccc}\hfill & & if\alpha \le 1:S_{\alpha }^{\prime }\left(Y\right|X)\ge T_{\alpha }^{\prime }\left(Y\right|X)\hfill \end{array}$$

(25)

$$\begin{array}{ccc}\hfill & & if\alpha >1:S_{\alpha }^{\prime }\left(Y\right|X)\le T_{\alpha }^{\prime }\left(Y\right|X).\hfill \end{array}$$

(26)

Proof. 

Consider first the case $\alpha <1$. Proving that $S_{\alpha }^{\prime }\left(Y\right|X)\ge T_{\alpha }^{\prime }\left(Y\right|X)$, by definition, is the same to prove:

$$\begin{array}{c}\hfill \frac{\left(1-\frac{{\displaystyle \sum _{x,y}\mathrm{P}{(X=x,Y=y)}^{\alpha }}}{{\displaystyle \sum _x\mathrm{P}{\left(x\right)}^{\alpha }}}\right)}{\alpha -1}\ge \frac{{\displaystyle \underset{x}{max}\left(1-\sum _y\mathrm{P}{\left(y\right|x)}^{\alpha }\right)}}{\alpha -1}.\end{array}$$

(27)

As $\alpha <1$, we have that ${\displaystyle \frac{1}{\alpha -1}<0}$. Thus, proving Equation (27) is the same, proves that:

$$\begin{array}{c}{\displaystyle 1-\frac{{\displaystyle \sum _{x,y}\mathrm{P}{(X=x,Y=y)}^{\alpha }}}{{\displaystyle \sum _x\mathrm{P}{(X=x)}^{\alpha }}}\le \underset{x}{max}\left(1-\sum _y\mathrm{P}{(Y=y|X=x)}^{\alpha }\right)}\\ {\displaystyle \iff \frac{{\displaystyle \sum _{x,y}\mathrm{P}{(X=x,Y=y)}^{\alpha }}}{{\displaystyle \sum _x\mathrm{P}{(X=x)}^{\alpha }}}\ge \underset{x}{min}\sum _y\mathrm{P}{(Y=y|X=x)}^{\alpha }}\\ {\displaystyle \iff \sum _{x,y}\mathrm{P}{(X=x,Y=y)}^{\alpha }\ge \left(\sum _x\mathrm{P}{(X=x)}^{\alpha }\right)\times \underset{x}{min}\left(\sum _y\mathrm{P}{(Y=y|X=x)}^{\alpha }\right).}\end{array}$$

Now, the result follows by observing that the last inequality is true, since, for $\alpha <1$ and for every x, we have that

$$\underset{x}{min}\left(\sum _y\mathrm{P}{(Y=y|X=x)}^{\alpha }\right)\le \sum _y\mathrm{P}{(Y=y,X=x)}^{\alpha }.$$

The case $\alpha >1$ is proved in a similar manner. □

Now, we derive the relation between $T_{\alpha }\left(Y\right|X)$ and $S_{\alpha }^{\prime }\left(Y\right|X)$.

Theorem 6.

For all joint probability distributions $\mathrm{P}(X,Y)$ and for every $\alpha >0$,

$$\begin{array}{ccc}\hfill & & if\alpha \le 1:T_{\alpha }\left(Y\right|X)\ge S_{\alpha }^{\prime }\left(Y\right|X)\hfill \end{array}$$

(28)

$$\begin{array}{ccc}\hfill & & if\alpha >1:T_{\alpha }\left(Y\right|X)\le S_{\alpha }^{\prime }\left(Y\right|X).\hfill \end{array}$$

(29)

Proof. 

Consider first the case $\alpha <1$. Thus,

$$\begin{array}{c}{T}_{\alpha }\left(Y\right|X)\ge S_{\alpha }^{\prime }\left(Y\right|X)\\ {\displaystyle \iff \frac{1}{\alpha -1}\sum _x\left(\mathrm{P}{(X=x)}^{\alpha }\left(1-\sum _y\mathrm{P}{(Y=y|X=x)}^{\alpha }\right)\right)\ge \frac{1}{\alpha -1}\left(1-\frac{{\displaystyle \sum _{x,y}\mathrm{P}{(X=x,Y=y)}^{\alpha }}}{{\displaystyle \sum _x\mathrm{P}{(X=x)}^{\alpha }}}\right)}\end{array}$$

$$\begin{array}{c}{\displaystyle \iff \sum _x\left(\mathrm{P}{(X=x)}^{\alpha }\left(1-\sum _{y}P{(Y=y|X=x)}^{\alpha }\right)\right)\le 1-\frac{{\displaystyle \sum _{x,y}\mathrm{P}{(X=x,Y=y)}^{\alpha }}}{{\displaystyle \sum _x\mathrm{P}{(X=x)}^{\alpha }}}}\\ {\displaystyle \iff \sum _x\mathrm{P}{(X=x)}^{\alpha }-\sum _x\left(\mathrm{P}{(X=x)}^{\alpha }\sum _y\frac{\mathrm{P}{(X=x,Y=y)}^{\alpha }}{\mathrm{P}{(X=x)}^{\alpha }}\right)\le 1-\frac{{\displaystyle \sum _{x,y}\mathrm{P}{(X=x,Y=y)}^{\alpha }}}{{\displaystyle \sum _x\mathrm{P}{(X=x)}^{\alpha }}}}\\ {\displaystyle \iff \sum _x\mathrm{P}{(X=x)}^{\alpha }-\sum _{x,y}\mathrm{P}{(X=x,Y=y)}^{\alpha }\le \frac{{\displaystyle \sum _x\mathrm{P}{(X=x)}^{\alpha }-\sum _{x,y}\mathrm{P}{(X=x,Y=y)}^{\alpha }}}{{\displaystyle \sum _{x}P{(X=x)}^{\alpha }}}.}\end{array}$$

The result follows by observing that the last inequality is true, since for $\alpha <1$, we have that:

$$\begin{array}{c}\hfill \mathrm{P}{(X=x)}^{\alpha }>\mathrm{P}(X=x)\end{array}$$

(30)

and consequently,

$$\begin{array}{c}\hfill {\displaystyle \sum _x\mathrm{P}{(X=x)}^{\alpha }>1.}\end{array}$$

(31)

The proof of the case $\alpha >1$ is similar to the previous one. □

Finally, we show that the values of $S_{\alpha }$ and $S_{\alpha }^{\prime }$ are incomparable in the sense that there are probability distributions for which $S_{\alpha }$ is greater than $S_{\alpha }^{\prime }$ and there are probability distributions for which $S_{\alpha }^{\prime }$ is greater than $S_{\alpha }$.

Theorem 7.

The values of $S_{\alpha }\left(Y\right|X)$ and of $S_{\alpha }^{\prime }\left(Y\right|X)$ are incomparable, i.e., for each $n\ge 2$ and $\alpha \ne 1$

$$\begin{array}{c}\hfill \exists \mathrm{P}(X,Y):S_{\alpha }\left(Y\right|X)<S_{\alpha }^{\prime }\left(Y\right|X)\end{array}$$

(32)

$$\begin{array}{c}\hfill \exists \mathrm{P}(X,Y):S_{\alpha }\left(Y\right|X)>S_{\alpha }^{\prime }\left(Y\right|X).\end{array}$$

(33)

Proof. 

For Statement (32) and $\alpha <1$, consider the following joint probability distribution:

$$\begin{array}{ccc}X\backslash Y& 1& 2\\ 1& 0.0625& 0.0625\\ 2& 0.0125& 0.8625\end{array}$$

$$\begin{array}{ccc}\hfill S_{0.25}\left(Y\right|X)& \approx & 0.513\hfill \end{array}$$

(34)

$$\begin{array}{ccc}\hfill S_{0.25}^{\prime }\left(Y\right|X)& \approx & 0.629\hfill \end{array}$$

(35)

For Statement (32) and $\alpha >1$, consider the following joint probability distribution:

$$\begin{array}{ccc}X\backslash Y& 1& 2\\ 1& 0.1125& 0.0125\\ 2& 0.4375& 0.4375\end{array}$$

$$\begin{array}{ccc}\hfill S_{2.5}\left(Y\right|X)& \approx & 0.396\hfill \end{array}$$

(36)

$$\begin{array}{ccc}\hfill S_{2.5}^{\prime }\left(Y\right|X)& \approx & 0.429\hfill \end{array}$$

(37)

For Statement (33) and $\alpha <1$, consider the following joint probability distribution:

$$\begin{array}{ccc}X\backslash Y& 1& 2\\ 1& 0.125& 0\\ 2& 0.5& 0.375\end{array}$$

$$\begin{array}{ccc}\hfill S_{0.25}\left(Y\right|X)& \approx & 0.792\hfill \end{array}$$

(38)

$$\begin{array}{ccc}\hfill S_{0.25}^{\prime }\left(Y\right|X)& \approx & 0.560\hfill \end{array}$$

(39)

Finally, for Statement (33) and $\alpha >1$, consider the following joint probability distribution:

$$\begin{array}{ccc}X\backslash Y& 1& 2\\ 1& 0.0625& 0.0625\\ 2& 0.0125& 0.8625\end{array}$$

$$\begin{array}{ccc}\hfill S_{1.25}\left(Y\right|X)& \approx & 0.125\hfill \end{array}$$

(40)

$$\begin{array}{ccc}\hfill S_{1.25}^{\prime }\left(Y\right|X)& \approx & 0.099\hfill \end{array}$$

(41)

□

5. Properties of the Conditional Tsallis Entropies

In this section, we investigate some properties of the proposals considered. In particular, we show that there are probability distributions and $\alpha \ne 1$ for which the conditional Tsallis entropies are bigger than the unconditional Tsallis entropy.

Theorem 8.

For any fixed joint probability distribution $\mathrm{P}(X,Y)$,

$$\begin{array}{ccc}\hfill \underset{\alpha \to 1}{lim}{T}_{\alpha }\left(Y\right|X)& =& H\left(Y\right|X)\hfill \end{array}$$

(42)

$$\begin{array}{ccc}\hfill \underset{\alpha \to 1}{lim}{S}_{\alpha }\left(Y\right|X)& =& H\left(Y\right|X)\hfill \end{array}$$

(43)

$$\begin{array}{ccc}\hfill \underset{\alpha \to 1}{lim}{S}_{\alpha }^{\prime }\left(Y\right|X)& =& H\left(Y\right|X)\hfill \end{array}$$

(44)

where $H\left(Y\right|X)$ is the conditional Shannon entropy. In general, it is not true that ${\displaystyle \underset{\alpha \to 1}{lim}{T}_{\alpha }^{\prime }\left(Y\right|X)=H\left(Y\right|X)}$.

Proof. 

The second equation is easy to derive directly from the definition of conditional probability and from Equation (2). Furthermore, using Equation (6) we can also easily obtain (using the previous derivation) that Equation (42) is also true.

The third equation was proven in Reference [16].

Now, it is only left to prove the last statement of the theorem, i.e., in general

$$\begin{array}{c}\hfill {\displaystyle \underset{\alpha \to 1}{lim}{T}_{\alpha }^{\prime }\left(Y\right|X)\ne H\left(Y\right|X).}\end{array}$$

(45)

From Equations (6) and (11) it is easy to check that $T_{\alpha }\left(Y\right|X)$ is the expectation over x of $T_{\alpha }\left(Y\right|x)$, while $T_{\alpha }^{\prime }\left(Y\right|X)$ is the maximum over x of $T_{\alpha }\left(Y\right|x)$.

The function $T_{\alpha }\left(Y\right|x)$ depends on the conditional probabilities $\mathrm{P}(Y=y|X=x)$. Therefore, there are joint probability distributions $\mathrm{P}(X=x,Y=y)$, such that:

$$\begin{array}{c}\hfill \underset{\alpha \to 1}{lim}{T}_{\alpha }^{\prime }\left(Y\right|X)\ne \underset{\alpha \to 1}{lim}{T}_{\alpha }\left(Y\right|X)=H\left(Y\right|X).\end{array}$$

(46)

□

Contrary to the Shannon entropy, the value of any conditional Tsallis entropy may exceed the corresponding unconditional Tsallis entropy for all proposals.

Theorem 9.

There are probability distributions $\mathrm{P}(X,Y)$ and values of α, such that:

$$\begin{array}{c}\hfill T_{\alpha }\left(Y\right|X)>T_{\alpha }\left(Y\right)\end{array}$$

(47)

$$\begin{array}{c}\hfill S_{\alpha }\left(Y\right|X)>T_{\alpha }\left(Y\right)\end{array}$$

(48)

$$\begin{array}{c}\hfill S_{\alpha }^{\prime }\left(Y\right|X)>T_{\alpha }\left(Y\right)\end{array}$$

(49)

$$\begin{array}{c}\hfill T_{\alpha }^{\prime }\left(Y\right|X)>T_{\alpha }\left(Y\right).\end{array}$$

(50)

Proof. 

Consider the following joint probability distribution:

$$\begin{array}{ccc}X=x\backslash Y=y& 1& 2\\ 1& 0.45& 0.45\\ 2& 0.1& 0.0\end{array}$$

For this distribution we have:

$$\begin{array}{ccc}\hfill T_{0.5}\left(Y\right)& \approx & 0.824\hfill \end{array}$$

(51)

$$\begin{array}{ccc}\hfill T_{0.5}\left(Y\right|X)& \approx & 1.047\hfill \end{array}$$

(52)

$$\begin{array}{ccc}\hfill S_{0.5}\left(Y\right|X)& \approx & 0.828\hfill \end{array}$$

(53)

$$\begin{array}{ccc}\hfill T_3\left(Y\right)& \approx & 0.371\hfill \end{array}$$

(54)

$$\begin{array}{ccc}\hfill S_3^{\prime }\left(Y\right|X)& \approx & 0.374\hfill \end{array}$$

(55)

$$\begin{array}{ccc}\hfill T_3^{\prime }\left(Y\right|X)& \approx & 0.375\hfill \end{array}$$

(56)

□

Bounds on Conditional Tsallis Entropy

As mentioned in the Introduction, one of the properties of the (conditional) Shannon entropy for discrete variables is to be bounded by the number of elements of the support of the distribution. Furthermore, it is well known that the unconditional Tsallis entropy is always between 0 and ${\displaystyle \frac{m^{1-\alpha }}{1-\alpha }}$, where m is the number of elements in the support of the distribution. In this subsection, we derive bounds for the conditional Tsallis entropies based on the number of elements in the support of each distribution.

Theorem 10.

Let $Z=(X,Y)$ be any joint random vector defined over sets of size m each. Then,

$$\begin{array}{c}\hfill 0\le S_{\alpha }\left(Y\right|X)\le \frac{m^{1-\alpha }}{1-\alpha }\end{array}$$

(57)

$$\begin{array}{c}\hfill 0\le T_{\alpha }^{\prime }\left(Y\right|X)\le \frac{m^{1-\alpha }}{1-\alpha }.\end{array}$$

(58)

Moreover all of these lower and upper bounds may be reached by suitable probability distributions $\mathrm{P}(X,Y)$.

Proof. 

The Inequalities (57) follow from the fact that $S_{\alpha }\left(Y\right|X)$ is the expectation of the unconditional Tsallis entropy.

For Inequalities (58), recall that Equation (10) can be written, for $\alpha <1$, as Equation (12). Note that, for all x, the values $T_{\alpha }\left(Y\right|X=x)$ are the (unconditional) Tsallis entropies of the marginal distribution, and are all defined in a set of cardinality m.

So, by definition of $T_{\alpha }^{\prime }$, for some particular x, we have $T_{\alpha }^{\prime }\left(Y\right|X)=T_{\alpha }\left(Y\right|X=x)$. The case $\alpha >1$ is similar. So, independently of $\alpha$, for every probability distributions $\mathrm{P}\left(X\right)$ and $\mathrm{P}\left(Y\right)$ defined over set with m elements, we have ${\displaystyle 0\le T_{\alpha }^{\prime }\left(Y\right|X)\le \frac{m^{1-\alpha }}{1-\alpha }}$, since the same bound applies for the unconditional version or any its marginal distributions. □

Theorem 11.

Let $Z=(X,Y)$ be any joint random vector defined over sets of size m each. Then,

$$\begin{array}{ccc}\hfill & & if\alpha >1:0\le T_{\alpha }\left(Y\right|X)\le \frac{m^{1-\alpha }}{1-\alpha }.\hfill \end{array}$$

(59)

For $\alpha <1$, in general, the inequality does not hold.

Proof. 

Consider first the case $\alpha >1$. The result follows directly from Inequalities (15) and (57).

In order to prove that the inequality does not hold for all $\alpha <1$, consider $\alpha =0.1$ and the following joint probability distribution:

$$\begin{array}{cccc}X=x\backslash Y=y& 1& 2& 3\\ 1& 1/9& 1/9& 0\\ 2& 1/9& 1/9& 1/9\\ 3& 0& 1/9& 1/3\end{array}$$

Notice that ${\displaystyle \frac{m^{1-\alpha }}{1-\alpha }}\approx 2.987$ and $T_{0.1}\left(Y\right|X)\approx 3.371$. For any other $\alpha <1$, one can construct similarly a joint probability distribution for which the inequality is also violated. □

Theorem 12.

Let $Z=(X,Y)$ be any joint random vector defined over sets of size m each. Then,

$$\begin{array}{ccc}\hfill & & if\alpha >1:0\le S_{\alpha }^{\prime }\left(Y\right|X)\le \frac{m^{1-\alpha }}{1-\alpha }.\hfill \end{array}$$

(60)

Proof. 

The result follows directly from the Inequalities (26) and (58). □

We conjecture that the above theorem also holds for $\alpha <1$. For example, the inequality is true for all uniform probability distribution over n variables.

We now show that, for any fixed joint probability distribution $\mathrm{P}(X,Y)$, three of the forms of conditional Tsallis entropy studied in this paper are non-increasing functions of $\alpha$. First, we state a simple theorem.

Lemma 1.

If $f_1\left(x\right)$,…, $f_m\left(x\right)$ are non-increasing real functions, then the function ${max}_i\left(f_i\left(x\right)\right)$ is also a non-increasing function.

Theorem 13.

For every probability distribution $\mathrm{P}(X,Y)$,

1. 

$T_{\alpha }\left(Y\right|X)$ is a non-increasing function of α.

2. 

$S_{\alpha }\left(Y\right|X)$ is a non-increasing function of α.

3. 

$T_{\alpha }^{\prime }\left(Y\right|X)$ is a non-increasing function of α.

Proof. 

1.

First consider the case $\alpha >1$, and consider the function $d{T}_{\alpha }\left(Y\right|X)$, the derivative of the function $T_{\alpha }\left(Y\right|X)$ in order to $\alpha$:

$$\frac{d{T}_{\alpha }\left(Y\right|X)}{d\alpha }=\frac{{\displaystyle -1+\sum _x\mathrm{P}{(X=x)}^{\alpha }}}{{(\alpha -1)}^2}-\frac{{\displaystyle \sum _x\left(\alpha \mathrm{P}{(Y=y|X=x)}^{\alpha -1}log\alpha \right)}}{\alpha -1}.$$

It is easy to see that, since $\alpha >1$, $\frac{d{T}_{\alpha }\left(X\right)}{d\alpha }<0$. Therefore, the function $T_{\alpha }\left(Y\right|X)$ is a non-increasing function of $\alpha$.

Consider now the case $\alpha <1$ and assume that $\alpha ,{\alpha }^{\prime }$ are such that $\alpha <{\alpha }^{\prime }<1$. In order to prove that $T_{\alpha }\left(Y\right|X)$ is non-increasing we have to show that $T_{\alpha }\left(X\right)\ge T_{\alpha }^{\prime }\left(X\right)$, i.e.,:

$${\displaystyle \begin{array}{cccc}& \hfill {\displaystyle 1-\frac{{\displaystyle 1-\sum _x\mathrm{P}{(X=x)}^{\alpha }}}{\alpha -1}}& \ge & {\displaystyle 1-\frac{{\displaystyle 1-\sum _x\mathrm{P}{(X=x)}^{{\alpha }^{\prime }}}}{{\alpha }^{\prime }-1}}\hfill \\ \iff & \hfill \frac{{\displaystyle -1+\sum _x\mathrm{P}{(X=x)}^{\alpha }}}{1-\alpha }& \ge & \frac{{\displaystyle -1+\sum _x\mathrm{P}{(X=x)}^{{\alpha }^{\prime }}}}{1-{\alpha }^{\prime }}\hfill \\ \iff & \hfill \frac{{\displaystyle 1-\sum _x\mathrm{P}{(X=x)}^{\alpha }}}{1-\alpha }& \le & \frac{{\displaystyle 1-\sum _x\mathrm{P}{(X=x)}^{{\alpha }^{\prime }}}}{1-{\alpha }^{\prime }}\hfill \end{array}}$$

Notice that, since $\alpha <{\alpha }^{\prime }<1$, Then $\frac{1}{1-\alpha }<\frac{1}{1-{\alpha }^{\prime }}$ and, therefore, ${\displaystyle 1-\sum _x\mathrm{P}{(X=x)}^{\alpha }}\le {\displaystyle 1-\sum _x\mathrm{P}{(X=x)}^{{\alpha }^{\prime }}}$. So, the last inequality is true.

2.

This part of the result follows from the fact that $S_{\alpha }\left(Y\right|X)$ is the expectation of unconditional Tsallis entropies; see Equation (6).

3.

Suppose that $\alpha >1$. The proof is a direct consequence of Equation (11) and Lemma 1. The case $\alpha <1$ can be proven in a similar way.

□

It is easy to show that $S^{\prime }$ does not fulfill the property of the last theorem.

Theorem 14.

There exists probability distributions $(X,Y)$ and $\alpha <{\alpha }^{\prime }$ for which $S_{\alpha }^{\prime }\left(Y\right|X)\le S_{{\alpha }^{\prime }}^{\prime }\left(Y\right|X)$.

Proof. 

Consider the following joint probability distribution:

$$\begin{array}{ccc}X=x\backslash Y=y& 1& 2\\ 1& 0.45& 0.45\\ 2& 0.1& 0.0\end{array}$$

We have:

$$S_{0.2}^{\prime }\left(Y\right|X)\approx 0.563$$

$$S_{0.5}^{\prime }\left(Y\right|X)\approx 0.621.$$

□

We developed a small application that, given two probability distributions, computes the values of all conditional Tsallis entropies considered in the paper. The application is self-contained and its use is extremely simple. There are two use case examples that the reader can use in order to try the calculator. The interested reader can find it in the following link: http://gloss.di.fc.ul.pt/tryit/Tsallis (accessed on 28 October 2021).

6. Conclusions

In this paper, we studied the definitions for the conditional Tsallis entropy existing in the literature. We also considered a possible alternative definition for it. This new proposal is a natural approach to consider as a possible definition. It defines the conditional value as the maximum value of all marginal distributions. Due to this fact, and similar to what happens with the Rényi entropy, this definition was also analyzed, although it was never considered in the literature before. The relationships between the four definitions, described in this work, are summarized in Figure 1.

As we understand, it would be expectable that a proposal for conditional Tsallis entropy would satisfy the following properties:

• Chain Rule;
• Convergence to Shannon entropy as the parameter $\alpha$ tended to 1;
• Its value would be between 0 and the upper bound of the unconditional version.

In

Table 1. Summary of the proved properties of all proposed conditional entropies. The question mark indicates that the property is not known to be fulfilled.

$\mathit{f}\left(\mathit{Y}\right|\mathit{X})$	${\mathit{T}}_{\mathit{\alpha }}\left(\mathit{Y}\right|\mathit{X})$	${\mathit{S}}_{\mathit{\alpha }}\left(\mathit{Y}\right|\mathit{X})$	${\mathit{S}}_{\mathit{\alpha }}^{\prime }\left(\mathit{Y}\right|\mathit{X})$	${\mathit{T}}_{\mathit{\alpha }}^{\prime }\left(\mathit{Y}\right|\mathit{X})$
Chain Rule	yes	no	no	no
${\displaystyle \underset{\alpha \to 1}{lim}f\left(Y\right|X)=H\left(Y\right|X)}$	yes	yes	yes	no
${\displaystyle 0\le f\left(Y\right|X)\le \frac{{\left|Y\right|}^{1-\alpha }}{1-\alpha }}$ and $\alpha >1$	yes	yes	yes	yes
${\displaystyle 0\le f\left(Y\right|X)\le \frac{{\left|Y\right|}^{1-\alpha }}{1-\alpha }}$ and $\alpha <1$	no	yes	?	yes
f is non-increasing with $\alpha$	yes	yes	no	yes

, we summarize the properties that the four proposals have (we also added the property of being a non-increasing function with $\alpha$). To conclude, we can say that none of the proposals fulfill all of the properties. The definition $T_{\alpha }\left(Y\right|X)$ is the candidate that fulfills more properties.

For future work, since all definitions focus on possible different aspects of the entropy, it would be important to consider a deeper study in this area and its possible applications, aiming to develop a theory that would emphasize the best proposal for each area, or eventually present an ultimate version for the conditional Tsallis entropy that would satisfy all of the desirable properties.