Skewed Jensen—Fisher Divergence and Its Bounds

Abstract

A non-uniform (skewed) mixture of probability density functions occurs in various disciplines. One needs a measure of similarity to the respective constituents and its bounds. We introduce a skewed Jensen–Fisher divergence based on relative Fisher information, and provide some bounds in terms of the skewed Jensen–Shannon divergence and of the variational distance. The defined measure coincides with the definition from the skewed Jensen–Shannon divergence via the de Bruijn identity. Our results follow from applying the logarithmic Sobolev inequality and Poincaré inequality.

1. Introduction

Comparison of a probability density function with a mixture, i.e., a weighted sum of several density functions, is frequently needed in various disciplines, such as statistics, information theory, signal processing, bioinformatics, machine learning, neuroscience, natural language processing, time series analysis, and many others. To quantify the closeness of the mixed density function to the respective one, the Jensen–Shannon divergence (JSD) has been used since the definition by Lin [1]. JSD is a symmetrized Kullback–Leibler (KL) divergence with an equally weighted mixture. The skewed version of JSD has also been introduced as the generalized JSD in [1]. A symmetrized JSD through a skewed KL divergence has been been studied in [2,3]. By the skewed divergence, we mean that one chooses its reference density function as an unequally weighted density.

Recently, on the other hand, a Jensen–Fisher divergence (JFD) has been introduced [4] to properly detect the highly oscillatory behavior of density functions. The JFD uses relative Fisher information instead of a KL divergence, by which it is more sensitive to the change of densities because of the gradient feature, whereas the KL divergence encapsulates the overall feature. In this paper, we show that the skewed version of the Jensen–Fisher divergence can be defined similarly to the skewed JSD, and express it in terms of the Fisher information of respective density functions.

In the next two sections, we define it in two different ways, but we see that they are identical. Some backgrounds for relative Fisher information and the previous applications to physical sciences are also mentioned there. Then, we provide some lower bounds by applying some integral inequalities for gradient functions. In the last section, we remark on the application of the Sobolev inequality to the skewed Jensen–Fisher divergence.

2. Definition via Jensen–Shannon Divergence

Let $p\left(x\right)$ and $q\left(x\right)$ be two continuous probability density functions on ${\mathbf{R}}^n$. The skewed Jensen–Shannon divergence is defined as the weighted sum of the respective skewed KL divergence:

$$\begin{array}{c}\hfill JS{D}_{\alpha }(p,q):=(1-\alpha )KL(p\parallel (1-\alpha )p+\alpha q)+\alpha KL(q\parallel (1-\alpha )p+\alpha q),\end{array}$$

(1)

where $\alpha \in [0,1]$ controlls the weights in the mixture. We will use the separation symbol ∥ between the two functions for the single divergence and the comma , for the combined divergences. Here, the skewed KL divergence of order $\alpha$ from $p\left(x\right)$ to $q\left(x\right)$ is defined as [1,5,6]

$$\begin{array}{c}\hfill KL(p\parallel (1-\alpha )p+\alpha q):={\int }_{{\mathbf{R}}^n}plog\frac{p}{(1-\alpha )p+\alpha q}dx.\end{array}$$

(2)

We always assume absolute continuity when we encounter a division of density functions. Similarly, from $q\left(x\right)$ to $p\left(x\right)$

$$\begin{array}{c}\hfill KL(q\parallel (1-\alpha )p+\alpha q):={\int }_{{\mathbf{R}}^n}qlog\frac{q}{(1-\alpha )p+\alpha q}dx.\end{array}$$

(3)

Thus, the weighted sum of the above two skewed KL divergences of order $\alpha$ provides the following expression for the definition Equation (1)

$$\begin{array}{c}\hfill JS{D}_{\alpha }(p,q)=S((1-\alpha )p+\alpha q)-(1-\alpha )S\left(p\right)-\alpha S\left(q\right),\end{array}$$

(4)

where $S\left(p\right)=-\int plogpdx$ is the differential entropy of an arbitrary random variable with a density function $p\left(x\right)$, and others are defined similarly. For $p\ne q$, $JS{D}_{\alpha }(p,q)$ shows positivity, which means equivalently that $S\left(p\right)$ is a concave function. When $\alpha =1/2$, Equation (1) gives the usual Jensen–Shannon divergence (JSD) [1,7,8]:

$$\begin{array}{ccc}\hfill JSD(p,q)& =& \frac{1}{2}{\int }_{{\mathbf{R}}^n}plog\frac{2p}{p+q}dx+\frac{1}{2}{\int }_{{\mathbf{R}}^n}qlog\frac{2q}{p+q}dx\hfill \\ & =& S\left(\frac{p+q}{2}\right)-\frac{1}{2}(S\left(p\right)+S\left(q\right)),\hfill \end{array}$$

(5)

which is a symmetric measure with respect to p and q and quantifies the deviation of the mean from their respective density functions in terms of the differential entropy.

Remark 1.

A special instance of $\alpha =1/2$ in Equation (2) was considered in [9]. The definition Equation (1) is also used in [10]. Lin [1] introduced the weighted combination of any finite number of density functions into the Jensen–Shannon divergence in terms of the differential entropy of the constituent functions. He termed it the generalized Jensen–Shannon divergence. However, the expression Equation (1), in terms of the skewed KL divergences, is useful as it clearly shows a distance of the mixture from the respective density functions, and this form motivates us to introduce the definition of the skewed Jensen–Fisher divergence in a similar way as we will see below. A special case of equal weight in the above definition has previously been defined in [2,3]:

$$\begin{array}{c}\hfill \frac{1}{2}KL(p\parallel (1-\alpha )p+\alpha q)+\frac{1}{2}KL(q\parallel (1-\alpha )p+\alpha q).\end{array}$$

To proceed, recall that the de Bruijn identity [11,12,13] relates the differential entropy of p to its Fisher information and it has been extended in such a way that the detailed statistics of the noise do not enter [14]:

$$\begin{array}{c}\hfill \frac{d}{d\delta }S(p+\sqrt{\delta }\eta ){|}_{\delta =0}=\frac{1}{2}I\left(p\right),\end{array}$$

(6)

where $\eta$ is an arbitrary symmetric density function with zero mean and unit variance, which is not necessarily a standard Gaussian. Fisher information of $p\left(x\right)$ on ${\mathbb{R}}^n$ is defined [15,16] as

$$\begin{array}{c}\hfill I\left(p\right):={\int }_{{\mathbf{R}}^n}\frac{{\left|\right|\nabla p\left|\right|}^2}{p}dx,\end{array}$$

(7)

where, as usual, $\left|\right|·\left|\right|$ denotes $l_2$-norm on ${\mathbb{R}}^n$. Fisher information, in general, reflects the gradient content (i.e., sharpness) of probability density functions. This feature is informative as a measure because it is more sensitive to the degree of localization and oscillatory behavior than using differential entropy. Indeed, it has been used to detect information on the radial densities of relativistic and non-relativistic hydrogenic atoms (e.g., [17,18], and many others).

Note that, for the perturbed density functions $p_{\delta }=p+\sqrt{\delta }\eta$ and $q_{\delta }=q+\sqrt{\delta }\eta$, we have

$$\begin{array}{ccc}\hfill \frac{d}{d\delta }S((1-\alpha )p_{\delta }+\alpha q_{\delta }){|}_{\delta =0}& =& \frac{d}{d\delta }S((1-\alpha )p+\alpha q+\sqrt{\delta }\eta ){|}_{\delta =0}\hfill \\ & =& \frac{1}{2}I((1-\alpha )p+\alpha q).\hfill \end{array}$$

(8)

Thus, in view of Equation (4), the skewed Jensen–Fisher divergence is induced by the skewed Jensen–Shannon divergence, as described in the following result:

Proposition 1.

The derivative of the skewed Jensen–Shannon divergence between the two perturbed density functions $p_{\delta }$ and $q_{\delta }$ can be given as half of the skewed Jensen–Fisher divergence, with the same order between the two original density functions:

$$\begin{array}{c}\hfill \frac{d}{d\delta }JS{D}_{\alpha }(p_{\delta },q_{\delta }){|}_{\delta =0}=-\frac{1}{2}JF{D}_{\alpha }(p,q),\end{array}$$

(9)

by which we can define the skewed Jensen–Fisher divergence of order α as follows:

Definition 1.

For $\alpha \in [0,1]$, let p and q be two probability density functions. Then, it is defined by

$$\begin{array}{c}\hfill JF{D}_{\alpha }(p,q)=(1-\alpha )I\left(p\right)+\alpha I\left(q\right)-I((1-\alpha )p+\alpha q).\end{array}$$

(10)

This definition is consistent with another one given in the next section. The special case of $\alpha =1/2$ was devised and applied to see the behavior of some probability distributions in [4]:

$$\begin{array}{c}\hfill \frac{d}{d\delta }JS{D}_{\frac{1}{2}}(p,q){|}_{\delta =0}=\frac{1}{2}I\left(\frac{p+q}{2}\right)-\frac{1}{4}\left(I\left(p\right)+I\left(q\right)\right)=-\frac{1}{2}JF{D}_{\frac{1}{2}}(p,q).\end{array}$$

(11)

The defined form Equation (10) can be advantageous, because we can control the mixture of Fisher information by changing the skew parameter α.

Remark 2.

A connection to nonequilibrium processes through the de Bruijn-type identity can be obtained. The probability density functions $p(x,t)$ and $q(x,t)$ for systems described by heat equations and linear Fokker-Planck equations satisfy the de Bruijn-type identity [19]

$$\begin{array}{c}\hfill \frac{d}{dt}KL(p\parallel q)=-\mathcal{D}I\left(p\right|q),\end{array}$$

where t is time and $\mathcal{D}$ is the diffusion constant associated with the probability current of the equations. We will define $I\left(p\right|q)$ in the right-hand side (i.e., the relative Fisher information) soon in the next section. Thus, we have an expression of $JF{D}_{\alpha }(p,q)$ in terms of the time derivative of the skewed KL divergence:

$$\begin{array}{c}\hfill JF{D}_{\alpha }(p,q)=-\frac{1}{\mathcal{D}}\frac{d}{dt}\left\{KL(p\parallel (1-\alpha )p+\alpha q)+KL(q\parallel (1-\alpha )p+\alpha q)\right\}.\end{array}$$

(12)

3. Definition via Relative Fisher Information

The relative Fisher information of p with respect to q is defined as

$$\begin{array}{c}\hfill I\left(p\right|q):={\int }_{{\mathbf{R}}^n}p\left(x\right)\parallel \nabla log\frac{p\left(x\right)}{q\left(x\right)}{\parallel }^{2}dx,\end{array}$$

(13)

where we use the separation symbol | between p and q instead of ∥, according to the convention. It is an asymmetric measure in p and q. It is non-negative and vanishes when $p=q$, which is desirable as a measure of statistical distance. Moreover, one does not suffer from a difficult computation of the normalization factor of q, which is often non-normalized in many cases: that is, $I\left(p\right|kq)=I(p\left|q\right)$ for any constant $k>0$. This merit has recently been used in the machine learning community, and in related fields such as Bayesian statistical inference [20,21,22,23,24,25]. Since this measure involves derivatives of density functions, it is more sensitive to reflect the oscillatory behavior in density functions that appear in quantum mechanical systems [4]. That is, since the probability densities of quantum systems such as a particle in a quantum potential well, isotropic harmonic oscillators, and hydrogen-like atoms oscillate spatially, and these characters increase as the corresponding quantum numbers increase, the Jensen–Shannon divergence is not necessarily an informative measure.

Some Background on Relative Fisher Information

First, we mention the original source of relative Fisher information, which has been defined and was named differently in different disciplines. As far as the author knows, the form was first proposed in 1978 in Equations (14) and (25) by Hammad in [26], in which the gradient of a density function $f(x,\theta )$ is taken with respect to the parameter $\theta$:

$$\begin{array}{c}\hfill {\int }_{{\mathbf{R}}^n}p(x,\theta ){\left(\frac{\partial }{\partial \theta }log\frac{p(x,\theta )}{q(x,\theta )}\right)}^{2}dx.\end{array}$$

(14)

In 1986, Barron [27] defined the counterpart of the above by replacing the $\theta$-derivative with the x-derivative (i.e., the so-called shift transformation families) to show that the sequence of Fisher information converges to the Cramér-Rao bound. He termed it standardized Fisher information. Johnson [28,29] in 2000 has also defined the same form as Equation (13) in connection with the central limit theorem and termed it as Fisher information distance. Moreover, in mathematics, Otto and Villani introduced it with a measure theoretical notation as an equivalent of the logarithmic Sobolev inequality [30,31]. On the other hand, Hyvärinen [20,21], in 2005, was able to reach the proposal of the same form to estimate statistical models whose normalization constant of a density function is not known in closed form. In a recent machine learning community, some researchers call it Fisher divergence [22,23]. However, Hammad used the term Fisher divergence to denote the symmetrized form $I\left(p\right|q)+I(q\left|p\right)$ (see Equation (41) of [26]). Relative Fisher information is potentially useful in many areas of science. In quantum mechanical applications, the excited states of density functions (the square of the wave functions) are compared with its ground state one, and the information shows significant changes according to the eigen states in respective quantum systems under potentials (e.g., [17,32,33,34,35,36,37,38]). We mention two more examples of the role of relative Fisher information in statistical physics. It connects the phase space gradient of the dissipated work in nonequilibrium processes: the distance between canonical equilibrium densities that correspond to forward and backward processes is proportional to the average of the squared gradient of the work dissipated into the environment [39]. When heat exchange occurs between two bodies with different temperatures under the exchange fluctuation theorem, relative Fisher information of the heat probability with respect to its reverse is proportional to the square of the inverse temperature difference before contact [40].

We now define the skewed Jensen–Fisher divergence of order $\alpha$ as follows:

Definition 2.

For $\alpha \in [0,1]$, let p and q be two probability density functions. Then it is defined by

$$\begin{array}{c}\hfill JF{D}_{\alpha }(p,q)=(1-\alpha )I\left(p\right|(1-\alpha )p+\alpha q)+\alpha I\left(q\right|(1-\alpha )p+\alpha q).\end{array}$$

(15)

When $\alpha =0$ and $\alpha =1$, it vanishes. By this definition and Equation (13), the skewed Jensen–Fisher divergence of order α can be expressed by the Fisher information of respective density functions

$$\begin{array}{ccc}\hfill JF{D}_{\alpha }(p,q)& =& (1-\alpha ){\int }_{{\mathbf{R}}^n}p\parallel \frac{\nabla p}{p}-\frac{(1-\alpha )\nabla p+\alpha \nabla q}{(1-\alpha )p+\alpha q}{\parallel }^{2}dx\hfill \\ & +& \alpha {\int }_{{\mathbf{R}}^n}q\parallel \frac{\nabla q}{q}-\frac{(1-\alpha )\nabla p+\alpha \nabla q}{(1-\alpha )p+\alpha q}{\parallel }^{2}dx\hfill \\ & =& (1-\alpha ){\int }_{{\mathbf{R}}^n}\frac{{\left|\right|\nabla p\left|\right|}^2}{p}dx+\alpha {\int }_{{\mathbf{R}}^n}\frac{{\left|\right|\nabla q\left|\right|}^2}{q}dx-{\int }_{{\mathbf{R}}^n}\frac{\left|\right|(1-\alpha )\nabla p+{\alpha \nabla q\left|\right|}^2}{(1-\alpha )p+\alpha q}dx\hfill \\ & =& (1-\alpha )I\left(p\right)+\alpha I\left(q\right)-I\left(\right(1-\alpha )p+\alpha q),\hfill \end{array}$$

(16)

which is the same form as Equation (10) as desired. From the definitions Equation (13) and Equation (10), $JF{D}_{\alpha }(p,q)$ is clearly positive, and vanishes when $p=q$. This property also means that Fisher information $I\left(p\right)$ is a convex function. We easily find that this divergence measure does not keep symmetry with respect to the exchange of p and q; that is, $JF{D}_{\alpha }(p,q)\ne JF{D}_{\alpha }(q,p)$. Instead, under the index change $\alpha \leftrightarrow 1-\alpha$, the following relation is satisfied:

$$\begin{array}{c}\hfill JF{D}_{1-\alpha }(p,q)=JF{D}_{\alpha }(q,p).\end{array}$$

(17)

Next, we find that an immediate consequence of the application of the Blachman–Stam inequality gives an interesting lower bound for $JF{D}_{\alpha }(p,q)$. The Blachman–Stam inequality [11,41] asserts that, for all $\alpha \in [0,1]$, Fisher information of p and q satisfies an inequality

$$\begin{array}{c}\hfill \alpha I\left(p\right)+(1-\alpha )I\left(q\right)\ge I(\sqrt{\alpha }p+\sqrt{1-\alpha }q).\end{array}$$

(18)

Therefore, we find the following bound:

Proposition 2.

The skewed Jensen–Fisher divergence of order α is lower bounded by the difference of two Fisher information as follows:

$$\begin{array}{c}\hfill JF{D}_{\alpha }(p,q)\ge I(\sqrt{1-\alpha }p+\sqrt{\alpha }q)-I((1-\alpha )p+\alpha q).\end{array}$$

(19)

From the positivity of $JF{D}_{\alpha }(p,q)$ because of the definition Equation (15) and of the positivity of the relative Fisher information, the above inequality also means an interesting relation $I(\sqrt{1-\alpha }p+\sqrt{\alpha }q)\ge I((1-\alpha )p+\alpha q)$.

Remark 3.

A similar bound can be obtained for the skewed Jensen–Shannon divergence of order α by way of the Shannon–Stam inequality [11]:

$$\begin{array}{c}\hfill (1-\alpha )S\left(p\right)+\alpha S\left(q\right)\ge S(\sqrt{1-\alpha }p+\sqrt{\alpha }q).\end{array}$$

Applying this to the definition Equation (1), we readily obtain an upper bound instead of the lower one

$$\begin{array}{c}\hfill JS{D}_{\alpha }(p,q)\le S((1-\alpha )p+\alpha q)-S(\sqrt{1-\alpha }p+\sqrt{\alpha }q).\end{array}$$

From the positivity of $JS{D}_{\alpha }(p,q)$, this inequality also means the relation $S((1-\alpha )p+\alpha q)\ge S(\sqrt{1-\alpha }p+\sqrt{\alpha }q)$.

4. A Bound by Skewed Jensen–Shannon Divergence

It will be useful to relate a divergence measure to another existing one in terms of a bound because the convergence (or similarity) in a divergence can be stronger or weaker than convergence in other measures. This section aims at providing such an example. We consider a lower bound from an integral inequality for gradient functions f. Contrary to the Sobolev inequality, the logarithmic Sobolev inequality does not depend on dimension n, in which we consider f as probability density functions. Recalling that when all functions f and the distributional gradient $\nabla f$ belong to $L^2$, the probability density function $\rho \left(x\right)$ on ${\mathbf{R}}^n$ satisfies the Gross’ logarithmic Sobolev inequality [42]

$$\begin{array}{c}\hfill {\int }_{{\mathbf{R}}^n}{\left|f\right|}^{2}log\left|f\right|\rho \left(x\right)dx\le c{\int }_{{\mathbf{R}}^n}{\left|\right|\nabla f\left|\right|}^2\rho \left(x\right)dx+{\left|\right|f\left|\right|}_2^2{log\left|\right|f\left|\right|}_2^2,\end{array}$$

(20)

where the constant c is independent of f. As a special case, when $\rho \left(x\right)$ is the standard Gaussian density with zero mean and unit variance, we have $c=1$ (the so-called LSI(1)). Thus, by applying the above by identifying $f=\sqrt{p/q}$ and $\rho \left(x\right)=q\left(x\right)$, we have an inequality

$$\begin{array}{c}\hfill \frac{c}{4}I\left(p\right|q)\ge \frac{1}{2}KL(p\parallel q).\end{array}$$

(21)

Therefore, the skewed Jensen–Fisher divergence of order $\alpha$ is lower bounded by the skewed Jensen–Shannon divergence, as

$$\begin{array}{ccc}\hfill JF{D}_{\alpha }(p,q)& \ge & \frac{2}{c}(1-\alpha )KL(p\parallel (1-\alpha )p+\alpha q)+\frac{2}{c}\alpha KL(q\parallel (1-\alpha )p+\alpha q)\hfill \\ & =& \frac{2}{c}JS{D}_{\alpha }(p,q).\hfill \end{array}$$

(22)

This lower bound indicates that, when we can set the constant as $c\ge 2$, we have a tighter distance than the skewed Jensen–Shannon divergence gives.

5. A Lower Bound by Variational Distance

The purpose of this section is to find a bound of the skewed Jensen–Fisher divergence of order $\alpha$ in terms of the variational distance between p and q. We apply the Poincaré inequality to bound $JF{D}_{\alpha }(p,q)$ from below. It states that the probability density function $\rho \left(x\right)$ on ${\mathbf{R}}^n$ satisfies the inequality (e.g., [31])

$$\begin{array}{c}\hfill {\int }_{{\mathbf{R}}^n}{\left(f\left(x\right)-{\int }_{{\mathbf{R}}^n}f\left(y\right)dy\right)}^2\rho \left(x\right)dx\le \frac{1}{\lambda }{\int }_{{\mathbf{R}}^n}{\left|\right|\nabla f\left|\right|}^2\rho \left(x\right)dx\end{array}$$

(23)

for all functions f that belong to $L^2$, such that its distributional gradient also belongs to $\nabla f\in L^2$. The Poincaré constant or the spectral gap $\lambda$ is a positive constant.

Proposition 3.

The skewed Jensen–Fisher divergence of order α is lower bounded by the variational distance V as

$$JF{D}_{\alpha }(p,q)\ge \frac{\lambda }{2}{\alpha }^2{V}^2.$$

(24)

Proof. 

Identifying $f=\sqrt{p/q}$ and $\rho \left(x\right)=q$ in the Poincaré inequality, the right-hand side becomes $I\left(p\right|q)/4\lambda$. On the other hand, we evaluate the left-hand side as

$$\begin{array}{c}\hfill 1-{\left({\int }_{{\mathbf{R}}^n}\sqrt{pq}dy\right)}^2\ge \frac{1}{2}{\int }_{{\mathbf{R}}^n}{(\sqrt{p}-\sqrt{q})}^{2}dy,\end{array}$$

(25)

where we have used the magnitute relationship $\int \sqrt{pq}dy\ge {(\int \sqrt{pq}dy)}^2$. In this sense, the bound becomes loose here, and it states

$$\begin{array}{c}\hfill \frac{1}{4\lambda }I\left(p\right|q)\ge \frac{1}{2}{\int }_{{\mathbf{R}}^n}{(\sqrt{p}-\sqrt{q})}^{2}dy.\end{array}$$

(26)

We want to bound $JF{D}_{\alpha }(p,q)$ in terms of the variational distance V:

$$\begin{array}{c}\hfill V={\int }_{{\mathbf{R}}^n}|p\left(x\right)-q\left(x\right)|dx.\end{array}$$

(27)

Applying the Schwarz inequality, one has $V^2\le (\int {(\sqrt{p}-\sqrt{q})}^{2}dx)(\int {(\sqrt{p}+\sqrt{q})}^{2}dx)$. Thus, the second factor of the right-hand side can be bounded by 4 due to $\int \sqrt{pq}dx\le 1$. Hence, the right-hand side of Equation (26), which is the squared Hellinger distance aside from $1/2$, is lower bounded by $V^2/8$. Regarding q as $(1-\alpha )p+\alpha q$ in this inequality, the skewed relative Fisher information is lower bounded as

$$\begin{array}{c}\hfill I\left(p\right|(1-\alpha )p+\alpha q)\ge \frac{\lambda }{2}{\alpha }^2{V}^2.\end{array}$$

(28)

Finally, from the definition of $JF{D}_{\alpha }(p,q)$ (i.e., Equation (15)), we have the lower bound as desired. □

6. Summary and Discussion

We have defined the skewed Jensen–Fisher divergence of order $\alpha$ and found some bounds. These bounds come from inequalities in which the broadness of a function in terms of $L^q$ norm bounds an average gradient of a function from below in suitable ways. The logarithmic Sobolev inequality is a tool that provides such a link with the skewed Jensen–Fisher divergence through the relative Fisher information. We know that the Sobolev inequality (not the logarithmic) is also an integral inequality that involves the gradient of a function. Therefore, one may think that it offers a bound for our divergence measure because the Sobolev inequality also relates the $L^2$ norm of the gradient of a function with an $L^r$ norm of the function, where r is an index that depends on the dimension of space n. However, an unthinking application of it leads to inappropriate bounds. To see this aspect clearly, we consider the case $n\ge 3$ and see first that $JF{D}_{\alpha }(p,q)$ is lower bounded in terms of the Chernoff $\alpha$ distance; then, we see that the derived bound loses its effectiveness.

When $n\ge 3$, the Sobolev inequality states [43]

$$\begin{array}{c}\hfill {\parallel \nabla f\parallel }_2^2\ge S_n{\parallel f\parallel }_r^2,S_n=\frac{n(n-2)2^{\frac{2}{n}}{\pi }^{1+\frac{1}{n}}}{4\Gamma {((n+1)/2)}^{\frac{2}{n}}},\end{array}$$

(29)

where r is the Sobolev exponent $r=2n/(n-2)$. Identifying $f=\sqrt{p/q}$ and taking the measure for the norms as $q\left(x\right)dx$, we have

$$\begin{array}{c}\hfill {\parallel \nabla f\parallel }_2^2=\frac{1}{4}I\left(p\right|q),{\parallel f\parallel }_r^2={\left({\int }_{{\mathbf{R}}^n}{p}^{\frac{r}{2}}{q}^{1-\frac{r}{2}}dx\right)}^{2/r}.\end{array}$$

(30)

Note that the power exponent appearing in the latter is $r/2=n/(n-2)>1$. We recall here that the Chernoff $\alpha$ distance [44] between two density functions p and q is defined as (we here use $\beta$ instead of $\alpha$ to avoid confusion with $\alpha$ for $JF{D}_{\alpha }(p,q)$)

$$\begin{array}{c}\hfill C_{\beta }(p,q)=-log\left({\int }_{{\mathbf{R}}^n}{p}^{\beta }{q}^{1-\beta }dx\right).\end{array}$$

(31)

This is also called the skewed Bhattacharyya distance [45], because when $\beta =1/2$, it reduces to the Bhattacharyya distance [46]. The distance $C_{\beta }(p,q)$ becomes zero when $p=q$, and takes positive values for $0\le \beta \le 1$; however, it can be negative for other ranges, which is our present case. Indeed, for $\beta \ge 1$, $C_{\beta }(p,q)\le 0$ follows, because $C_{\beta }(p,q)$ is concave with respect to $\beta$ and $C_0(p,q)=C_1(p,q)=0$. Thus, rewriting ${\int }_{{\mathbf{R}}^n}{p}^{\beta }{q}^{1-\beta }dx=e^{-C_{\beta }(p,q)}$, the Sobolev inequality provides

$$\begin{array}{c}\hfill \frac{1}{4}I\left(p\right|q)\ge S_{n}exp\left[-\frac{n-2}{n}{C}_{\frac{n}{n-2}}(p,q)\right].\end{array}$$

(32)

Therefore, identifying this q as $\overline{pq}:=(1-\alpha )p+\alpha q$ and from the definition of Equation (15), we have the following lower bound in terms of the Chernoff $\alpha$ distance:

$$\begin{array}{c}\hfill JF{D}_{\alpha }(p,q)\ge 4S_n\left\{(1-\alpha )exp\left[-\frac{n-2}{n}{C}_{\frac{n}{n-2}}(p,\overline{pq})\right]+\alpha exp\left[-\frac{n-2}{n}{C}_{\frac{n}{n-2}}(q,\overline{pq})\right]\right\},\\ \hfill (n\ge 3).\end{array}$$

(33)

The value of the Chernoff distance becomes negatively large as p and q differ largely, and when we consider a higher dimensional space. Indeed, the coefficient $S_n$ is an increasing function of n, which starts from $S_3=5.4779...$. Even when the Chernoff distance vanishes by increasing n (recall $C_1(p,q)=0$) or by approaching the one density function q to the other one p, Equation (33) asserts the finitely large lower bound $4S_n$. Thus, the derived lower bound loses its effectiveness. The same argument is true for two- and one-dimensional cases. Where does this inexpedience come from? The Sobolev inequality holds for functions f that have compact support (thus vanishes at infinity) with $\nabla f\in L^2\left({\mathbf{R}}^n\right)$. On the other hand, the log-Sobolev inequality holds for a class of functions with $f\in L^2\left({\mathbf{R}}^n\right)$ and $\nabla f\in L^2\left({\mathbf{R}}^n\right)$. The density functions vanish at infinity. However, when we choose $f=\sqrt{p/q}$ to make $I\left(p\right|q)$ (and $\sqrt{q/p}$ for $I\left(q\right|p)$), the ratio $p/q$ (and $q/p$) does not vanish when $\left|x\right|\to \infty$ or at the boundary. This fact indicates that the Sobolev inequality cannot be applied in our present study.