Causal Information Rate

Abstract

Information processing is common in complex systems, and information geometric theory provides a useful tool to elucidate the characteristics of non-equilibrium processes, such as rare, extreme events, from the perspective of geometry. In particular, their time-evolutions can be viewed by the rate (information rate) at which new information is revealed (a new statistical state is accessed). In this paper, we extend this concept and develop a new information-geometric measure of causality by calculating the effect of one variable on the information rate of the other variable. We apply the proposed causal information rate to the Kramers equation and compare it with the entropy-based causality measure (information flow). Overall, the causal information rate is a sensitive method for identifying causal relations.

1. Introduction

Entropy-related concepts and information theory [1,2,3,4,5,6,7,8,9] are useful for understanding complex dynamics in equilibrium and out of equilibrium. Examples include information (Shannon) entropy (measuring disorder, or lack of information) [1], Fisher information [2], relative entropy [3], mutual information [10], and their microscopic versions (e.g., trajectory entropy) [11,12], etc. In particular, while, in equilibrium, the Shannon entropy has a unique thermodynamic meaning, this is no longer the case in non-equilibrium, with different proposals for generalized entropies (e.g., see the review paper of Reference [13] and references therein). Recent years have witnessed the increased awareness of information as a useful physical concept, for instance, in resolving the famous Maxwell’s demon paradox [14], setting various thermodynamic inequality/uncertainty relations [15,16,17], and establishing theoretical and conceptual links between physics and biology [18]. Information-related ideas are also useful to uncover unexpected relations between apparently unrelated problems, for instance, the connections between Fisher information and Schrödinger equation, inspiring new development in non-equilibrium statistical mechanics [19].

We have recently proposed information-geometric theory as a powerful tool to understand non-equilibrium stochastic processes that often involve high temporal variabilities and large fluctuations [20,21,22,23,24,25,26,27,28,29,30,31,32], as often the case of rare, extreme events. This is based on the surprisal rate, $r(x,t)={\partial }_{t}s(x,t)=-{\partial }_{t}lnp(x,t)$, where $p(x,t)$ is a probability density function (PDF) of a random variable x at time t, and $s(x,t)=-lnp(x,t)$ is a local entropy. $r(x,t)$, informing how rapidly $p(x,t)$ or $r(x,t)$ changes in time, is especially useful for understanding time-varying non-equilibrium processes. As the name indicates, the surprisal rate r measures the degree of surprise when $p(x,t)$ changes in time (no surprise in equilibrium with $r=0$). We can easily show that the average of the surprisal rate $\int dxp(x,t)r(x,t)=0$ since $\int dxp(x,t)=1$. We note that, in this paper, averages refer to ensemble averages, which vary with time. A non-zero value is obtained from the second moment of $r(x,t)$ as

$$\mathcal{E}\left(t\right)={\mathsf{\Gamma }}^2\left(t\right)=\int dxp(x,t){\left(r(x,t)\right)}^2=\int dxp(x,t){({\partial }_{t}lnp(x,t))}^2,$$

(1)

where $\mathsf{\Gamma }\left(t\right)$ represents the information rate at which a new information is revealed (a new statistical state is accessed) due to time-evolution. Alternatively, $\tau \left(t\right)=\mathsf{\Gamma }{\left(t\right)}^{-1}$ is the characteristic time scale over which information changes, linked to the smallest time scale of fluctuations [17].

It is important to highlight that $\mathcal{E}$, $\mathsf{\Gamma }$, and $\tau$ have the dimensions of (time)${}^{-2}$, (time)${}^{-1}$, and (time), respectively. In addition, we note that $\mathcal{E}$ is proportional to the average of an infinitesimal relative entropy (Kullback–Leibler divergence) (e.g., see Reference [20]),

$$\begin{array}{ccc}\hfill \mathcal{E}& =& \underset{dt\to 0}{lim}\frac{2}{{\left(dt\right)}^2}\int dxp(x,t+dt)ln\left[\frac{p(x,t+dt)}{p(x,t)}\right]=\underset{dt\to 0}{lim}\frac{2}{{\left(dt\right)}^2}\int dxp(x,t)ln\left[\frac{p(x,t)}{p(x,t+dt)}\right].\hfill \end{array}$$

(2)

The total change in information between the initial time 0 and the time t is then obtained by integrating $\frac{1}{\tau \left(t\right)}$ over time as $\mathcal{L}\left(t\right)={\int }_0^t\frac{d{t}_1}{\tau \left(t_1\right)}$ which is the information length, quantifying the total number of statistically different states that a system passes through in time. In the limit of a Gaussian PDF where the variance is constant in time, one statistically distinguishable state is generated when a PDF peak moves by one standard deviation since the latter provides the uncertainty in measuring the peak position of the PDF. In a nutshell, $\mathcal{L}$ is an information-geometric measure, enabling us to quantify how the “information” unfolds in time by dimensionless distance. Unlike other information measures, $\mathcal{E}$, $\tau$, and $\mathcal{L}$ are invariant under (time-independent) change of variables and are not system-specific. This non-system-specificity is especially useful for comparing the evolution of different variables/systems having different units.

Furthermore, $\mathcal{L}$ is a path-dependent dimensionless distance and is uniquely defined as a function of time for fixed parameters and initial condition. These properties are advantageous for quantifying correlation in time-varying data and understanding self-organization, long memory, and hysteresis involved in phase transitions [20,24,27,28,29,30,32]. In particular, we recently investigated a non-autonomous Kramer equation by including a sudden perturbation to the system to mimic the onset of a sudden event [32], demonstrating that our information rate predicts the onset of a sudden event better than one of the entropy-based measures (information flow) (see Section 5.4 for details).

The purpose of this paper is to develop an information-geometric measure of causality (the causal information rate) by generalizing $\mathcal{E}$ ($\mathsf{\Gamma }$). Like Reference [32], our intention here is not on modeling the appearance of rare, extreme events (that are nonlinear, non-Gaussian) themselves, but on developing a new information-geometric causality method which is useful for predicting and understanding those events. The remainder of this paper is organized as follows. We propose the causal information rate in Section 2 and apply it to the Kramers equation in Section 3. One of the entropy-based methods (the information flow) is calculated in Section 4 and is compared with our proposed method in Section 5. Conclusions are provided in Section 6. Appendix A, Appendix B and Appendix C show some detailed steps involved in our calculations. We note that, while the usual convention in statistics and probability theory is to use upper case letters for random variables and lower case letters for their realizations, we do not make such distinctions in this paper as their meanings should be clear from the context.

2. Causal Information Rate

Information theoretical measures of causality are often based on entropy, joint entropy, conditional entropy, or mutual information, where the causality is measured by the improvement of the predictability of one of the variables at future time by the knowledge of the other variable, the improved predictability being measured by the decrease in entropy [16,33,34,35,36,37,38,39,40,41]. However, there have been questions raised as to whether predictability improvement (e.g., as measured by the Granger causality, transfer entropy) is directly linked to causality (e.g., Reference [39]) and the suggestion that causality is better understood by performing an intervention experiment to measure the effect of the change (some type of perturbation or intervention) in one variable on another. In particular, spurious causalities between the two (observed) state variables can arise through unobserved state variables that interact with both (observed) state variables, calling for care in dealing with a system with more than two variables. On the other hand, to deal with strongly time-dependent data, the concepts of transfer entropy rate [34], information flow [16,40,41], etc., are proposed.

It is not the aim of this paper to provide detailed discussions about these methods, but to introduce a new information geometric measure of causality (see below) and to compare our new method with one of them (information flow) (see Section 4). Our new information geometric measure of causality focuses on how one variable affects the information rate of another variable. To this end, we generalize $\mathsf{\Gamma }$ in Equation (1) and define the causal information rate for multiple variables.

In order to demonstrate the basic idea required, it is instructive to consider a stochastic system consisting of two variables $X_1$ and $X_2$ which have a bivariate joint PDF $p(X_1,t_1;X_2,t_2)$ at different times $t_1$ and $t_2$, its equal-time joint PDF $p(X_1,t;X_2,t)\equiv p(X_1,X_2,t)$, and the conditional PDFs $p(X_2,t_2|X_1,t_1)=p(X_2,t_2;X_1,t_1)/p(X_1,t_1)$, as well as marginal PDFs $p(X_1,t)=\int d{X}_{2}p(X_1,X_2,t)$ and $p(X_2,t)=\int d{x}_{1}p(X_1,X_2,t)$. Using the index notation $i,j=1,2$, we then define causal information rate ${\mathsf{\Gamma }}_{i\to j}$ for $i\ne j$ from the variable $X_i$ to $X_j$ as follows:

$$\begin{array}{ccc}\hfill {\mathsf{\Gamma }}_{i\to j}& \equiv & {\mathsf{\Gamma }}_j^{*}-{\mathsf{\Gamma }}_j,\hfill \end{array}$$

(3)

$$\begin{array}{ccc}\hfill {\mathcal{E}}_j& \equiv & {\mathsf{\Gamma }}_j{\left(t\right)}^2=\int d{X}_{j}p(X_j,t){\left({\partial }_{t}ln(p(X_j,t)\right)}^2,\hfill \end{array}$$

(4)

$$\begin{array}{ccc}\hfill {\mathcal{E}}_j^{*}& \equiv & {\mathsf{\Gamma }}_j^{*}{\left(t\right)}^2=\underset{t_1\to t^{+}}{lim}\int d{X}_{i}d{X}_{j}p(X_j,t_1;X_i,t){\left({\partial }_{t_1}ln\left[p(X_j,t_1|X_i,t)\right]\right)}^2\hfill \\ & =& \underset{t_1\to t^{+}}{lim}\int d{X}_{i}d{X}_{j}p(X_j,t_1;X_i,t){\left({\partial }_{t_1}ln\left[p(X_j,t_1;X_i,t)\right]\right)}^2.\hfill \end{array}$$

(5)

Here, ${\partial }_{t_1}p(X_i,t)=0$ for $t_1\ne t$ was used. ${\mathsf{\Gamma }}_i=\frac{1}{{\tau }_i\left(t\right)}$ represents the information rate of $X_i$ with its characteristic timescale ${\tau }_i\left(t\right)$. Note that the subscript j in ${\mathsf{\Gamma }}_j$ denotes that the information rate is calculated for the variable j. Since ${\mathsf{\Gamma }}_j$ contains the contribution from the variable j itself and other variable $i\ne j$, we denote the (auto) contribution from j-th itself (where the other variable $i\ne j$ is frozen in time) by using the superscript *. That is, ${\mathsf{\Gamma }}_j^{*}$ represents the information rate of $X_j$ for given (frozen) $X_i$. Subtracting ${\mathsf{\Gamma }}_j$ from ${\mathsf{\Gamma }}_j^{*}$ in Equation (3) then gives us the contribution of dynamic (time-evolving) $X_i$ to ${\mathsf{\Gamma }}_j$, signifying how $X_i$ instantaneously influences the information rate of $X_j$.

It is important to note that, as in the case of the information rate $\mathsf{\Gamma }$ or $\mathcal{L}$, the calculation of ${\mathsf{\Gamma }}_{i\to j}$, ${\mathsf{\Gamma }}_j$ and ${\mathsf{\Gamma }}_j^{*}$ in Equations (3)–(5) does not require the knowledge of the main governing equations (stochastic differential equations). This is because Equations (3)–(5) can be calculated from any (numerical or experimental) data as long as time-dependent (marginal, joint) PDFs can be constructed. For instance, we used a time-sliding window method to construct time-dependent PDFs of different variables and then calculated $\mathcal{E}$ and $\mathcal{L}$ to analyze numerically generated time-series data for fusion turbulence [26], time-series music data [20], and numerically generated time-series data for global circulation model [28]. However, it is not always clear how many hidden variables are in a given data set.

It is also useful to note that, as in the case of Equation (2), Equation (5) can be shown to be related to the infinitesimal relative entropy as

$$\begin{array}{ccc}\hfill {\mathcal{E}}_j^{*}& =& {{\mathsf{\Gamma }}_j^{*}}^2=2\underset{dt\to 0}{lim}\frac{1}{{\left(dt\right)}^2}\int d{X}_{i}d{X}_{j}p(X_j,t+dt;X_i,t)ln\left[\frac{p(X_j,t+dt;X_i,t)}{p(X_j,X_i,t)}\right].\hfill \end{array}$$

(6)

The method presented above is for a stochastic process with two variables. For stochastic processes involving three or more variables ($i,j=1,2,\dots n$, $n\ge 3$), one way to proceed is to calculate multivariate PDFs, and then bivariate joint PDFs $p(X_j,t_1;X_j,t_2)$ and its equal-time joint PDF $p(X_i,t;X_j,t)\equiv p(X_i,X_j,t)$, and marginal PDFs $p(X_i,t)$ and $p(X_j,t)$, and then calculate the information rate from $X_i$ to $X_j$, where $i\ne j$ ($i,j=1,2,\dots n$, $n\ge 3$), via Equations (3)–(5). This will give us an effective causal information rate. Another way is to deal with the multivariate PDFs directly (to be reported in future work).

3. Kramers Equation

To demonstrate how the methods Equations (3)–(5) work, in this section, we investigate an analytically solvable, Kramers equation, governed by the following Langevin equations [31,42]:

$$\begin{array}{ccc}\hfill \frac{dx}{dt}& =& v,\hfill \end{array}$$

(7)

$$\begin{array}{ccc}\hfill \frac{dv}{dt}& =& -\gamma v-{\omega }^{2}x+\xi .\hfill \end{array}$$

(8)

Here, $\xi$ is a short (delta) correlated Gaussian noise with a zero average (mean) $\langle \xi \rangle =0$ and the strength D with the following property:

$$\langle \xi \left(t\right)\xi \left(t^{\prime }\right)\rangle =2D\delta (t-t^{\prime }),$$

(9)

where the angular brackets denote the average over $\xi$ ($\langle \xi \rangle =0$).

Assuming an initial Gaussian PDF, time-dependent PDFs remain Gaussian for all time. Thus, the bivariate joint PDF $p(x,t_1;v,t_2)$ and the marginal PDFs $p(x,t)$ and $p(v,t)$ are completely determined by covariance and mean values as

$$\begin{array}{ccc}\hfill p(x,t_1;v,t_2)& =& \frac{1}{\left(2\pi \right)\sqrt{\left|\mathsf{\Sigma }\right(t_1,t_2\left)\right|}}exp(-\frac{1}{2}{\mathsf{\Sigma }}_{ij}^{-1}(t_1,t_2)(X_i-\langle X_i\rangle )(X_j-\langle X_j\rangle )),\hfill \end{array}$$

(10)

$$\begin{array}{ccc}\hfill p(x,t)& =& \sqrt{\frac{{\beta }_x}{\pi }}exp(-{\beta }_x{(x-\langle x\left(t\right)\rangle )}^2),\hfill \end{array}$$

(11)

$$\begin{array}{ccc}\hfill p(v,t)& =& \sqrt{\frac{{\beta }_v}{\pi }}exp(-{\beta }_v{(v-\langle v\left(t\right)\rangle )}^2).\hfill \end{array}$$

(12)

Here, $(X_1,X_2)=(x\left(t_1\right),v\left(t_2\right))$. $\langle x\rangle$ and $\langle v\rangle$ are the mean values. $\mathsf{\Sigma }(t_1,t_2)$ is the covariance matrix with the elements ${\mathsf{\Sigma }}_{11}={\mathsf{\Sigma }}_{xx}\left(t_1\right)=\langle {\left(\delta x\left(t_1\right)\right)}^2\rangle =\frac{1}{2{\beta }_x\left(t_1\right)}$, ${\mathsf{\Sigma }}_{12}={\mathsf{\Sigma }}_{21}={\mathsf{\Sigma }}_{xv}(t_1,t_2)=\langle \delta x\left(t_1\right)\delta v\left(t_2\right)\rangle$, and ${\mathsf{\Sigma }}_{22}={\mathsf{\Sigma }}_{vv}\left(t_2\right)=\langle {\left(\delta v\left(t_2\right)\right)}^2\rangle =\frac{1}{2{\beta }_v\left(t_2\right)}$, where $\delta x\left(t_1\right)=x-\langle x\left(t_1\right)\rangle$ and $\delta v\left(t_2\right)=v\left(t_2\right)-\langle v\left(t_2\right)\rangle$. ${\mathsf{\Sigma }}^{-1}$ is the inverse of $\mathsf{\Sigma }$, while $\left|\mathsf{\Sigma }\right|={\mathsf{\Sigma }}_{11}{\mathsf{\Sigma }}_{22}-{\mathsf{\Sigma }}_{12}^2$ is the determinant. Appendix A shows how to calculate mean values and the elements of covariance matrix.

Entropy of the joint PDF $p(x,t_1;v,t_2)$ and marginal PDFs $p(x,t)$ and $p(v,t)$ can easily be shown to be

$$\begin{array}{ccc}\hfill S(t_1,t_2)& =& -\int dxdvp(x,t_1;v,t_2)lnp(x,t_1;v,t_2)=\frac{1}{2}\left[1+ln\left({\left(2\pi \right)}^2\right|\mathsf{\Sigma }(t_1,t_2)|)\right],\hfill \end{array}$$

(13)

$$\begin{array}{ccc}\hfill S_x\left(t\right)& =& -\int dxp(x,t)lnp(x,t)=\frac{1}{2}\left[1+ln(2\pi {\mathsf{\Sigma }}_{xx}\left(t\right))\right],\hfill \end{array}$$

(14)

$$\begin{array}{ccc}\hfill S_v\left(t\right)& =& -\int dvp(v,t)lnp(v,t)=\frac{1}{2}\left[1+ln(2\pi {\mathsf{\Sigma }}_{vv}\left(t\right))\right].\hfill \end{array}$$

(15)

On the other hand, the information rates for the equal-time joint PDF and the marginal PDFs are given by

$$\begin{array}{ccc}\hfill \mathcal{E}& =& {\mathsf{\Gamma }}^2=\int dxdvp(x,v,t){({\partial }_{t}lnp(x,v,t))}^2={\partial }_t\langle X_i\rangle {\mathsf{\Sigma }}_{ij}^{-1}{\partial }_t\langle X_j\rangle +\frac{1}{2}\mathrm{Tr}\left[{\left({\mathsf{\Sigma }}^{-1}\dot{\mathsf{\Sigma }}\right)}^2\right],\hfill \end{array}$$

(16)

$$\begin{array}{ccc}\hfill {\mathcal{E}}_x& =& {\mathsf{\Gamma }}_x^2=\int dxp(x,t){({\partial }_{t}lnp(x,t))}^2=\frac{1}{{\mathsf{\Sigma }}_{xx}}{\left(\frac{d\langle x\rangle }{dt}\right)}^2+\frac{1}{2{\mathsf{\Sigma }}_{xx}^2}{\left(\frac{d{\mathsf{\Sigma }}_{xx}}{dt}\right)}^2,\hfill \end{array}$$

(17)

$$\begin{array}{ccc}\hfill {\mathcal{E}}_v& =& {\mathsf{\Gamma }}_v^2=\int dvp(v,t){({\partial }_{t}lnp(v,t))}^2=\frac{1}{{\mathsf{\Sigma }}_{vv}}{\left(\frac{d\langle v\rangle }{dt}\right)}^2+\frac{1}{2{\mathsf{\Sigma }}_{vv}^2}{\left(\frac{d{\mathsf{\Sigma }}_{vv}}{dt}\right)}^2.\hfill \end{array}$$

(18)

It is useful to note that the first term on the RHS of Equations (17) and (18) is caused by the temporal change in the mean values of x and v, respectively, while the second term is due to that in the variance. Equation (16) for the joint PDF contains the contribution from the temporal changes in the mean values of x and v and in the covariance matrix. The derivation of Equations (16)–(18) is provided in Appendix B (also see Reference [31]).

To clarify the key idea behind the causality information rate, we provide detailed mathematical steps involved in the definition and calculation of ${\mathsf{\Gamma }}_{x\to v}$ and ${\mathsf{\Gamma }}_{v\to x}$ in Section 3.1 and Section 3.2, respectively.

3.1. ${\mathsf{\Gamma }}_{v\to x}$

We start with the Kramers process Equations (7)–(9), where $X_2=v$ is frozen for time $(t,t_1)$;

$$\begin{array}{ccc}\hfill \frac{dx}{dt}& =& v,\hfill \end{array}$$

(19)

$$\begin{array}{ccc}\hfill \frac{dv}{dt}& =& 0.\hfill \end{array}$$

(20)

Then, the bivariate Gaussian PDF in Equation (10) for a fixed v takes the following form:

$$\begin{array}{ccc}& & p(x,t_1;v,t)=\frac{1}{\left(2\pi \right)\sqrt{\left|\mathsf{\Sigma }\right(t_1,t\left)\right|}}exp(-\frac{1}{2}{\mathsf{\Sigma }}_{ij}^{-1}(t_1,t)(X_i-\langle X_i\rangle )(X_j-\langle X_j\rangle )),\hfill \end{array}$$

(21)

$$\begin{array}{ccc}& & \mathsf{\Sigma }(t_1,t)=\left[\begin{array}{cc}{\mathsf{\Sigma }}_{xx}\left(t_1\right)& {\mathsf{\Sigma }}_{xv}(t_1,t)\\ {\mathsf{\Sigma }}_{xv}(t_1,t)& {\mathsf{\Sigma }}_{vv}\left(t\right)\end{array}\right]=\left[\begin{array}{cc}\langle {\left(\delta x\left(t_1\right)\right)}^2\rangle & \langle \delta x\left(t_1\right)\delta v\left(t\right)\rangle \\ \langle \delta x\left(t_1\right)\delta v\left(t\right)\rangle & \langle {\left(\delta v\left(t\right)\right)}^2\rangle \end{array}\right],\hfill \end{array}$$

(22)

where $X_1=x\left(t_1\right)$ and $X_2\left(t_2\right)$, $\langle X_1\rangle =\langle x\left(t_1\right)\rangle$, $\langle X_2\rangle =\langle v\left(t\right)\rangle$, $\delta x\left(t_1\right)=x\left(t_1\right)-\langle x\left(t_1\right)\rangle$, and $\delta v\left(t\right)=v\left(t\right)-\langle v\left(t\right)\rangle$.

For $i=2$ and $j=1$ in Equations (3) and (5), we have

$$\begin{array}{ccc}\hfill {\mathsf{\Gamma }}_{v\to x}\left(t\right)& =& {\mathsf{\Gamma }}_x^{*}-{\mathsf{\Gamma }}_x,\hfill \end{array}$$

(23)

$$\begin{array}{ccc}\hfill {\mathcal{E}}_x^{*}\left(t\right)& =& {\mathsf{\Gamma }}_x^{*}{\left(t\right)}^2=\underset{t_1\to t^{+}}{lim}\int dxdvp(x,t_1;v,t){({\partial }_{t_1}lnp(x,t_1;v,t))}^2,\hfill \end{array}$$

(24)

where ${\mathsf{\Gamma }}_x=\sqrt{{\mathcal{E}}_x}$. In Appendix B, we show that ${\mathcal{E}}_x^{*}$ is given by

$$\begin{array}{ccc}\hfill {\mathcal{E}}_x^{*}& =& \underset{t_1\to t^{+}}{lim}\left[{\partial }_{t_1}\langle X_i\rangle {\mathsf{\Sigma }}_{ij}^{-1}(t_1,t){\partial }_{t_1}\langle X_j\rangle +\frac{1}{2}\mathrm{Tr}\left[{\left({\mathsf{\Sigma }}^{-1}(t_1,t){\partial }_{t_1}\mathsf{\Sigma }(t_1,t)\right)}^2\right]\right].\hfill \end{array}$$

(25)

Since v is frozen during time $(t,t_1)$, ${lim}_{t_1\to t}{\partial }_{t_1}\langle X_i\rangle ={\delta }_{i1}{\partial }_t\langle x\left(t\right)\rangle$; ${\mathsf{\Sigma }}_{vv}$ remains constant, while ${\mathsf{\Sigma }}_{xx}$ and ${\mathsf{\Sigma }}_{xv}$ change, as follows:

$$\begin{array}{ccc}\hfill \underset{t_1\to t}{lim}{\partial }_{t_1}{\mathsf{\Sigma }}_{vv}\left(t\right)& =& 0,\hfill \end{array}$$

(26)

$$\begin{array}{ccc}\hfill \underset{t_1\to t}{lim}{\partial }_{t_1}{\mathsf{\Sigma }}_{xv}(t_1,t)& =& \underset{t_1\to t}{lim}\langle {\partial }_{t_1}\left(\delta x\left(t_1\right)\right)\delta v\left(t\right)\rangle ={\mathsf{\Sigma }}_{vv}\left(t\right)\equiv {\dot{\mathsf{\Sigma }}}_{xv},\hfill \end{array}$$

(27)

$$\begin{array}{ccc}\hfill \underset{t_1\to t}{lim}{\partial }_{t_1}{\mathsf{\Sigma }}_{xx}\left(t_1\right)& =& \underset{t_1\to t}{lim}\langle {\partial }_{t_1}\left(\delta x\left(t_1\right)\right)\delta x\left(t_1\right)\rangle =2{\mathsf{\Sigma }}_{vx}\left(t\right)\equiv {\dot{\mathsf{\Sigma }}}_{xx}.\hfill \end{array}$$

(28)

Then, to calculate the two terms on RHS of Equation (25), we note

$$\begin{array}{ccc}\hfill {\mathsf{\Sigma }}^{-1}(t_1,t)=\frac{1}{\left|\mathsf{\Sigma }\right|}\left[\begin{array}{cc}{\mathsf{\Sigma }}_{vv}\left(t\right)& -{\mathsf{\Sigma }}_{xv}(t_1,t)\\ -{\mathsf{\Sigma }}_{xv}(t_1,t)& {\mathsf{\Sigma }}_{xx}\left(t\right)\end{array}\right],& & \underset{t_1\to t}{lim}{\partial }_{t_1}\mathsf{\Sigma }(t_1,t)=\left[\begin{array}{cc}{\dot{\mathsf{\Sigma }}}_{xx}& {\dot{\mathsf{\Sigma }}}_{xv}\\ {\dot{\mathsf{\Sigma }}}_{xv}& 0\end{array}\right],\hfill \end{array}$$

(29)

where ${\dot{\mathsf{\Sigma }}}_{xx}$ and ${\dot{\mathsf{\Sigma }}}_{xv}$ are given in Equations (27) and (28), while ${\mathsf{\Sigma }}^{-1}={\mathsf{\Sigma }}^{-1}\left(t\right)$ is the inverse of the equal time covariance matrix. Using Equation (29), we can show that the second term on RHS of Equation (25) becomes

$$\begin{array}{ccc}& & \underset{t_1\to t^{+}}{lim}\left[\frac{1}{2}\mathrm{Tr}\left[{\left({\mathsf{\Sigma }}^{-1}(t_1,t){\partial }_{t_1}\mathsf{\Sigma }(t_1,t)\right)}^2\right]\right]\hfill \\ & & =\frac{1}{{2\left|\mathsf{\Sigma }\right|}^2}\left[{({\mathsf{\Sigma }}_{vv}{\dot{\mathsf{\Sigma }}}_{xx}-{\mathsf{\Sigma }}_{xv}{\dot{\mathsf{\Sigma }}}_{xv})}^2+2\left({\mathsf{\Sigma }}_{vv}{\dot{\mathsf{\Sigma }}}_{xv}\right)(-{\mathsf{\Sigma }}_{xv}{\dot{\mathsf{\Sigma }}}_{xx}+{\mathsf{\Sigma }}_{xx}{\dot{\mathsf{\Sigma }}}_{xv})+{\left({\mathsf{\Sigma }}_{xv}{\dot{\mathsf{\Sigma }}}_{xv}\right)}^2\right]\hfill \\ & & =\frac{1}{{2\left|\mathsf{\Sigma }\right|}^2}\left[2\left|\mathsf{\Sigma }\right|{\dot{\mathsf{\Sigma }}}_{xv}^2+{\left({\Delta }_{vx}\right)}^2\right].\hfill \end{array}$$

(30)

Here,

$$\begin{array}{ccc}\hfill {\Delta }_{vx}& \equiv & {\mathsf{\Sigma }}_{vv}{\dot{\mathsf{\Sigma }}}_{xx}-2{\dot{\mathsf{\Sigma }}}_{xv}{\mathsf{\Sigma }}_{xv}=2{\mathsf{\Sigma }}_{vv}{\mathsf{\Sigma }}_{xv}-2{\mathsf{\Sigma }}_{vv}{\mathsf{\Sigma }}_{xv}=0,\hfill \end{array}$$

(31)

where ${\dot{\mathsf{\Sigma }}}_{xx}=2{\mathsf{\Sigma }}_{xv}$ and ${\dot{\mathsf{\Sigma }}}_{xv}={\mathsf{\Sigma }}_{vv}$ in Equations (27) and (28) are used. It is useful to note that ${\Delta }_{vx}$ represents the rate at which the determinant of the covariance matrix changes in time for a fixed v and becomes zero. This is because, for a fixed v (essentially for $\gamma =D=0$ as seen below in regard to Equation (52)), the evolution is conservative (reversible) where the phase space volume is conserved. Thus, the contribution from the variance to the information rate of x for a given v is solely determined by the temporal change in the cross-correlation ${\mathsf{\Sigma }}_{xv}$.

Finally, by using Equations (30) and (31) in Equation (25), we have

$$\begin{array}{ccc}\hfill {\mathcal{E}}_x^{*}& =& {\mathsf{\Sigma }}_{xx}^{-1}{\langle v\left(t\right)\rangle }^2+\frac{{\dot{\mathsf{\Sigma }}}_{xv}^2}{\left|\mathsf{\Sigma }\right|}.\hfill \end{array}$$

(32)

It is interesting to compare the first term (caused by the mean motion $\frac{d\langle x\rangle }{dt}$) on the RHS of Equation (32) with that in Equation (17). For instance, if ${\mathsf{\Sigma }}_{xv}=0$, ${\mathsf{\Sigma }}_{xx}^{-1}=\frac{{\mathsf{\Sigma }}_{vv}}{\left|\mathsf{\Sigma }\right|}=\frac{1}{{\mathsf{\Sigma }}_{xx}}$, they take the same value. It should also be noted that, even when both ${\mathsf{\Sigma }}_{xv}=0$ and $\frac{d}{dt}{\mathsf{\Sigma }}_{xv}={\mathsf{\Sigma }}_{vv}-{\omega }^2{\mathsf{\Sigma }}_{xx}=0$ (as in equilibrium), ${\dot{\mathsf{\Sigma }}}_{xv}={\mathsf{\Sigma }}_{vv}\ne 0$ (unless ${\mathsf{\Sigma }}_{vv}=0$).

Putting Equations (17) and (32) and ${lim}_{t_1\to t}{\partial }_{t_1}\langle X_i\rangle ={\delta }_{i1}{\partial }_t\langle x\left(t\right)\rangle$ in Equation (23) gives us

$$\begin{array}{ccc}\hfill {\mathsf{\Gamma }}_{v\to x}& =& \sqrt{{\mathcal{E}}_x^{*}}-\sqrt{{\mathcal{E}}_x}\hfill \\ & =& {\left[\frac{{\mathsf{\Sigma }}_{vv}{\langle v\left(t\right)\rangle }^2}{\left|\mathsf{\Sigma }\right|}+\frac{{\mathsf{\Sigma }}_{vv}^2}{\left|\mathsf{\Sigma }\right|}\right]}^{\frac{1}{2}}-{\left[\frac{{\langle v\left(t\right)\rangle }^2}{{\mathsf{\Sigma }}_{xx}}+\frac{2{\mathsf{\Sigma }}_{xv}^2}{{\mathsf{\Sigma }}_{xx}^2}\right]}^{\frac{1}{2}},\hfill \end{array}$$

(33)

where we used ${\dot{\mathsf{\Sigma }}}_{xv}={\mathsf{\Sigma }}_{vv}$ and ${\dot{\mathsf{\Sigma }}}_{xx}=2{\mathsf{\Sigma }}_{xv}$. (See Appendix A for the values for means and covariance matrix.) Therefore, even when ${\mathsf{\Sigma }}_{xv}=0$, Equation (33) can have a non-trivial contribution from a non-zero mean velocity.

To understand the difference between ${\mathcal{E}}_x^{*}$ and ${\mathcal{E}}_x$, it is useful to define the following quantify

$$\begin{array}{ccc}\hfill {\mathcal{E}}_{v\to x}& =& {\mathcal{E}}_x^{*}-{\mathcal{E}}_x=\frac{{\mathsf{\Sigma }}_{xv}^2}{\left|\mathsf{\Sigma }\right|{\mathsf{\Sigma }}_{xx}}{\langle v\left(t\right)\rangle }^2+\frac{{\left|\mathsf{\Sigma }\right|}^2+{\mathsf{\Sigma }}_{xv}^4}{\left|\mathsf{\Sigma }\right|{\mathsf{\Sigma }}_{xx}^2}.\hfill \end{array}$$

(34)

The cross-correlation ${\mathsf{\Sigma }}_{xv}$ plays a more important role in Equation (34) than in Equation (33). For instance, ${\mathsf{\Sigma }}_{xv}=0$ reduces Equation (34) into a simple form ${\mathcal{E}}_{v\to x}=\frac{{\mathsf{\Sigma }}_{vv}}{{\mathsf{\Sigma }}_{xx}}$, with no contribution from the mean velocity v. As noted above, such simplification does not occur for ${\mathsf{\Gamma }}_{v\to x}$ in Equation (33).

Nevertheless, if $\frac{d\langle x\rangle }{dt}=\frac{d\langle v\rangle }{dt}=0$ and ${\mathsf{\Sigma }}_{xv}=0$, Equations (33) and (34) become

$${\mathcal{E}}_{v\to x}=\frac{{\mathsf{\Sigma }}_{vv}}{{\mathsf{\Sigma }}_{xx}},{\mathsf{\Gamma }}_{v\to x}=\sqrt{\frac{{\mathsf{\Sigma }}_{vv}}{{\mathsf{\Sigma }}_{xx}}}.$$

(35)

For instance, in equilibrium, ${\mathsf{\Sigma }}_{xv}=0$, ${\mathsf{\Sigma }}_{xx}=\frac{D}{\gamma {\omega }^2}$, ${\mathsf{\Sigma }}_{vv}=\frac{D}{\gamma }$; thus, ${\mathsf{\Gamma }}_{v\to x}=\omega$. In Section 3.2 below, we show that the equality ${\mathsf{\Gamma }}_{v\to x}={\mathsf{\Gamma }}_{x\to v}=\omega$ holds in equilibrium (see the discussion below Equation (57)).

3.2. ${\mathsf{\Gamma }}_{x\to v}$

We now consider the Kramers equation Equations (7)–(9), where $X_1=x$ is frozen for time $(t,t_1)$;

$$\begin{array}{ccc}\hfill \frac{dx}{dt}& =& 0,\hfill \end{array}$$

(36)

$$\begin{array}{ccc}\hfill \frac{dv}{dt}& =& -\gamma v-{\omega }^{2}x+\xi .\hfill \end{array}$$

(37)

Then, during $(t,t_1)$, the bivariate Gaussian PDF in Equation (10) for a fixed x takes the following form:

$$\begin{array}{ccc}& & p(x,t;v,t_1)=\frac{1}{\left(2\pi \right)\sqrt{\left|\mathsf{\Sigma }\right(t,t_1\left)\right|}}exp(-\frac{1}{2}{\mathsf{\Sigma }}_{ij}^{-1}(t,t_1)(X_i-\langle X_i\rangle )(X_j-\langle X_j\rangle )),\hfill \end{array}$$

(38)

$$\begin{array}{ccc}& & \mathsf{\Sigma }(t,t_1)=\left[\begin{array}{cc}{\mathsf{\Sigma }}_{xx}\left(t\right)& {\mathsf{\Sigma }}_{xv}(t,t_1)\\ {\mathsf{\Sigma }}_{xv}(t,t_1)& {\mathsf{\Sigma }}_{vv}\left(t_1\right)\end{array}\right]=\left[\begin{array}{cc}\langle {\left(\delta x\left(t\right)\right)}^2\rangle & \langle \delta x\left(t\right)\delta v\left(t_1\right)\rangle \\ \langle \delta x\left(t\right)\delta v\left(t_1\right)\rangle & \langle {\left(\delta v\left(t_1\right)\right)}^2\rangle \end{array}\right],\hfill \end{array}$$

(39)

where $\langle X_1\rangle =\langle x\left(t\right)\rangle$, $\langle X_2\rangle =\langle v\left(t_1\right)\rangle$, $\delta x\left(t\right)=x\left(t\right)-\langle x\left(t\right)\rangle$, and $\delta v\left(t_1\right)=v\left(t_1\right)-\langle v\left(t_1\right)\rangle$.

We define ${\mathcal{E}}_v^{*}$, ${\mathsf{\Gamma }}_v^{*}=\sqrt{{\mathcal{E}}_v^{*}}$, ${\mathsf{\Gamma }}_v=\sqrt{{\mathcal{E}}_v}$, and ${\mathcal{E}}_{x\to v}$ as

$$\begin{array}{ccc}\hfill {\mathsf{\Gamma }}_{x\to v}& =& {\mathsf{\Gamma }}_v^{*}-{\mathsf{\Gamma }}_v,\hfill \end{array}$$

(40)

$$\begin{array}{ccc}\hfill {\mathcal{E}}_{x\to v}& =& {\mathcal{E}}_v^{*}-{\mathcal{E}}_v,\hfill \end{array}$$

(41)

$$\begin{array}{ccc}\hfill {\mathcal{E}}_v^{*}\left(t\right)& =& {\mathsf{\Gamma }}_v^{*}{\left(t\right)}^2=\underset{t_1\to t^{+}}{lim}\int dxdvp(x,t;v,t_1){({\partial }_{t_1}lnp(x,t;v,t_1))}^2\hfill \end{array}$$

(42)

$$\begin{array}{ccc}& =& \underset{t_1\to t^{+}}{lim}\left[{\partial }_{t_1}\langle X_i\rangle {\mathsf{\Sigma }}_{ij}^{-1}(t,t_1){\partial }_{t_1}\langle X_j\rangle +\frac{1}{2}\mathrm{Tr}\left[{\left({\mathsf{\Sigma }}^{-1}(t,t_1){\partial }_{t_1}\mathsf{\Sigma }(t,t_1)\right)}^2\right]\right],\hfill \end{array}$$

(43)

where ${\mathcal{E}}_v$ is given in Equation (18). Note that Equation (43) simply follows by replacing $\mathsf{\Sigma }(t_1,t)$ by $\mathsf{\Sigma }(t,t_1)$ in Equation (25).

Since we are considering the evolution of joint PDF of v for a given x for an infinitesimal time interval $(t,t_1)$ through Equations (36) and (37), ${\mathsf{\Sigma }}_{xx}$ remains constant, while ${\mathsf{\Sigma }}_{xv}$ and ${\mathsf{\Sigma }}_{vv}$ evolve in time as follows:

$$\begin{array}{ccc}\hfill \underset{t_1\to t}{lim}{\partial }_{t_1}{\mathsf{\Sigma }}_{xx}\left(t\right)& =& \underset{t_1\to t}{lim}\langle {\partial }_{t_1}\left(\delta x\left(t\right)\right)\delta x\left(t\right)\rangle =0,\hfill \end{array}$$

(44)

$$\begin{array}{ccc}\hfill \underset{t_1\to t}{lim}{\partial }_{t_1}{\mathsf{\Sigma }}_{xv}(t,t_1)& =& \underset{t_1\to t}{lim}\langle \delta x\left(t\right){\partial }_{t_1}\left(\delta v\left(t_1\right)\right)\rangle \equiv {{\mathsf{\Sigma }}^{\prime }}_{xv},\hfill \end{array}$$

(45)

$$\begin{array}{ccc}\hfill \underset{t_1\to t}{lim}{\partial }_{t_1}{\mathsf{\Sigma }}_{vv}\left(t_1\right)& =& 2\underset{t_1\to t}{lim}\langle \left({\partial }_{t_1}\delta v\left(t_1\right)\right)\delta v\left(t_1\right)\rangle \equiv {{\mathsf{\Sigma }}^{\prime }}_{vv}.\hfill \end{array}$$

(46)

Here, by using Equations (7)–(9), we can show (see Appendix C for comments):

$$\begin{array}{ccc}\hfill {{\mathsf{\Sigma }}^{\prime }}_{vv}\left(t\right)& =& 2\langle ({\partial }_t\left(\delta v\left(t\right)\right)\delta v\left(t\right)\rangle =2\langle (-\gamma \delta v\left(t\right)-{\omega }^2\delta x\left(t\right)+\xi )\delta v\left(t\right)\rangle \hfill \\ & =& 2(-\gamma {\mathsf{\Sigma }}_{vv}-{\omega }^2{\mathsf{\Sigma }}_{xv}+D),\hfill \end{array}$$

(47)

$$\begin{array}{ccc}\hfill {{\mathsf{\Sigma }}^{\prime }}_{xv}\left(t\right)& =& \langle ({\partial }_t\left(\delta v\left(t\right)\right)\delta x\left(t\right)\rangle =\langle (-\gamma \delta v\left(t\right)-{\omega }^2\delta x\left(t\right)+\xi )\delta x\left(t\right)\rangle =-\gamma {\mathsf{\Sigma }}_{xv}-{\omega }^2{\mathsf{\Sigma }}_{xx}.\hfill \end{array}$$

(48)

We now need to calculate the two terms on RHS of Equation (43). First, since $X_1=x\left(t\right)$ is frozen,

$$\begin{array}{ccc}\hfill \underset{t_1\to t^{+}}{lim}{\partial }_{t_1}\langle X_i\rangle {\mathsf{\Sigma }}_{ij}^{-1}(t,t_1){\partial }_{t_1}\langle X_j\rangle & =& {\mathsf{\Sigma }}_{vv}^{-1}\left(t\right){\left({\partial }_t\langle v\left(t\right)\rangle \right)}^2,\hfill \end{array}$$

(49)

$$\begin{array}{ccc}\hfill \underset{t_1\to t^{+}}{lim}{\partial }_{t_1}\mathsf{\Sigma }(t,t_1)& =& \left[\begin{array}{cc}0& {{\mathsf{\Sigma }}^{\prime }}_{xv}\\ {{\mathsf{\Sigma }}^{\prime }}_{xv}& {{\mathsf{\Sigma }}^{\prime }}_{vv}\end{array}\right].\hfill \end{array}$$

(50)

Secondly, using Equations (50) and (39), we can show that the second term on RHS of Equation (43) becomes

$$\begin{array}{ccc}& & \underset{t_1\to t^{+}}{lim}\left[\frac{1}{2}\mathrm{Tr}\left[{\left({\mathsf{\Sigma }}^{-1}(t,t_1){\partial }_{t_1}\mathsf{\Sigma }(t,t_1)\right)}^2\right]\right]\hfill \\ & & =\frac{1}{{2\left|\mathsf{\Sigma }\right|}^2}\left[{({\mathsf{\Sigma }}_{xx}{{\mathsf{\Sigma }}^{\prime }}_{vv}-{\mathsf{\Sigma }}_{xv}{{\mathsf{\Sigma }}^{\prime }}_{xv})}^2+2\left({\mathsf{\Sigma }}_{xx}{{\mathsf{\Sigma }}^{\prime }}_{xv}\right)(-{\mathsf{\Sigma }}_{xv}{{\mathsf{\Sigma }}^{\prime }}_{vv}+{\mathsf{\Sigma }}_{vv}{{\mathsf{\Sigma }}^{\prime }}_{xv})+{\left({\mathsf{\Sigma }}_{xv}{{\mathsf{\Sigma }}^{\prime }}_{xv}\right)}^2\right]\hfill \\ & & =\frac{1}{{2\left|\mathsf{\Sigma }\right|}^2}\left[2\left|\mathsf{\Sigma }\right|{{\mathsf{\Sigma }}^{\prime }}_{xv}^2+{\left({\Delta }_{vx}\right)}^2\right].\hfill \end{array}$$

(51)

Here,

$$\begin{array}{ccc}\hfill {\Delta }_{xv}& \equiv & {\mathsf{\Sigma }}_{xx}{{\mathsf{\Sigma }}^{\prime }}_{vv}-2{{\mathsf{\Sigma }}^{\prime }}_{xv}{\mathsf{\Sigma }}_{xv}=-2\gamma \left|\mathsf{\Sigma }\right|+2D{\mathsf{\Sigma }}_{xx},\hfill \end{array}$$

(52)

where we used Equations (47) and (48). It is useful to note that ${\Delta }_{xv}$ in Equation (52), representing the rate at which the determinant of the covariance changes in time for a fixed x, contains the two terms involving $\gamma$ (damping) and D (stochasticity) due to irreversibility. We also note that, in equilibrium, ${\mathsf{\Sigma }}_{vv}=\frac{D}{\gamma }$ and ${\mathsf{\Sigma }}_{xv}=0$; thus, ${\Delta }_{xv}=0$; $\frac{d}{dt}{\mathsf{\Sigma }}_{xv}=0$, but ${{\mathsf{\Sigma }}^{\prime }}_{vx}=-\gamma {\mathsf{\Sigma }}_{xv}-{\omega }^2{\mathsf{\Sigma }}_{xx}=-{\omega }^2{\mathsf{\Sigma }}_{xx}\ne 0$ in Equation (51), in general, contributing to ${\mathcal{E}}_v^{*}$ in Equation (43).

We use Equations (49), (51) and (52) in Equation (43) to obtain

$$\begin{array}{ccc}\hfill {\mathcal{E}}_v^{*}& =& {\mathsf{\Sigma }}_{vv}^{-1}{\langle {\partial }_{t}v\left(t\right)\rangle }^2+\frac{1}{{2\left|\mathsf{\Sigma }\right|}^2}\left(2\left|\mathsf{\Sigma }\right|{{\mathsf{\Sigma }}^{\prime }}_{xv}^2+{\left({\Delta }_{vx}\right)}^2\right),\hfill \end{array}$$

(53)

where ${\mathsf{\Sigma }}_{vv}^{-1}=\frac{{\mathsf{\Sigma }}_{xx}}{\left|\mathsf{\Sigma }\right|}$. Using Equation (53) and (18) in Equation (40) gives us

$$\begin{array}{ccc}\hfill {\mathsf{\Gamma }}_{x\to v}& =& \sqrt{{\mathcal{E}}_x^{*}}-\sqrt{{\mathcal{E}}_x}\hfill \\ & =& {\left[\frac{{\langle {\partial }_{t}v\left(t\right)\rangle }^2}{{\mathsf{\Sigma }}_{vv}}+\frac{1}{2{\mathsf{\Sigma }}_{vv}^2}{\left(\frac{d{\mathsf{\Sigma }}_{vv}}{dt}\right)}^2\right]}^{\frac{1}{2}}-{\left[\frac{{\mathsf{\Sigma }}_{xx}{\langle {\partial }_{t}v\left(t\right)\rangle }^2}{\left|\mathsf{\Sigma }\right|}+\frac{2\left|\mathsf{\Sigma }\right|{{\mathsf{\Sigma }}^{\prime }}_{xv}^2+{\left({\Delta }_{vx}\right)}^2}{{2\left|\mathsf{\Sigma }\right|}^2}\right]}^{\frac{1}{2}},\hfill \end{array}$$

(54)

where $\frac{d{\mathsf{\Sigma }}_{vv}}{dt}=2(-\gamma {\mathsf{\Sigma }}_{vv}-{\omega }^2{\mathsf{\Sigma }}_{xv}+D)$, ${{\mathsf{\Sigma }}^{\prime }}_{xv}=-\gamma {\mathsf{\Sigma }}_{xv}-{\omega }^2{\mathsf{\Sigma }}_{xx}$ and ${\Delta }_{vx}=-2\gamma \left|\mathsf{\Sigma }\right|+2D{\mathsf{\Sigma }}_{xx}$.

Again, to understand the difference between ${\mathcal{E}}_x$ and ${\mathcal{E}}_x^{*}$, we perform straightforward but lengthy calculations using Equations (18), (41), (52), and (53) and find the following:

$$\begin{array}{ccc}\hfill {\mathcal{E}}_{x\to v}& =& {\mathsf{\Sigma }}_{vv}^{-1}{\langle {\partial }_{t}v\left(t\right)\rangle }^2+\frac{1}{{2\left|\mathsf{\Sigma }\right|}^2}\left[2\left|\mathsf{\Sigma }\right|{{\mathsf{\Sigma }}^{\prime }}_{xv}^2+{\left({\Delta }_{vx}\right)}^2\right]-\left[\frac{{\langle {\partial }_{t}v\left(t\right)\rangle }^2}{{\mathsf{\Sigma }}_{vv}}+\frac{{{\mathsf{\Sigma }}^{\prime }}_{vv}^2}{2{\mathsf{\Sigma }}_{vv}^2}\right]\hfill \\ & \equiv & \frac{{\mathsf{\Sigma }}_{xv}^2}{\left|\mathsf{\Sigma }\right|{\mathsf{\Sigma }}_{vv}}{\langle {\partial }_{t}v\left(t\right)\rangle }^2+Q\left(t\right).\hfill \end{array}$$

(55)

Here, Q is defined by

$$\begin{array}{ccc}\hfill Q& =& \frac{1}{{2\left|\mathsf{\Sigma }\right|}^2}\left[2\left|\mathsf{\Sigma }\right|{{\mathsf{\Sigma }}^{\prime }}_{xv}^2+{\left({\Delta }_{vx}\right)}^2\right]-\frac{{{\mathsf{\Sigma }}^{\prime }}_{vv}^2}{2{\mathsf{\Sigma }}_{vv}^2}\hfill \\ & =& \frac{1}{\left|\mathsf{\Sigma }\right|{\mathsf{\Sigma }}_{vv}^2}\left[(2D{\mathsf{\Sigma }}_{xv}-\gamma {\mathsf{\Sigma }}_{xv}{\mathsf{\Sigma }}_{vv}+\left|\mathsf{\Sigma }\right|{\omega }^2){}^2+2\gamma {\omega }^2{\mathsf{\Sigma }}_{xv}^3{\mathsf{\Sigma }}_{vv}+{\mathsf{\Sigma }}_{xv}^4{\omega }^4+2D^2\frac{{\mathsf{\Sigma }}_{xv}^4}{\left|\mathsf{\Sigma }\right|}\right].\hfill \end{array}$$

(56)

We again note that ${\mathsf{\Sigma }}_{xv}$ plays a key role in Equation (55), as in the case of Equation (34). In particular, if ${\mathsf{\Sigma }}_{xv}=0$, $Q=\frac{\left|\mathsf{\Sigma }\right|{\omega }^4}{{\mathsf{\Sigma }}_{vv}^2}$. If both $\frac{d\langle x\rangle }{dt}=\frac{d\langle v\rangle }{dt}=0$ and ${\mathsf{\Sigma }}_{xv}=0$, Equations (40) and (55) become

$${\mathsf{\Gamma }}_{x\to v}={\omega }^2\sqrt{\frac{{\mathsf{\Sigma }}_{xx}}{{\mathsf{\Sigma }}_{vv}}},{\mathcal{E}}_{x\to v}={\omega }^4\frac{{\mathsf{\Sigma }}_{xx}}{{\mathsf{\Sigma }}_{vv}}.$$

(57)

In equilibrium where ${\mathsf{\Sigma }}_{xx}=\frac{D}{\gamma {\omega }^2}$ and ${\mathsf{\Sigma }}_{vv}=\frac{D}{\gamma }$, ${\mathsf{\Gamma }}_{x\to v}=\omega$ and ${\mathcal{E}}_{x\to v}={\omega }^2$. Thus, we have the equality ${\mathsf{\Gamma }}_{v\to x}={\mathsf{\Gamma }}_{x\to v}=\omega$ and ${\mathcal{E}}_{x\to v}={\mathcal{E}}_{v\to x}={\omega }^2$, as alluded to in the discussion following Equation (35).

4. Entropy-Based Causality Measures

As noted previously, most of information theoretical measures of causality are based on entropy, joint entropy, conditional entropy, or mutual information, etc. Specifically, for the two dependent stochastic variables $X_1$ and $X_2$ with the marginal PDFs $p(X_1,t)$ and $p(X_2,t)$ and joint PDF $p(X_1,t_1;X_2,t_2)$, entropy $S\left(X_1\right)$, joint entropy $S(X_1,X_2)$, mutual entropy $S\left(X_1\right|X_2)$, and mutual information $I(X_1,X_2)$ are defined by

$$\begin{array}{ccc}\hfill S\left(X_1\left(t_1\right)\right)& =& -\int d{X}_{1}p(X_1,t_1)lnp(X_1,t_1),\hfill \end{array}$$

(58)

$$\begin{array}{ccc}\hfill S\left(X_2\left(t_2\right)\right)& =& -\int d{X}_{2}p(X_2,t_2)lnp(X_2,t_2),\hfill \end{array}$$

(59)

$$\begin{array}{ccc}\hfill S(X_1\left(t_1\right),X_2\left(t_2\right))& =& -\int d{X}_{1}d{X}_{2}p(X_1,t_1;X_2,t_2)lnp(X_1,t_1;X_2,t_2),\hfill \end{array}$$

(60)

$$\begin{array}{ccc}\hfill S\left(X_1\left(t_1\right)\right|X_2\left(t_2\right))& =& S(X_1\left(t_1\right),X_2\left(t_2\right))-S\left(X_2\left(t_2\right)\right),\hfill \end{array}$$

(61)

$$\begin{array}{ccc}\hfill I(X_1\left(t_1\right):X_2\left(t_2\right))& =& S\left(X_1\left(t_1\right)\right)-S(X_1\left(t_1\right),X_2\left(t_2\right))\hfill \\ & =& S\left(X_1\left(t_1\right)\right)+S\left(X_2\left(t_2\right)\right)-S(X_1\left(t_1\right),X_2\left(t_2\right)).\hfill \end{array}$$

(62)

For Gaussian processes, Equations (13)–(15) show that the entropy depends only on the variance/covariance, being independent of the mean value. This can be problematic as entropy fails to capture the effect of one variable on the mean value of another variable, for instance, caused by rare events associated with coherent structures, such as vortices, shear flows, etc. This is explicitly shown in Section 5.4 (see Figure 5) in regard to causality. Although not widely recognized, it is important to point out the limitation of entropy-based measures in measuring perturbations (in particular, caused by abrupt events) that do not affect entropy, as shown in Reference [32]. In addition, entropy has shortcomings, such as being non-invariant under coordinate transformations and insensitive to the local arrangement (shape) of $p(x,t)$ for fixed t. Similar comments are applicable to other entropy-based measures. To demonstrate this point, in this section, we provide a detailed analysis of information flow based on conditional entropy [16,41].

4.1. $T_{v\to x}$

Information flow is based on predicting gain (or loss) of the future of subsystem 1 from the present state of subsystems 2 and defined as

$$\begin{array}{ccc}\hfill T_{v\to x}& =& \underset{t_1\to t^{+}}{lim}{\partial }_{t_1}I(x\left(t_1\right):v\left(t\right))\hfill \\ & =& \frac{dS\left(x\right(t\left)\right)}{dt}-\underset{t_1\to t^{+}}{lim}{\partial }_{t_1}S\left(x\left(t_1\right)\right|v\left(t\right))=\frac{dS\left(x\right(t\left)\right)}{dt}-\underset{t_1\to t^{+}}{lim}{\partial }_{t_1}S(x\left(t_1\right),v\left(t\right)),\hfill \end{array}$$

(63)

where Equation (62) is used. Here, the first term and the second term on the RHS represent the rate of change of the marginal entropy of $x\left(t\right)$ and the rate of change of the conditional entropy of $x\left(t_1\right)$ conditional on $v\left(t\right)$ (i.e., frozen v). The difference between these two rates then quantifies the effect of the evolution of v on the entropy of x. Note that $T_{v\to x}$ can be both negative and positive; a negative $T_{v\to x}$ means that v acts to reduce the marginal entropy of x ($S_1$), as numerically observed in Reference [32].

Using Equation (13), we have ${\partial }_{t_1}S(x\left(t_1\right),v\left(t\right))=\frac{{\partial }_{t_1}\left|\mathsf{\Sigma }(t_1,t)\right|}{\left|\mathsf{\Sigma }\right(t_1,t\left)\right|}$. Then, by using Equations (26)–(28) and (29), we obtain

$$\begin{array}{ccc}\hfill \underset{t_1\to t}{lim}{\partial }_{t_1}\left|\mathsf{\Sigma }(t_1,t)\right|& =& {\dot{\mathsf{\Sigma }}}_{xx}{\mathsf{\Sigma }}_{vv}-2{\dot{\mathsf{\Sigma }}}_{xv}{\mathsf{\Sigma }}_{xv}=2{\mathsf{\Sigma }}_{xv}{\mathsf{\Sigma }}_{vv}-2{\mathsf{\Sigma }}_{vv}{\mathsf{\Sigma }}_{xv}=0,\hfill \end{array}$$

(64)

and

$$\begin{array}{ccc}\hfill T_{v\to x}& =& \frac{dS\left(x\right(t\left)\right)}{dt}=\frac{1}{2}\frac{{\dot{\mathsf{\Sigma }}}_{xx}}{{\mathsf{\Sigma }}_{xx}}=\frac{{\mathsf{\Sigma }}_{xv}}{{\mathsf{\Sigma }}_{xx}}.\hfill \end{array}$$

(65)

As can be seen from Equation (65), $T_{v\to x}$ depends only on the variance, being independent of the mean value. Furthermore, $T_{v\to x}$ is proportional to the cross-correlation ${\mathsf{\Sigma }}_{xv}$, becoming zero for ${\mathsf{\Sigma }}_{xv}=0$ as in the case of equilibrium. (Note that Equation (65) is derived using a different method in Reference [32] for the Kramers equation.)

4.2. $T_{x\to v}$

Similarly, information flow is based on predicting gain (or loss) of the future of subsystem 2 from the present state of subsystems 1 and defined as

$$\begin{array}{ccc}\hfill T_{x\to v}& =& \underset{t_1\to t^{+}}{lim}{\partial }_{t_1}I(x\left(t\right):v\left(t_1\right))\hfill \\ & =& \frac{dS\left(v\right(t\left)\right)}{dt}-\underset{t_1\to t^{+}}{lim}{\partial }_{t_1}S\left(x\left(t\right)\right|v\left(t_1\right))=\frac{dS\left(v\right(t\left)\right)}{dt}-\underset{t_1\to t^{+}}{lim}{\partial }_{t_1}S(x\left(t\right),v\left(t_1\right)),\hfill \end{array}$$

(66)

where Equation (62) is used. Here, the first term and the second term on the RHS represent the rate of change of the marginal entropy of $v\left(t\right)$ and the rate of change of the conditional entropy of $v\left(t_1\right)$ conditional on $x\left(t\right)$ (i.e., frozen x). The difference between these two rates then quantifies the effect of the evolution of x on the entropy of v. Note again that $T_{x\to v}$ can be both negative and positive; a negative $T_{x\to v}$ means that x acts to reduce the marginal entropy of v ($S_2$), as numerically observed in Reference [32].

For ${\partial }_{t_1}S(x\left(t\right),v\left(t_1\right)]=\frac{{\partial }_{t_1}\left|\mathsf{\Sigma }(t,t_1)\right|}{\left|\mathsf{\Sigma }\right(t,t_1\left)\right|}$, we use Equations (44)–(48) and (50) to obtain

$$\begin{array}{ccc}\hfill \underset{t_1\to t}{lim}{\partial }_{t_1}\left|\mathsf{\Sigma }(t,t_1)\right|& =& {{\mathsf{\Sigma }}^{\prime }}_{vv}\left(t\right){\mathsf{\Sigma }}_{xx}\left(t\right)-2{{\mathsf{\Sigma }}^{\prime }}_{xv}\left(t\right){\mathsf{\Sigma }}_{xv}\hfill \\ & =& \gamma (-2{\mathsf{\Sigma }}_{xx}{\mathsf{\Sigma }}_{vv}+2{\mathsf{\Sigma }}_{xv}^2)+2D{\mathsf{\Sigma }}_{xx}=-2\gamma \left|\mathsf{\Sigma }\right|+2D{\mathsf{\Sigma }}_{xx}.\hfill \end{array}$$

(67)

Thus,

$$\begin{array}{ccc}\hfill T_{x\to v}& =& \frac{dS\left(v\right(t\left)\right)}{dt}-\underset{t_1\to t^{+}}{lim}{\partial }_{t_1}S(X_1\left(t\right),X_2\left(t_1\right))\hfill \\ & =& \frac{1}{2}\left[\frac{{{\mathsf{\Sigma }}^{\prime }}_{vv}}{{\mathsf{\Sigma }}_{vv}}-\frac{-2\gamma \left|\mathsf{\Sigma }\right|+2D{\mathsf{\Sigma }}_{xx}}{\left|\mathsf{\Sigma }\right|}\right]=-{\omega }^2\frac{{\mathsf{\Sigma }}_{xv}}{{\mathsf{\Sigma }}_{vv}}-D\frac{{\left({\mathsf{\Sigma }}_{xv}\right)}^2}{{\mathsf{\Sigma }}_{vv}\left|\mathsf{\Sigma }\right|}.\hfill \end{array}$$

(68)

Again, Equation (68) is derived using a different method in Reference [32]. As in the case of $T_{v\to x}$ in Equation (65), $T_{x\to v}$ depends only on the variance, being independent of the mean value while being proportional to the cross-correlation ${\mathsf{\Sigma }}_{xv}$, becoming zero for ${\mathsf{\Sigma }}_{xv}=0$ as in the case of equilibrium.

5. Comparisons between Causal Information Rate and Information Flow

In this section, we compare the causal information rate in Equations (33) and (54) with the information flow in Equations (65) and (68) for the Kramers equation by focusing on several interesting cases. We start by noting that if $\gamma \ne 0$, $\omega \ne 0$, and $D\ne 0$, x and v evolve to equilibrium where the covariance matrix takes the values:

$${\mathsf{\Sigma }}_{xx}=\frac{D}{\gamma {\omega }^2},{\mathsf{\Sigma }}_{vv}=\frac{D}{\gamma },{\mathsf{\Sigma }}_{xv}=0.$$

(69)

We use the same initial conditions

$$\langle x\left(0\right)\rangle =-0.5,\langle v\left(0\right)\rangle =0.5,{\mathsf{\Sigma }}_{xx}\left(0\right)={\mathsf{\Sigma }}_{vv}\left(0\right)=0.01,{\mathsf{\Sigma }}_{xv}\left(0\right)=0,$$

(70)

and present various statistical quantifies in Figure 1, Figure 2, Figure 3, Figure 4 and Figure 5, including snapshots of PDFs, ${\mathsf{\Sigma }}_{xx}\left(t\right)$, ${\mathsf{\Sigma }}_{vv}\left(t\right)$, and ${\mathsf{\Sigma }}_{xv}\left(t\right)$ in panel (a); $T_{x\to v}$, $T_{v\to x}$, ${\mathsf{\Gamma }}_{x\to v}$, ${\mathsf{\Gamma }}_{v\to x}$ in panel (b). Note that PDF snapshots are shown for one-standard deviation, using different colors for different times.

5.1. No Stochastic Noise $D=0$

It is useful to look at the deterministic case without the stochastic noise $\xi =0$ in Equations (7) and (8), where a time-dependent PDF evolves due to non-zero initial conditions. Specifically, the two cases where $\gamma =\omega =0$ and $\gamma =0$ and $\omega =1$ are considered in Figure 1 and Figure 2, respectively.

To gain a key insight into the meaning of our causal information rate, we start with the simplest case in Figure 1, where $\gamma =0=\omega$, with v being fixed to its initial value $v\left(0\right)$ so that $x\left(t\right)=x\left(0\right)+v\left(0\right)t$. The snapshots of the PDF and the covariance matrix in Figure 1a show that the PDF center (peak) undergoes a drift according to $\langle x\left(t\right)\rangle =\langle x\left(0\right)\rangle +\langle v\left(0\right)\rangle t$, while the PDF broadens with time in the x-direction since $\delta x\left(t\right)=\delta x\left(0\right)+t\delta v\left(0\right)$. As a result, ${\mathsf{\Sigma }}_{xv}$ increases linearly with time. ${\mathsf{\Sigma }}_{xv}\ne 0$ causes a rapid initial increase in information flow $T_{v\to x}\ne 0$ in Figure 1b. However, as $t\to \infty$, $T_{v\to x}\to 0$ since ${\mathsf{\Sigma }}_{xx}$ increases faster than ${\mathsf{\Sigma }}_{xv}$ in time, leading to $T_{v\to x}=\frac{{\mathsf{\Sigma }}_{xv}}{{\mathsf{\Sigma }}_{xx}}\to 0$ (see Equation (65)). Thus, $T_{v\to x}$ fails to reflect the feedback from v to x in the long time limit. In contrast, ${\mathsf{\Gamma }}_{v\to x}$ monotonically increases with time, approaching a constant value (${\mathsf{\Gamma }}_{v\to x}\to 5$) as $t\to \infty$. On the other hand, ${\mathsf{\Gamma }}_{x\to v}=T_{x\to v}=0$ at all times, reflecting the lack of the coupling from x to v at all times, consistent with our expectation. That is, the lack of the feedback from x on v is reflected in both ${\mathsf{\Gamma }}_{x\to v}=T_{x\to v}=0$, while the one-way coupling of v to x is captured only by ${\mathsf{\Gamma }}_{v\to x}\ne 0$ at any time.

To include the feedback of x on v, we now consider the case $\gamma =0$ and $\omega =1$ in Figure 2. Non-zero value of $\omega$ ($=1$)—only the difference from Figure 1—now establishes the two-way (mutual) communications between x and v, leading to the harmonic motion (see Equations (7) and (8)). Figure 2a shows how the PDF center drifts according to this harmonic motion, while the cross-correlation ${\mathsf{\Sigma }}_{xv}\left(t\right)=0$ at any time. The latter leads to the information flow $T_{x\to v}=T_{v\to x}=0$ shown in Figure 2b. In contrast, ${\mathsf{\Gamma }}_{x\to v}$ and ${\mathsf{\Gamma }}_{v\to x}$ in Figure 2b exhibit oscillations with a 90 degree phase-shift between the two due to the harmonic motion, capturing the two-way feedback between x and v. To highlight an exact symmetry between ${\mathsf{\Gamma }}_{x\to v}$ and ${\mathsf{\Gamma }}_{v\to x}$, we can consider a global, path-dependent measure of causality by integrating ${\mathsf{\Gamma }}_{x\to v}$ and ${\mathsf{\Gamma }}_{v\to x}$ over the same integer multiples of the period ($2\pi /\omega$), which would give the same value. These results reveal that our causal information rate captures the dependence between x and v even in the absence of their cross-correlation. (We recall that zero cross-correlation does not imply independence.)

5.2. Equilibrium: v

We now consider the Ornstein-Uhlenbeck (O-U) process of v by choosing $\gamma =1$, $\omega =0$ and $D\ne 0$ in Figure 3. In this case, v approaches asymptotically its equilibrium distribution where ${\mathsf{\Sigma }}_{vv}=\frac{D}{\gamma }$, while $p(x,t)$ evolves in time. Specifically, we choose $D=D_{xx}\left(0\right)=D_{vv}\left(0\right)=0.01$ (see also Equation (70)). Figure 3a shows that the PDF broadens in the x-direction (${\mathsf{\Sigma }}_{xx}\propto t$), while keeping its original width in the v-direction. On the other hand, due to the non-zero $D\ne 0$, the cross-correlation ${\mathsf{\Sigma }}_{xv}\left(t\right)>0$ is seen to grow in time, approaching a constant value 0.01 as $t\to \infty$. As in the case of Figure 2, $T_{x\to v}$ and $T_{v\to x}$ take finite values due to non-zero ${\mathsf{\Sigma }}_{xv}\ne 0$ but become zero asymptotically as $t\to \infty$ (due to ${\mathsf{\Sigma }}_{xx}\propto t$), as shown in Figure 3b. On the other hand, the behavior of ${\mathsf{\Gamma }}_{x\to v}$ and ${\mathsf{\Gamma }}_{v\to x}$ is quite similar.

5.3. Equilibrium: x and v

To ensure a quick evolution to the equilibrium distribution in time, we choose $D=0.01$, $\omega =1$, and $\gamma =1$ so that the initial and final (equilibrium) PDFs have the same variance in Figure 4. Figure 4a shows that the PDF center undergoes a damped oscillation without changing its shape since ${\mathsf{\Sigma }}_{xx}\left(t\right)={\mathsf{\Sigma }}_{vv}\left(t\right)=0.01$ and ${\mathsf{\Sigma }}_{xv}\left(t\right)=0$ at all times. ${\mathsf{\Sigma }}_{xv}\left(t\right)=0$ leads to $T_{x\to v}=T_{v\to x}=0$ in Figure 4b, as in the case of Figure 2b. In contrast, the drift of the PDF center leads to non-zero values of ${\mathsf{\Gamma }}_{v\to x}$ and ${\mathsf{\Gamma }}_{x\to v}$, which asymptotically approach 1 (see Equations (57)). The latter reveals that the equilibrium is maintained through the two-way communications between x and v.

5.4. An Abrupt Change Introduced by an Impulse

In Reference [32], we showed that an abrupt event (modeled by an impulse function, with a peak at a certain time) caused a sudden increase in $\mathcal{E}={\mathsf{\Gamma }}^2$ in all cases, while it caused a sudden increase in the magnitude of the information flow $|T_{i\to j}|$ ($i\ne j$) only when the perturbation affects entropy (variance). Furthermore, it was shown that the peak of $|T_{i\to j}|$ ($i\ne j$) followed (not preceded) the actual impulse peak, while the peak of $\mathcal{E}={\mathsf{\Gamma }}^2$ tended to precede the impulse peak. This means that, by measuring the temporal change in $\mathcal{E}$, especially, when its peak appears, we can forecast the onset of an abrupt event (whose peak appears later than $\mathcal{E}$-peak in time). We now look at the effect of a sudden impulse on the causal information rate.

To this end, we introduce a sudden perturbation to the Kramers equation by adding an impulse function $u\left(t\right)$ as a time-dependent additive force to Equation (8) as follows:

$$\begin{array}{ccc}\hfill \frac{dv}{dt}& =& -\gamma v-{\omega }^{2}x+u\left(t\right)+\xi ,\hfill \\ \hfill u\left(t\right)& =& \frac{1}{c\sqrt{\pi }}{e}^{-{\left(\frac{t-t_0}{c}\right)}^2}.\hfill \end{array}$$

(71)

We use the analytical expressions for the mean values and covariance in Reference [32] and choose the parameter values $c=0.1$ and $t_0=4$ in Equation (71) and $D=\omega =\gamma =1$ (the same as in Figure 4). The results are shown in Figure 5, where the impulse function $u\left(t\right)$ localized around $t=4$ is shown in red dotted line, using the right y axis, in the bottom panels in Figure 5b.

Figure 5a shows that $u\left(t\right)$ causes a sudden drift of the PDF center with no change in variance. Therefore, $T_{x\to v}\left(t\right)=T_{v\to x}\left(t\right)=0$ in Figure 5b, with no influence of $u\left(t\right)$. In sharp contrast, ${\mathsf{\Gamma }}_{x\to v}$ and ${\mathsf{\Gamma }}_{v\to x}$ exhibit abrupt change around time $t=4$. Furthermore, the peak in ${\mathsf{\Gamma }}_{x\to v}$ (or ${\mathsf{\Gamma }}_{v\to x}$) tends to proceed the impulse peak (in red dotted line). We observe similar results when an impulse is applied to the covariance matrix (results not shown). These results, thus, suggest that our causality information rate is sensitive to the perturbation (in both mean and variance) and predicts the onset of a sudden event very well, especially in comparison with the information flow. We emphasize that the information flow (and other entropy-based methods) cannot detect the onset of a sudden event that does not affect entropy (e.g., Reference [32] for different examples).

6. Conclusions

Information geometry in general concerns the distinguishability between two PDFs (e.g., constructed from data) and is sensitive to the local dynamics (e.g., Reference [27]), depending on a local arrangement (the shape) of the PDFs. This is different from entropy, which is a global measure of a PDF, being insensitive to such a local arrangement. When a PDF is a continuous function of time, the information rate and information length are helpful in understanding far-from-equilibrium phenomena in terms of the number of distinguishable statistical states that a system evolves through in time. Being very sensitive to evolving dynamics, it enables us to compare different far-from-equilibrium processes using the same dimensionless distance, as well as quantifying the relation (correlation, self-regulation, etc.) among variables (e.g., References [27,28,29,30]).

In this paper, by extending our previous work [20,21,22,23,24,25,26,27,28,29,30,31], we introduced the causal information rate as a general information-geometric method that can elucidate causality relations in stochastic processes involving temporal variabilities and strong fluctuations. The key idea was to quantify the effect of one variable on the information rate of the other variable. The cross-correlation between the variables was shown to play a key role in the information flow, zero cross-correlation leading to zero information flow. In comparison, the causal information rate can take a non-zero value in the absence of cross-correlation. Since zero cross-correlation (measuring only the linear dependence) does not imply independence in general, this means that the causal information rate captures the (directional) dependence between two variables even when they are uncorrelated with each other.

Furthermore, the causal information rate captures the temporal change in both covariance matrix and mean value. In comparison, the information flow depends only on the temporal change in the covariance matrix. Thus, the causal information rate is a sensitive method for predicting an abrupt event and quantifying causal relations. These properties are welcome for predicting rare, large-amplitude events. Application has been made to the Kramers equation to highlight these points. Although the analysis in this paper is limited to the Gaussian variables that are entirely characterized by the mean and variance, similar results are likely to hold for non-Gaussian variables because the information rate captures the temporal changes of a PDF itself, while entropy-based measures (e.g., information flow) depend only on variance.

Given that causality (directional dependence) plays a crucial role in science and engineering (e.g., References [43,44]), our method could be useful in a wide range of problems. In particular, it could be utilized to elucidate causal relations among different players in nonlinear dynamical systems, fluid/plasma dynamics, laboratory plasmas, astrophysical systems, environmental science, finance, etc. For instance, in fluid/plasmas turbulence, it could help resolving the controversy over causality in the low-to-high (L-H) confinement transition [29,30,45,46], as well as contributing to identifying a causal relationship among different players responsible for the onset of sudden abrupt events (e.g., fusion plasmas eruption) (e.g., References [47,48]), with a better chance of control. It could also elucidate causal relationships among different physiological signals, how different parts of a human body (e.g., brain-heart-connection) are self-regulated to maintain homeostasis (the optimal living condition for survival), and how this homeostasis degrades with the onset of diseases.

Finally, it will be interesting to investigate the effects of coarse-graining in future works. In Reference [49], for the information geometry given by the Fisher metric, relevant directions were shown to be exactly maintained under coarse-graining, while irrelevant directions contract. The analysis for more than two variables will also be addressed in future work.