Information Geometry, Fluctuations, Non-Equilibrium Thermodynamics, and Geodesics in Complex Systems

Abstract

Information theory provides an interdisciplinary method to understand important phenomena in many research fields ranging from astrophysical and laboratory fluids/plasmas to biological systems. In particular, information geometric theory enables us to envision the evolution of non-equilibrium processes in terms of a (dimensionless) distance by quantifying how information unfolds over time as a probability density function (PDF) evolves in time. Here, we discuss some recent developments in information geometric theory focusing on time-dependent dynamic aspects of non-equilibrium processes (e.g., time-varying mean value, time-varying variance, or temperature, etc.) and their thermodynamic and physical/biological implications. We compare different distances between two given PDFs and highlight the importance of a path-dependent distance for a time-dependent PDF. We then discuss the role of the information rate $\Gamma =\frac{d\mathcal{L}}{dt}$ and relative entropy in non-equilibrium thermodynamic relations (entropy production rate, heat flux, dissipated work, non-equilibrium free energy, etc.), and various inequalities among them. Here, $\mathcal{L}$ is the information length representing the total number of statistically distinguishable states a PDF evolves through over time. We explore the implications of a geodesic solution in information geometry for self-organization and control.

1. Introduction

Information geometry refers to the application of the techniques of differential geometry to probability and statistics. Specifically, it uses differential geometry to define the metric tensor that endows the statistical space (consisting of probabilities) with the notion of distance [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31]. While seemingly too abstract, it permits us to measure quantitative differences among different probabilities. It then makes it possible to link a stochastic process, complexity, and geometry, which is particularly useful in classifying a growing number of data from different research areas (e.g., from astrophysical and laboratory systems to biosystems). Furthermore, it can be used to obtain desired outcomes [6,7,8,9,10,15] or to understand statistical complexity [4].

For instance, the Wasserstein metric [6,7,8,9,10] was widely used in the optimal transport problem where the main interest is to minimize transport cost which is a quadratic function of the distance between two locations. It satisfies the Fokker-Planck equation for gradient flow which minimizes the entropy/energy functional [7]. For Gaussian distributions, the Wasserstein metric space consists of physical distances – Euclidean and positive symmetric matrices for the mean and variance, respectively (e.g., see [8]).

In comparsion, the Fisher (Fisher-Rao) information [32] can be used to define a dimensionless distance in statistical manifolds [33,34]. For instance, the statistical distance $d{s}_{}$ represents the number of indistinguishable states as [5,33]

$${\left(ds\right)}_{}^2\equiv \sum _j\frac{d{p}_j^2}{p_j}=\sum _j{p}_j{(dln{p}_j)}^2=\sum _{j,\alpha ,\beta }{p}_j\frac{\partial ln{p}_j}{\partial {\lambda }^{\alpha }}\frac{\partial ln{p}_j}{\partial {\lambda }^{\beta }}d{\lambda }^{\alpha }d{\lambda }^{\beta }=\sum _{\alpha ,\beta }d{\lambda }^{\alpha }{g}_{\alpha \beta }d{\lambda }^{\beta }.$$

(1)

Here the Fisher information metric $g_{\alpha \beta }=\langle \frac{\partial ln{p}_j}{\partial {\lambda }^{\alpha }}\frac{\partial ln{p}_j}{\partial {\lambda }^{\beta }}\rangle ={\sum }_j{p}_j\frac{\partial ln{p}_j}{\partial {\lambda }^{\alpha }}\frac{\partial ln{p}_j}{\partial {\lambda }^{\beta }}$ provides natural (Riemannian) distinguishability metric on the space of probability distributions. ${\lambda }^{\alpha }$’s are the parameters of the probability $p_j$ and the angular brackets represent the ensemble average over $p_j$. Note that Equation (1) is given for a discrete probability $p_j$. For a continuous Probability Density Function (PDF) $p\left(x\right)$ for a variable x, Equation (1) becomes ${\left(ds\right)}^2=\int dxp\left(x\right){[dln\left(p\right(x)]}^2={\sum }_{\alpha ,\beta }d{\lambda }^{\alpha }{g}_{\alpha \beta }d{\lambda }^{\beta }$ where $g_{\alpha \beta }=\int dxp\left(x\right)\frac{\partial ln\left(p\right(x\left)\right)}{\partial {\lambda }^{\alpha }}\frac{\partial ln\left(p\right(x\left)\right)}{\partial {\lambda }^{\beta }}$.

For Gaussian processes, the Fisher metric is inversely proportional to the covariance matrices of fluctuations in the systems. Thus, in thermodynamic equilibrium, strong fluctuations lead to a strong correlation and a shorter distance between the neighboring states [34,35]. Alternatively, fluctuations determine the uncertainty in measurements, providing the resolution (the distance unit) that normalizes the distance between different thermodynamic states.

To appreciate the meaning of fluctuation-based metric, let us consider the (equilibrium) Maxwell-Boltzmann distribution $p\left(E_j\right)=\beta e^{-\beta E_j}$ for the energy state $E_j$

$$\frac{p\left(E_i\right)}{p\left(E_j\right)}=e^{-\beta (E_i-E_j)}.$$

(2)

Here $\beta =1/k_{B}T$ is the inverse temperature; $k_B$ is the Boltzmann constant; T is the temperature of the heat bath. In Equation (2), the thermal energy $k_{B}T=\langle E\rangle$ of the heat bath (the width/uncertainty of the probability) provides the resolution to differentiate different states $\Delta E=E_i-E_j$. The smaller is the resolution (temperature), the more distinguishable states (more accessible information in the system) there are. It agrees with the expectation that a PDF gradient (the Fisher-information) increases with information [32].

This concept has been generalized to non-equilibrium systems [36,37,38,39,40,41,42,43], including the utilization for controlling systems to minimize entropy production [38,40,42], the measurement of the statistical distance in experiments to validate theoretical predictions [41], etc. However, some of these works rely on the equilibrium distribution Equation (2) that is valid only in or near equilibrium while many important phenomena in nature and laboratories are often far from equilibrium with strong fluctuations, variability, heterogeneity, or stochasticity [44,45,46,47,48,49,50,51,52]. Far from equilibrium, there is no (infinite-capacity) heat bath that can maintain the system at a certain temperature, or constant fluctuation level. One of the important questions far from equilibrium is indeed to understand how fluctuation level $\beta {\left(t\right)}^{-1}$ changes with time. Furthermore, PDFs no longer follow the Maxwell-Boltzmann nor Gaussian distributions and can involve the contribution from (rare) events of large amplitude fluctuations [53,54,55,56,57,58,59,60,61,62]. Therefore, the full knowledge of time-varying PDFs and the application of information geometry to such PDFs have become of considerable interest.

Furthermore, while in equilibrium [63,64], information theoretical measures (e.g., Shannon information entropy) can be given thermodynamic meanings (e.g., heat), in non-equilibrium such interpretations are not always possible and equilibrium thermodynamic rules can break down locally (e.g., see [65,66] and references therein). Much progress on these issues has been made by different authors (e.g., [65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82]) through the development of information theory, stochastic thermodynamics, and non-equilibrium fluctuation theorems with the help of the Fisher information [32], relative entropy [83], mutual information [84,85], etc. Exemplary works include the Landauer’s principle which links information loss to the ability to extract work [86,87]; the resolution of Maxwell’s demon paradox [88]; black hole thermodynamics [89,90]; various thermodynamic inequality/uncertainty relations [65,68,91,92,93,94,95,96,97]; and linking different research areas (e.g., non-equilibrium processes to quantum mechanics [98,99,100], physics to biology [101]).

The paper aims to discuss some recent developments in the information geometric theory of non-equilibrium processes. Since this would undoubtedly span a broad range of topics, this paper will have to be selective and will focus on elucidating the dynamic aspect of non-equilibrium processes and thermodynamic and physical/biological implications. Throughout the paper, we highlight that time-varying measures (esp. variance) introduces extra complication in various relations, in particular, between the information geometric measure and entropy production rate. We make the efforts to make this paper self-contained (e.g., by including the derivations of some well-known results) wherever possible.

The remainder of this paper is organized as follows. Section 2 discusses different distances between two PDFs and the generalization for a time-dependent non-equilibrium PDF. Section 3 compares the distancs from Section 2. Section 4 discusses key thermodynamic relations that are useful for non-equilibrium processes. Section 5 establishes relations between information geometric quantities (in Section 2) and thermodynamics (in Section 4). In Section 6, we discuss the concept of a geodesic in information geometry and its implications for self-organization or designing optimal protocols for control. Conclusions are provided in Section 7.

2. Distances/Metrics

This section discusses the distance defined between two probabilities (Section 2.1) and along the evolution path of a time-dependent probability (Section 2.2). Examples and comparisons of these distances are provided in Section 3.1. For illustration, we use a PDF $p(x,t)$ of a stochastic variable x and differential entropy $S=-\int dxp(x,t)ln\left(p\right(x,t\left)\right)$ by using the unit $k_B=1$.

2.1. Distance between Two PDFs

We consider the distance between two PDFs $p_1=p(x,t_1)$ and $p_2=p(x,t_2)$ of a stochastic variable x at two times $t_1$ and $t_2$, respectively where $t_1=t_2$ or $t_1\ne t_2$ in general.

2.1.1. Wootters’ Distance

The Wootters’ distance [5,33] is defined in quantum mechanics by the shortest distance between the two $p_1$ and $p_2$ that have the wave functions ${\psi }_1$ and ${\psi }_2$ ($p_1={\left|{\psi }_1\right|}^2$ and $p_2={\left|{\psi }_2\right|}^2$), respectively. Specifically, for given $p_1$ and $p_2$, the distance $s(p_1,p_2)$ between $p_1$ and $p_2$ can be parameterized by infinitely many different paths between $p_1$ and $p_2$. Letting z be the affine parameter of a path, we have

$$s(p_1,p_2)={\int }_1^{2}ds=\int dz\frac{ds\left(z\right)}{dz}=\int dz\sqrt{\sum _{\alpha ,\beta }\frac{d{\lambda }^{\alpha }}{dz}{g}_{\alpha \beta }\frac{d{\lambda }^{\beta }}{dz}},$$

(3)

where $ds$ is given in Equation (1). Among all possible paths, the minimum of $s(p_1,p_2$) is obtained for a particular path that optimizes the quantum distinguishability; the (Hilbert-space) angle between the two wave functions provides such minimum distance as

$$\begin{array}{ccc}\hfill W[p_1,p_2]& =& {cos}^{-1}\left[\int dx{\left[p(x,t_1)\right]}^{\frac{1}{2}}{\left[p(x,t_2)\right]}^{\frac{1}{2}}\right].\hfill \end{array}$$

(4)

Equation (4) is for a pure state and has been generalized to mixed states (e.g., see [37,102] and references therein). Note that the Wootters’ distance is related to the Hellinger distance [43].

2.1.2. Kullback-Leibler (K-L) Divergence/Relative Entropy

Kullback-Leibler (K-L) divergence between the two PDFs [83], also called relative entropy, is defined by

$$\begin{array}{ccc}\hfill \mathcal{K}\left(p_1\right|p_2)& =& \int dxp(x,t_1)ln\left(\frac{p(x,t_1)}{p(x,t_2)}\right).\hfill \end{array}$$

(5)

Relative entropy quantifies the difference between a PDF $p_1$ and another PDF $p_2$. It takes the minimum zero value for identical two PDFs $p_1=p_2$ and becomes large as $p_1$ and $p_2$ become more different. However, as it is defined in Equation (5), it is not symmetric between $p_1$ and $p_2$ and does not satisfy the triangle inequality. It is thus not a metric in a strict sense.

2.1.3. Jensen Divergence

The Jensen divergence (also called Jensen distance) is the symmetrized Kullback–Leibler divergence defined by

$$\begin{array}{ccc}\hfill \mathcal{J}\left(p_1\right|p_2)& =& \frac{1}{2}\left[\mathcal{K}\left(p_1\right|p_2)+\mathcal{K}\left(p_2\right|p_1))\right].\hfill \end{array}$$

(6)

While the square root of the Jensen-Shannon divergence $\sqrt{\mathcal{J}\left(p_1\right|p_2)}$ is a metric [4,103], $\mathcal{J}\left(p_1\right|p_2)$ itself has also been used in examining statistical complexity (e.g., see [43,104,105]).

2.1.4. Euclidean Norm

In analysis of big data, the Euclidean norm [5,106] is used, which is defined by

$$|p_1-p_2{|}^2=\int dx{\left[p(x,t_1)-p(x,t_2)\right]}^2.$$

(7)

While Equation (7) has a direct analogy to the physical distance, it has a limitation in measuring statistical complexity due to the neglect of the stochastic nature [5]. For instance, the Wootters’ distance in Equation (4) was shown to work better than Equation (7) in capturing complexity in the logistic map [5].

2.2. Distance along the Path

Equations (4)–(7) can be used to define the distance between the two given PDFs $p(x,t_1)$ and $p(x,t_2)$ at times $t_1$ and $t_2$ ($t_2>t_1$). However, $p(x,t)$ at the intermediate time $t=(t_1,t_2)$ can take an infinite number of different values depending on the exact path that a system takes between $p(x,t_1)$ and $p(x,t_2)$. One example would be (i) $p(x,t_1)=p(x,t_2)=p(x,t)$ for all $t=(t_1,t_2)$ and x, in comparison with (ii) $p(x,t_1)=p(x,t_2)$ but $p(x,t)\ne p(x,t_1)$ and $p(x,t)\ne p(x,t_2)$. What is necessary is a path-dependent distance that depends on the exact evolution and the form of $p(x,t)$ for $t=(t_1,t_2)$.

2.2.1. Information Rate

Calculating a path-dependent distance for a time-dependent PDF $p(x,t)$ requires the generalization of the distance in Section 2.1. To this end, we consider two (temporally) adjacent PDFs along the trajectory, say, $p(x,t)$ and $p(x,t+dt)$ and calculate the (infinitesimal) relative entropy between them in the limit $dt\to 0$ to the leading order in $O\left(dt\right)$:

$$\begin{array}{ccc}& & \underset{dt\to 0}{lim}\frac{1}{{\left(dt\right)}^2}\mathcal{K}\left[p(x,t+dt)\right|p(x,t)]=\underset{dt\to 0}{lim}\frac{1}{{\left(dt\right)}^2}\mathcal{K}\left[p(x,t)\right|p(x,t+dt)]\hfill \\ & & =\underset{dt\to 0}{lim}\frac{1}{{\left(dt\right)}^2}\mathcal{J}\left[p(x,t+dt)\right|p(x,t)]=\frac{1}{2}\int dxp(x,t){({\partial }_{t}lnp(x,t))}^2\equiv \frac{1}{2}{\Gamma }^2.\hfill \end{array}$$

(8)

Here, we used $p(x,t+dt)=p(x,t)+\left(dt\right){\partial }_{t}p+\frac{1}{2}{\left(dt\right)}^2{\left({\partial }_{t}p\right)}^2$ + O(${\left(dt\right)}^3$), $ln(1+r)=r-\frac{1}{2}{r}^2$ + O($r^3$) for $r\ll 1$, and $\int dx{\partial }_{t}p(x,t)=\int dx{\partial }_{tt}p(x,t)=0$ because of the total probability conservation $\int dxp(x,t)=1$. Due to the symmetry of $\mathcal{K}\left[p\right(x,t+dt\left)\right|p(x,t)]$ to leading order O(${\left(dt\right)}^2$), $\mathcal{K}\left[p\right(x,t+dt\left)\right|p(x,t)]=\mathcal{J}[p(x,t+dt)\left|p\right(x,t\left)\right]$ to O(${\left(dt\right)}^2$).

In Equation (8), the information rate $\Gamma$ is defined by [15,16,17,18,19,20,21,22,23,24,25,26,27,28,29]

$${\Gamma }^2\left(t\right)=\underset{dt\to 0}{lim}\frac{2}{{\left(dt\right)}^2}J\left[p(x,t+dt)\right|p(x,t)]=\int dxp(x,t){({\partial }_{t}lnp(x,t))}^2=4\int dx{\left({\partial }_{t}q\left(t\right)\right)}^2.$$

(9)

Here, $q=\sqrt{p}$, and $\Gamma \ge 0$ by definition. We note that the last term in terms of q in Equation (9) can be used when $q=p=0$. The dimensions of ${\Gamma }^2\equiv \mathcal{E}$ and $\Gamma$ are (time)${}^{-2}$ and (time)${}^{-1}$, respectively. They do not change their values under nonlinear, time-independent transformation of variables (see Appendix A). Thus, using the unit where the length is dimensionless, $\mathcal{E}$ and $\Gamma$ can be viewed as the kinetic energy per mass and velocity, respectively. For this reason, $\Gamma$ was called the velocity (e.g., in [15,17]).

Note that $\mathcal{E}$ can be viewed as the Fisher information [32] if time is interpreted as a parameter (e.g., [97]). However, time in classical mechanics is a passive quantity that cannot be changed by an external control. $\Gamma$ is also called the entropy production rate in quantum mechanics [107]. However, as shown in Section 4.1 and Section 4.4, the relation between $\Gamma$ and thermodynamic entropy production rate is more complicated (see Equation (28)).

$\Gamma$ in Equation (9) is the information rate representing how quickly new information is revealed as a PDF evolves in time. Here, ${\Gamma }^{-1}=\tau$ is the characteristic time scale of this information change in time. To show that $\Gamma$ is related to fluctuation’s smallest time scale [97], we assume that ${\lambda }^{\alpha }$’s are the estimators (parameters) of a $p(x,t)$ and use the Cramér-Rao bound on the Fisher information $g_{\alpha \beta }=\int dxp(x,t){\partial }_{{\lambda }^{\alpha }}[lnp(x,t)]{\partial }_{{\lambda }^{\beta }}[lnp(x,t)]\ge C_{\alpha \beta }^{-1}$ where $C^{\alpha \beta }\equiv \langle \delta {\lambda }^{\alpha }\delta {\lambda }^{\beta }\rangle$ is the covariance matrix (e.g., see [32]); $\delta {\lambda }^{\alpha }={\lambda }^{\alpha }-\langle {\lambda }^{\alpha }\rangle$ denotes fluctuation. Using $\frac{dlnp}{dt}=\frac{\partial lnp}{\partial {\lambda }^{\alpha }}\frac{d{\lambda }^{\alpha }}{dt}$ then leads to

$$\begin{array}{ccc}\hfill {\Gamma }^2& =& \sum _{\alpha \beta }\frac{d{\lambda }^{\alpha }}{dt}{g}_{\alpha \beta }\frac{d{\lambda }^{\beta }}{dt}\ge \sum _{\alpha \beta }\frac{d{\lambda }^{\alpha }}{dt}{C}_{\alpha \beta }^{-1}\frac{d{\lambda }^{\beta }}{dt}.\hfill \end{array}$$

(10)

For the diagonal $g_{\alpha \beta }=g_{\alpha }{\delta }_{\alpha \beta }$, Equation (10) is simplified as

$$\begin{array}{ccc}\hfill \sum _{\alpha }\frac{1}{\langle {\left(\delta {\lambda }^{\alpha }\right)}^2\rangle }{\left(\frac{d{\lambda }^{\alpha }}{dt}\right)}^2& \le & {\Gamma }^2.\hfill \end{array}$$

(11)

Equation (11) shows how the RMS fluctuation-normalized rate at which the parameter ${\lambda }^{\alpha }$ can change is bounded above by $\Gamma$. If there is only $\alpha =1$ (${\lambda }^{\alpha }=\lambda {\delta }_{\alpha ,1}$), Equation (11) is further simplified:

$$\begin{array}{cc}\hfill \frac{1}{\sqrt{\langle {\left(\delta \lambda \right)}^2\rangle }}\left|\frac{d\lambda }{dt}\right|& \le \Gamma ,\end{array}$$

(12)

clearly showing that $\lambda$ normalized by its RMS fluctuations cannot change faster than the information rate.

Finally, it is worth highlighting that Equation (9) is general and can be used even when the parameters ${\lambda }^{\alpha }$’s and $g_{\alpha \beta }$ in ${\Gamma }^2$ in Equation (10) are unknown. Examples include the cases where PDFs are empirically inferred from experimental/observational data. Readers are referred to Refs. [21,23,28] for examples. It is only the special case where we have a complete set of parameters ${\lambda }^{\alpha }$’s of a PDF that we can express $\Gamma$ using the Fisher information as in Equation (10). For instance, for a Gaussian $p(x,t)$ that is fully described by the mean value $\langle x\rangle$ and variance $\frac{1}{2\beta }$, $({\lambda }^1,{\lambda }^2)=(\langle x\rangle ,\beta )$.

2.2.2. Information Length

Since $\Gamma \propto \sqrt{\mathcal{J}\left[p\right(x,t+dt\left)\right|p(x,t)]}$ is a metric [103] as noted in Section 2.1, $\Gamma$ is also a metric. Thus, we sum $\Gamma$ along the trajectory to define a finite distance. Specifically, starting with an initial PDF $p(x,t=0)$, we integrate $\Gamma \left(t\right)$ over time to obtain the dimensionless number as a function time as

$$\begin{array}{ccc}\hfill \mathcal{L}\left(t\right)& =& {\int }_0^{t}d{t}_1\Gamma \left(t_1\right).\hfill \end{array}$$

(13)

$\mathcal{L}$ is the information length [15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31] that quantifies the total change in information along the trajectory of $p(x,t)$ or the total number of statistically distinguishable states it evolves through over time. [We note that different names (e.g., statistical length [108], or statistical distance [97]) were also used for $\mathcal{L}$.] It is important to note that unlike the Wootters’ distance (the shortest distance among all possible paths between the two PDFs) in Equation (3) (e.g., [5]), $\mathcal{L}\left(t\right)$ in Equation (13) is fixed for a given time-evolving PDF $p(x,t)$.

By definition in Equation (13), $\mathcal{L}(t=0)=0$ and $\mathcal{L}\left(t\right)$ monotonically increases with time since $\Gamma \ge 0$ (e.g., see Figure A2 in [22]). $\mathcal{L}\left(t\right)$ takes a constant value only in a stationary state ($\Gamma ={\partial }_{t}p=0$). One of its important consequences is that when $p(x,t)$ relaxes into a stationary PDF in the long time limit $t\to \infty$, $\Gamma \left(t\right)\to 0$ and $\mathcal{L}\left(t\right)\to {\mathcal{L}}_{\infty }$ as $t\to \infty$ where ${\mathcal{L}}_{\infty }$ is a constant depending on initial conditions and parameters. This property of ${\mathcal{L}}_{\infty }$ was used to understand attractor structure in a relaxation problem; specifically, Refs. [15,16,18,22,28] calculated ${\mathcal{L}}_{\infty }$ for different values of the mean position $x_0$ of an initial PDF and examined how ${\mathcal{L}}_{\infty }$ depends on $x_0$. Furthermore, $\Gamma$ and $\mathcal{L}$ were shown to be useful for quantifying hysteresis in forward-backward processes [19], correlation and self-regulation among different players [23,25], and predicting the occurrence of sudden events [27] and phase transitions [23,25]. Some of these points are illustrated in Section 3.1.

3. Model and Comparison of Metrics

For discussing/comparing different metrics in Section 2 and statistical measures in Section 4, we use the following Langevin model [109]

$$\frac{dx}{dt}=f(x,t)+\xi =-{\partial }_{x}V(x,t)+\xi .$$

(14)

Here, $V(x,t)$ is, in general, a time-dependent potential which can include an internal potential and an external force; $\xi$ is a short (delta)-correlated Gaussian noise with the following statistical property

$$\begin{array}{ccc}& & \langle \xi \left(t\right)\xi \left(t^{\prime }\right)\rangle =2D\delta (t-t^{\prime }).\hfill \end{array}$$

(15)

Here, the angular brackets represent the ensemble average over the stochastic noise $\xi$; $D\ge 0$ is the amplitude of $\xi$. It is important to note that far from equilibrium, the average (e.g., $\langle x\left(t\right)\rangle$) is a function of time, in general.

The exact PDFs can be obtained for the Ornstein-Uhlenbeck (O-U) process which has $V=\frac{\gamma }{2}{(x-v\left(t\right))}^2$ and $f=-\gamma (x-v(t\left)\right)$ in Equation (14). Here, $v\left(t\right)$ is a deterministic function of time. Specifically, for the initial Gaussian PDF $p(x,0)$

$$p(x,0)=\sqrt{\frac{{\beta }_0}{\pi }}{e}^{-{\beta }_0{(x-x_0)}^2},$$

(16)

a time-dependent PDF remains Gaussian at all time:

$$\begin{array}{ccc}\hfill p(x,t)& =& \int d{x}_{1}p(x,t;x_1,0)p(x_1,0)=\sqrt{\frac{\beta }{\pi }}{e}^{-\beta {(x-\langle x\rangle )}^2},\hfill \end{array}$$

(17)

$$\begin{array}{ccc}\hfill \frac{1}{2\beta \left(t\right)}& =& \frac{e^{-2\gamma t}}{2{\beta }_0}+\frac{D(1-e^{-2\gamma t})}{\gamma },\hfill \end{array}$$

(18)

$$\begin{array}{ccc}\hfill \langle x\left(t\right)\rangle & =& x_0{e}^{-\gamma t}+\gamma {\int }_0^{t}d{t}_1{e}^{-\gamma (t-t_1)}v\left(t_1\right).\hfill \end{array}$$

(19)

In Equations (16)–(19), $x_0=\langle x(t=0)\rangle$, ${\beta }_0=\beta (t=0)$, and $\langle {\left(\delta x\right)}^2\rangle =\frac{1}{2\beta }={\sigma }^2$. Here, $\beta$, $\sigma$ and ${\sigma }^2$ are the inverse temperature, standard deviation, and variance, respectively; ${\beta }_0$ and $x_0$ are the values of $\beta$ and $\langle x\rangle$, respectively, at $t=0$. Equation (18) shows that as $t\to \infty$, $\beta (t\to \infty )=\frac{\gamma }{2D}$. Note that we use both $\beta$ and $\sigma$ here to clarify the connections to the previous works [15,17,22,26,27,28].

3.1. Geometric Structure of Equilibrium/Attractors

To elucidate the main difference between the distances in Equations (4)–(6) and Equation (13), we consider the relaxation problem by assuming $v\left(t\right)=0$. In the following, we compare the distance between $p(x,0)$ and $p(x,t\to \infty )$ by using $p_1(x,t_1)=p(x,0)$ and $p_2(x,t_2)=p(x,t\to \infty )$ in Equations (4)–(6) and Equation (13). Analytical expressions for these distances are given in [22].

Each curve in Figure 1 shows how each distance depends on the initial mean position $x_0$. The four different curves are for ${\mathcal{L}}_{\infty }$ (in blue), Wootters’ distance (in orange), K-L relative entropy (in green), and Jensen divergence (in red), respectively. The relative entropy and Jensen divergence exhibit similar behavior, the red and green color curves being superimposed on each other. Of note is a linear relation between ${\mathcal{L}}_{\infty }$ and $x_0$ in Figure 1. Such linear relation is not seen in other distances. This means that the information length is a unique measure that manifests a linear geometry around its equilibrium point in a linear Gaussian process [28,30]. Note that for a nonlinear force f, ${\mathcal{L}}_{\infty }$ has a power-law relation with $x_0$ for a sufficiently large $x_0$ [18,28]. These contrast with the behaviour in a chaotic system [16,28] where ${\mathcal{L}}_{\infty }$ depends sensitively on the initial condition and abruptly changes with $x_0$. Thus, the information length provides a useful tool to geometrically understand attractor structures in relaxation problems.

3.2. Correlation between Two Interacting Components

We next show that the information length is also useful in elucidating the correlation between two interacting species such as two competing components relaxing to the same equilibrium in the long time limit. Specifically, the two interacting components with the time-dependent PDFs $P_1(x,t)$ and $P_2(x,t)$ are coupled through the Dichotomous noise [110,111] (see Appendix B) and relax into the same equilibrium Gaussian PDF $P_1(x,t\to \infty )=P_2(x,t\to \infty )=\frac{1}{2}P(x,t\to \infty )$ around $x=0$ in the long time limit. Here, $P(x,t)=P_1(x,t)+P_2(x,t)$ is the total PDF. For the case considered below, $P(x,t)$ satisfies the O-U process (see Appendix B for details). We choose the initial conditions where $P_1(t=0)=P_1(t\to \infty )$ with zero initial mean value while $P_2(t=0)$ takes an initial mean value $x_0$. These are demonstrated in the cartoon figure, Figure 2a,c.

Although $P_1(t=0)=P_1(t\to \infty )$, at the intermediate time $t=(0,\infty )$, $P_1(x,t)$ evolves in time due to its coupling to $P_2$ and thus $P_1(x,t)\ne P_1(x,t=0)$, as shown in Figure 2b. Consequently, $\mathcal{L}\left(t\right)$ calculated from $P_1$ monotonically increases to its asymptotic value ${\mathcal{L}}_{\infty }$ until it reaches the equilibrium (see Figure A2 in [22] for time-evolution of $\mathcal{L}$ from $P_1$ and $P_2$). On the other hand, $P_2$ with an initial mean value $x_0$ undergoes a different time evolution (unless $x_0=0$) until it reaches the equilibrium.

The distances in Equations (4)–(7) and Equation (13) can be calculated from the total $P=P_1+P_2$, $P_1$ and $P_2$ for different values of $x_0$. Results are shown in Figure 3a–c, respectively; (a) $P(x,0)$ and $P(x,t\to \infty )$, (b) $P_1(x,0)$ and $P_1(x,t\to \infty )$, and (c) $P_2(x,0)$ and $P_2(x,t\to \infty )$, respectively. Specifically, for each value of $x_0$, we calculate the distances in Equations (4)–(7) and Equation (13) by using $p_1(x,t_1)=P(x,0)$ and $p_2(x,t_2)=P(x,t\to \infty )$ for Figure 3a; $p_1(x,t_1)=P_1(x,0)$ and $p_2(x,t_2)=P_1(x,t\to \infty )$ for Figure 3b; $p_1(x,t_1)=P_2(x,0)$ and $p_2(x,t_2)=P_2(x,t\to \infty )$ for Figure 3c. The same procedure above is then repeated for many other $x_0$’s to show how each distance depends on $x_0$.

For the total P, a linear relation between ${\mathcal{L}}_{\infty }$ and $x_0$ is seen in Figure 3a (like in Figure 1). This linear relation is not seen in ${\mathcal{L}}_{\infty }$ calculated from either $P_1$ or $P_2$ in Figure 3b or Figure 3c; a non-monotonic dependence of ${\mathcal{L}}_{\infty }$ in Figure 3b,c is due to large-fluctuations and strong-correlation between $P_1$ and $P_2$ during time-evolution for large $x_0$. What is quite remarkable is that in contrast to other distances, ${\mathcal{L}}_{\infty }$ calculated from $P_1$ and $P_2$ in Figure 3b,c exhibits a very similar dependence on $x_0$. It means that despite very different time-evolutions of $P_1$ and $P_2$ (see Figure 2), they undergo similar total change in information. These results suggest that strong coupling between two components can be inferred from their similar information length (see also [24,25]).

4. Thermodynamic Relations

To elucidate the utility of information geometric theory in understanding non-equilibrium thermodynamics, we review some of the important thermodynamic measures of irreversibility and dissipation [112] and relate them to information geometric measures $\Gamma$ and $\mathcal{K}$ [29]. For illustration below, we use the model in Equations (14) and (15) unless stated otherwise. Corresponding to Equations (14) and (15) is the following Fokker-Planck equation [109]

$$\begin{array}{ccc}\hfill \frac{\partial p(x,t)}{\partial t}& =& -\frac{\partial }{\partial x}\left[f(x,t)p(x,t)\right]+D\frac{{\partial }^{2}p(x,t)}{\partial x^2}=-{\partial }_{x}J(x,t),\hfill \end{array}$$

(20)

where $J=fp-D{\partial }_{x}p$ is the probability current.

4.1. Entropy Production Rate and Flow

For non-equilibrium thermodynamics, we need to consider the entropy in the system S and the environment $S_m$, and the total entropy $S_T=S+S_m$. To clarify the difference among these, we go over some derivation by using ${\partial }_{t}p=-{\partial }_{x}J$ and $J=fp-D{\partial }_{x}p$ to obtain

$$\dot{S}=\frac{dS(x,t)}{dt}=-\int dx{\partial }_{t}plnp=\frac{d{S}_T(x,t)}{dt}-\frac{d{S}_m(x,t)}{dt},$$

(21)

where,

$$\begin{array}{ccc}\hfill {\dot{S}}_T=\frac{d{S}_T}{dt}=\int dx\left(\frac{1}{Dp}{J}^2\right),& & {\dot{S}}_m=\frac{d{S}_m}{dt}=\int dx\left(\frac{1}{D}Jf\right).\hfill \end{array}$$

(22)

Here, we used integration by parts in t and x. ${\dot{S}}_T$ denotes the (total) entropy production rate, which is non-negative ${\dot{S}}_T\ge 0$ by definition, and serves as a measure of irreversibility [112]. The sign of ${\dot{S}}_m$ in Equation (22) represents the direction in which the entropy flows between the system and environment. Specifically, ${\dot{S}}_m>0$ (${\dot{S}}_m<0$) when the entropy flows from the system (environment) to the environment (system). ${\dot{S}}_m$ is related to the heat flux $Q=D{S}_m$ from the system to the environment. The equality ${\dot{S}}_T=0$ holds in an equilibrium reversible process. In this case, $\dot{S}=-{\dot{S}}_m=-\frac{Q}{D}$, which is the usual equilibrium thermodynamic relation. In comparison, when $\dot{S}=0$, ${\dot{S}}_T={\dot{S}}_m\ge 0$.

For the O-U process with $V=\frac{\gamma }{2}{(x-v\left(t\right))}^2$ and $f=-\gamma (x-v(t\left)\right)$ in Equation (14), Equations (17)–(19) and Equations (21)–(22) lead to (see [29] for details)

$$\begin{array}{ccc}\hfill S\left(t\right)& =& \frac{1}{2}\left[1+ln\frac{\pi }{\beta }\right],\hfill \end{array}$$

(23)

$$\begin{array}{ccc}\hfill D\dot{S_T}& =& \frac{{\left({\partial }_t\beta \right)}^2}{8{\beta }^3}+{\left({\partial }_t\langle x\rangle \right)}^2={\left({\partial }_t\sigma \right)}^2+{\left({\partial }_t\langle x\rangle \right)}^2,\hfill \end{array}$$

(24)

$$\begin{array}{ccc}\hfill \dot{S}& =& -\frac{{\partial }_t\beta }{2\beta }=\frac{{\partial }_t\sigma }{\sigma },\hfill \end{array}$$

(25)

$$\begin{array}{ccc}\hfill {\dot{S}}_m& =& {\dot{S}}_T-\dot{S}.\hfill \end{array}$$

(26)

Here, we used $J=\left[f+2D\beta \left(\delta x\right)\right]p=\left[-\frac{{\partial }_t\beta }{2\beta }\left(\delta x\right)+{\partial }_t\langle x\rangle \right]p$, ${\partial }_{x}p=-2\beta \left(\delta x\right)p$, $f=-\gamma (x-v(t\left)\right)$, ${\partial }_t\langle x\rangle =\langle f\rangle$, $2D\beta -\gamma =-\frac{{\partial }_t\beta }{2\beta }$, and $\frac{{\partial }_t\beta }{\beta }=-2\frac{{\partial }_t\sigma }{\sigma }$.

In order to relate these thermodynamical quantities ${\dot{S}}_T$ and $\dot{S}$ above to the information rate $\Gamma$, we recall that for the O-U process [15,17,26,27,28],

$$\begin{array}{ccc}\hfill \mathcal{E}& =& {\Gamma }^2=2\beta {\left({\partial }_t\langle x\rangle \right)}^2+\frac{{\left({\partial }_t\beta \right)}^2}{2{\beta }^2}=\frac{1}{{\sigma }^2}\left[2{\left({\partial }_t\sigma \right)}^2+{\left({\partial }_t\langle x\rangle \right)}^2\right].\hfill \end{array}$$

(27)

Equations (24) and (27) then give us

$$\begin{array}{ccc}\hfill {\Gamma }^2& =& \frac{D}{{\sigma }^2}{\dot{S}}_T+{\dot{S}}^2.\hfill \end{array}$$

(28)

Interestingly, Equation (28) reveals that the entropy production rate needs be normalized by variance ${\sigma }^2$. This is because of the extensive nature of ${\dot{S}}_T$ unlike $\Gamma$ or $\dot{S}$. That is, ${\dot{S}}_T$ changes its value when the variable x is rescaled by a scalar factor, say, $\alpha$ ($>0$) as $y=\alpha x$. Furthermore, Equation (28) shows that the information rate $\Gamma$ in general does not have a simple relation to the entropy production rate (c.f., [107]).

One interesting limit of Equation (28) is the case of constant $\beta \left(t\right)$ with $\dot{S}=0$. In that case, Equation (24) becomes $D{\dot{S}}_T={\left({\partial }_t\langle x\rangle \right)}^2$ while Equations (13), (27) and (28) give us

$$\begin{array}{ccc}\hfill \mathcal{L}\left(t\right)& =& \frac{\langle x\left(t\right)\rangle -\langle x(t=0)\rangle }{\sigma },\hfill \end{array}$$

(29)

$$\begin{array}{ccc}\hfill {\Gamma }^2& =& \frac{1}{{\sigma }^2}{\left({\partial }_t\langle x\rangle \right)}^2=\frac{D}{{\sigma }^2}{\dot{S}}_T.\hfill \end{array}$$

(30)

Equation (29) simply states that $\mathcal{L}$ measures the total change in the mean value normalized by fluctuation level $\sigma$. Equation (30) manifests a linear relation between ${\Gamma }^2$ and ${\dot{S}}_T$ when ${\partial }_t\sigma =0$, as invoked in the previous works (e.g., [107]). Furthermore, a linear relation between ${\Gamma }^2$ and ${\dot{S}}_T$ in Equation (30) implies that minimizing the entropy production ${\int }^{t}d{t}_1{\dot{S}}_T$ along the trajectory corresponds to minimizing ${\int }_0^{t}d{t}_1{\Gamma }^2\left(t_1\right)$, which, in turn, is equivalent to minimizing $\mathcal{L}\left(t\right)$ (see Section 5 for further discussions).

Finally, to demonstrate how entropy production rate and thermal bath temperature (D) are linked to the speed of fluctuations $c=\sigma \Gamma$ [97], we rewrite Equation (28) as

$$c=\sigma \Gamma ={\left[D{\dot{S}}_T+{\sigma }^2{\dot{S}}^2\right]}^{\frac{1}{2}}.$$

(31)

For constant variance $\dot{\beta }=0$, Equation (31) gives a simple relation $c=\sigma \Gamma =\sqrt{D{\dot{S}}_T}$.

4.2. Non-Equilibrium Thermodynamical Laws

To relate the statistical measures in Section 4.1 to thermodynamics, we let U (internal energy) be the average potential energy $U=\langle V\rangle$ and obtain (see also [66,113] and references therein)

$$\begin{array}{ccc}\hfill \frac{dU}{dt}& =& \frac{d}{dt}\langle V\rangle \equiv \dot{W}-\dot{Q},\hfill \end{array}$$

(32)

where

$$\begin{array}{ccc}\hfill \dot{W}& =& \int dx\left({\partial }_{t}V\right)p=\langle {\partial }_{t}V\rangle ,\hfill \end{array}$$

(33)

$$\begin{array}{ccc}\hfill \dot{Q}& =& -\int dxV\left({\partial }_{t}p\right)=\int dxJf=\langle f\dot{x}\rangle =D{\dot{S}}_m.\hfill \end{array}$$

(34)

The power $\dot{W}$ represents the average rate at which the work is done to the system because of time-varying potential; the average work during the time interval $[t_0,t]$ is calculated by $W={\int }_{t_0}^{t}d{t}^{\prime }\dot{W}\left(t^{\prime }\right)$. On the other hand, $\dot{Q}$ represents the rate of dissipated heat.

Equation (32) establishes the non-equilibrium thermodynamic relation $\dot{U}=\dot{W}-\dot{Q}$. Physically, it simply means that the work done to the system $\dot{W}$ increases U while the dissipated heat to the environment $\dot{Q}$ decreases it. Equations (21), (32), and (34) permit us to define a non-equilibrium (information) free energy $\mathcal{F}\left(t\right)=U\left(t\right)-DS\left(t\right)$ [92] and its time-derivative

$$\begin{array}{ccc}\hfill \dot{\mathcal{F}}=\dot{U}-D\dot{S}& =& \dot{W}-D{\dot{S}}_T,\hfill \end{array}$$

(35)

where $\dot{\mathcal{F}}=\frac{d\mathcal{F}}{dt}$ and $\dot{U}=\frac{dU}{dt}$. Since ${\dot{S}}_T\ge 0$, Equation (35) leads to the following inequality

$$\begin{array}{ccc}\hfill D{\dot{S}}_T& =& \dot{W}-\dot{\mathcal{F}}\equiv {\dot{W}}_D\ge 0,\hfill \end{array}$$

(36)

where the non-negative dissipated power (lost to the environment) ${\dot{W}}_D$ is defined. Finally, the time-integral version of Equation (36) provides the bound on the average work performed on the system as $W-\Delta \mathcal{F}=W_D\ge 0$ (e.g., [68]).

4.3. Relative Entropy as a Measure of Irreversibility

The relative entropy has proven to be useful in understanding irreversibilities and non-equilibrium thermodynamic inequality relations [91,92,93,94,114,115,116]. In particular, the dissipated work $W_D=W-\Delta \mathcal{F}$ (in Equation (36)) is related to the relative entropy between the PDFs in the forward and reverse processes

$$W_D=D\mathcal{K}\left[p_F\left({\gamma }_F\left(t\right)\right)\right|p_R\left({\gamma }_R\left(t\right)\right)].$$

(37)

(e.g., see [91,92,93,94].) Here, $p_F\left({\gamma }_F\left(t\right)\right)$ and $p_R\left({\gamma }_R\left(t\right)\right)$ are the PDFs for the forward and reverse processes driven by the forward ${\gamma }_F\left(t\right)$ and reverse ${\gamma }_R\left(t\right)$ protocols, respectively. Using Equation (36) in Equation (37) immediately gives

$${\dot{S}}_T=\frac{d}{dt}\mathcal{K}\left[p_F\left({\gamma }_F\left(t\right)\right)\right|p_R\left({\gamma }_R\left(t\right)\right)]\ge 0,$$

(38)

which is a proxy for irreversibility (see [115,116] for a slightly different expression of Equation (38)). It is useful to note that forward and reversal protocols are also used to establish various fluctuations theorems for different dissipative measures such as entropy production, dissipated work, etc. (see, e.g., [80] for a nice review and references therein).

However, we cannot consider forward and reversal protocols in the absence of a model control parameter that can be prescribed as a function of time. Even in this case, the relative entropy is useful in quantifying irreversibility through inequalities, and this is what we focus on in the remainder of Section 4.3.

To this end, let us consider a non-equilibrium state $p(x,t)$ which has an instantaneous non-equilibrium stationary state $p_s(x,t)$ and calculate the relative entropy between the two. Here, $p_s(x,t)$ is a steady solution of the Fokker-Planck equation ${\partial }_t{p}_s=0$ in Equation (20) (e.g., see [29]). Specifically, one finds $p_s(x,t)=e^{-\frac{V(x,t)-{\mathcal{F}}_s\left(t\right)}{D}}$ by treating the parameters to be constant (being frozen to their instantaneous values at a given time). Here, V and ${\mathcal{F}}_s$ are the potential energy and the stationary free energy, respectively. For clarity, an example of $p_s(x,t)$ is given in Section 4.4.

The average of $ln{p}_s\left(t\right)$ in the non-equilibrium state $p(x,t)$ can be expressed as follows:

$$\begin{array}{c}\hfill \int dxp(x,t)ln{p}_s(x,t)=-\frac{1}{D}\int dxp(x,t)(V(x,t)-{\mathcal{F}}_s\left(t\right))=-\frac{1}{D}(U\left(t\right)-{\mathcal{F}}_s\left(t\right)).\end{array}$$

(39)

Equations (35) and (39) then give us

$$\begin{array}{c}\hfill \mathcal{F}\left(t\right)-{\mathcal{F}}_s\left(t\right)=D\int dxp(x,t)ln\left[\frac{p(x,t)}{p_s(x,t)}\right]\equiv D\mathcal{K}\left[p(x,t)\right|p_s(x,t)]\ge 0.\end{array}$$

(40)

Here, we used the fact the relative entropy is non-negative. Equation (40) explicitly shows that non-equilibrium free energy is bounded below by the stationary one $\mathcal{F}\ge {\mathcal{F}}_s$ (see also [1,92] and references therein for open Hamiltonian systems).

Equation (40) together with Equation (35) then lead to the following irreversible work $W_{irr}$ [29,92]:

$$\begin{array}{c}\hfill W_{irr}\equiv W-\Delta {\mathcal{F}}_s=D\Delta S_T+\Delta (\mathcal{F}-{\mathcal{F}}_s)=D\Delta S_T+D\Delta \mathcal{K}\left[p(x,t)\right|p_s(x,t)].\end{array}$$

(41)

Here, $\Delta \mathcal{K}\left[p(x,t)\right|p_s(x,t)]=\mathcal{K}\left[p(x,t)\right|p_s(x,t)]-\mathcal{K}\left[p(x,t_0)\right|p_s(x,t_0)]$, etc. The derivation of Equation (41) for open-driven Hamiltonian systems is provided in [92] (see their Equation (38)).

On the other hand, we directly calculate the time-derivative of $\mathcal{K}\left[p(x,t)\right|p_s(x,t)]$ in Equation (40) by using $p_s(x,t)=e^{-\frac{V(x,t)-{\mathcal{F}}_s\left(t\right)}{D}}$, ${\dot{S}}_T=\dot{S}+{\dot{S}}_m$, $\int dxp{\partial }_{t}V=\dot{W}$ and $\dot{Q}=-\int dx{\partial }_{t}pV=D{\dot{S}}_m$, and $W_{irr}=W-\Delta {\mathcal{F}}_s$:

$$\begin{array}{ccc}& & \frac{d}{dt}\mathcal{K}\left[p(x,t)\right|p_s(x,t)]=-\dot{S}+\frac{1}{D}\frac{d}{dt}\left[\int dxVp-{\mathcal{F}}_s\right]=-{\dot{S}}_T+\frac{1}{D}\left[\dot{W}-\frac{d}{dt}{\mathcal{F}}_s\right].\hfill \end{array}$$

(42)

One can see easily that equating Equation (42) to $\frac{1}{D}[\dot{\mathcal{F}}-\dot{{\mathcal{F}}_s}]$ (from Equation (40)) simply recovers $\dot{W}-\frac{d}{dt}\mathcal{F}=D{\dot{S}}_T$ in Equation (35).

Finally, we obtain a differential form of Equation (41) by using ${\dot{W}}_{irr}=\dot{W}-\frac{d}{dt}{\mathcal{F}}_s$ in Equation (42) as follows

$${\dot{W}}_{irr}=D{\dot{S}}_T+D\frac{d}{dt}\mathcal{K}\left[p(x,t)\right|p_s(x,t)].$$

(43)

4.4. Example

We consider $v\left(t\right)=ut$ with a constant u so that $V=-\frac{\gamma }{2}{(x-ut)}^2$ in Equation (14). While the discussion below explicitly involves $v\left(t\right)$, the results are general and valid for the limiting case $v\left(t\right)=0$. The case with $v\left(t\right)=0$ is an example where the forward and reversal protocols do not exist while a non-equilibrium stationary state does.

For $f=-\gamma (x-ut)$, Equation (19) is simplified as follows

$$\langle x\left(t\right)\rangle =x_0{e}^{-\gamma t}+ut-\frac{u}{\gamma }\left(1-e^{-\gamma t}\right).$$

(44)

For the non-equilibrium stationary state with fixed $\gamma$ and D, ${\beta }_s=\frac{\gamma }{2D}$ is also constant ($\frac{d}{dt}{\mathcal{F}}_s=0$). Therefore, we have

$$p_s(x,t)=\sqrt{\frac{{\beta }_s}{\pi }}{e}^{-{\beta }_s{(x-ut)}^2}.$$

(45)

Then, we can find (see [29] for details)

$$\begin{array}{ccc}\hfill \mathcal{K}\left[p(x,t)\right|p_s(x,t)]& =& \frac{1}{2}\left[-1+ln(\beta /{\beta }_s)\right]+{\beta }_s\left({(\langle x\rangle -ut)}^2+\frac{1}{2\beta }\right),\hfill \end{array}$$

(46)

$$\begin{array}{ccc}\hfill D{\dot{S}}_T& =& \left[{\left(-\gamma \langle x_0\rangle e^{-\gamma t}+u\left(1-e^{-\gamma t}\right)\right)}^2+\frac{1}{2\beta }{(2\beta D-\gamma )}^2\right],\hfill \end{array}$$

(47)

$$\begin{array}{ccc}\hfill \dot{Q}& =& {\left[-\gamma \langle x_0\rangle e^{-\gamma t}+u\left(1-e^{-\gamma t}\right)\right]}^2-\frac{\gamma }{2\beta }\left(2\beta D-\gamma \right),\hfill \end{array}$$

(48)

$$\begin{array}{ccc}\hfill \dot{W}& =& -u\int dx\gamma (x-ut)p=-u\gamma \langle x-ut\rangle =u^2\left(1-e^{-\gamma t}\right),\hfill \end{array}$$

(49)

$$\begin{array}{ccc}\hfill \frac{d}{dt}\mathcal{K}\left[p(x,t)\right|p_s(x,t)]& =& -\frac{1}{D}\left[{\left({\partial }_t\langle x\rangle \right)}^2+{\left({\partial }_t\sigma \right)}^2-u{\partial }_t\langle x\rangle \right]=-{\dot{S}}_T+\frac{1}{D}\dot{W}.\hfill \end{array}$$

(50)

Here, we used Equations (23)–(26), $\langle x\rangle$ and $\beta$ in Equations (44) and (18), respectively, and $\dot{W}=\langle -\gamma u(x-ut)\rangle =u{\partial }_t\langle x\rangle$.

It is worth looking at the two interesting limits of Equations (46)–(50). First, in the long time limit as $t\to \infty$, the following simpler relations are found:

$$\begin{array}{ccc}& & \langle x\rangle \to u(t-{\gamma }^{-1}),2\beta \to \frac{\gamma }{D}=2{\beta }_s,\hfill \\ & & D{\dot{S}}_T=\dot{Q}=\dot{W}={(\sigma \Gamma )}^2\to u^2,\hfill \\ & & \dot{S}\to 0,\frac{d}{dt}\mathcal{K}\left[p(x,t)\right|p_s(x,t)]\to 0.\hfill \end{array}$$

(51)

Equation (51) illustrates how the external force $v\left(t\right)=ut\ne 0$ keeps the system out of equilibrium even in the long time limit, with non-zero entropy production and dissipation. When there is no external force $u=0$, the system reaches equilibrium as $t\to \infty$, and all quantities in Equation (51) apart from $\beta$ become zero.

The second is when the system is initially in equilibrium with $\beta (t=0)=\beta (t\to \infty )=\frac{\gamma }{2D}$ and $\langle x_0\rangle =0$ and evolve in time as it is driven out of equilibrium by $u\ne 0$. As u does not affect variance, $\beta \left(t\right)={\beta }_0=\frac{\gamma }{2D}$ (${\partial }_t\sigma =0$) and $\dot{S}=0$ for all time. In this case, we find

$$\begin{array}{ccc}& & D{\dot{S}}_T=\dot{Q}=u^2{(1-e^{-\gamma t})}^2={(\sigma \Gamma )}^2,\dot{W}=u^2(1-e^{-\gamma t}),\hfill \\ & & D\mathcal{K}\left[p(x,t)\right|p_s(x,t)]=\frac{u^2}{2\gamma }{(1-e^{-\gamma t})}^2,\hfill \\ & & D\frac{d}{dt}\mathcal{K}\left[p(x,t)\right|p_s(x,t)]=u^2(1-e^{-\gamma t})e^{-\gamma t}.\hfill \end{array}$$

(52)

Equation (52) shows that ${\dot{S}}_T$, $\dot{Q}$, ${\Gamma }^2$, $\dot{W}$, and $\mathcal{K}$ start with zero values at $t=0$ and monotonically increase to their asymptotic values as $t\to \infty$.

Finally, both cases considered above in Equations (51) and (52) have ${\partial }_t\sigma =0$ and thus recover Equation (30):

$${\Gamma }^2=\frac{D{\dot{S}}_T}{{\sigma }^2}=\frac{\dot{Q}}{{\sigma }^2}.$$

(53)

5. Inequalities

Section 4 utilized the average (first moment) of a variable (e.g., $\langle V\rangle$) and the average of its first time derivative ($\langle {\partial }_{t}V\rangle =\dot{W}$) while the work $W=\int dt\dot{W}$ is defined by the time integral of $\dot{W}=\langle {\partial }_{t}V\rangle$ in Equation (33). This section aims to show that the rates at which average quantities vary with time are bounded by fluctuations and $\Gamma$. Since the average and time derivatives do not commute, we pay particular attention to when the average is taken.

To this end, let us first define the microscopic free energy $\mu =V+Dlnp$ (called the chemical potential energy in [113]). In terms of $\mu$, we have $J=-p{\partial }_x\mu$ and $\langle \mu \rangle =U-DS=\mathcal{F}$. On the other hand,

$$\begin{array}{ccc}\hfill {\partial }_t\mu ={\partial }_{t}V+D\frac{\dot{p}}{p},& & \langle {\partial }_t\mu \rangle =\langle {\partial }_{t}V\rangle =\dot{W}.\hfill \end{array}$$

(54)

$\langle {\partial }_t\mu \rangle =\dot{W}$ means that the average rate of change in the microscopic free is the power. From Equation (54), it follows

$$\langle {({\partial }_t\mu -{\partial }_{t}V)}^2\rangle =D^2\int dx\frac{{\dot{p}}^2}{p}=D^2{\Gamma }^2.$$

(55)

Equation (55) establishes the relation between the microscopic free energy and $\Gamma$.

Next, we calculate the time-derivative of $\mathcal{F}$

$$\begin{array}{ccc}\hfill \frac{d}{dt}\mathcal{F}& =& \frac{d}{dt}\langle \mu \rangle =\langle \dot{\mu }\rangle +\int dx\mu \dot{p}=\dot{W}+\int dx\mu \dot{p}.\hfill \end{array}$$

(56)

Using $\frac{d}{dt}\mathcal{F}=\dot{W}-D{\dot{S}}_T$ in Equation (56) gives ${\dot{S}}_T$ in terms of $\mu$ as

$$\begin{array}{ccc}\hfill \dot{W}-\frac{d}{dt}\mathcal{F}=D\dot{S_T}& =& -\int dx\mu \dot{p}.\hfill \end{array}$$

(57)

Equation (57) is to be used in Section 5.1 for linking ${\dot{S}}_T$ to $\Gamma$ through an inequality.

5.1. General Inequality Relations

We now use $\int dx\dot{p}=0$, $\int dx\dot{p}\langle A\rangle =\langle A\rangle \int dx\dot{p}=0$ for any $A=A(x,t)$ and apply the Schwartz inequality $|{\int }^{t}d{t}_{1}A\left(t_1\right)B\left(t_1\right)|\le {[{\int }^{t}d{t}_1{A}^2\left(t_1\right)]}^{\frac{1}{2}}{[{\int }^{t}d{t}_1{B}^2\left(t_1\right)]}^{\frac{1}{2}}$ to Equations (21), (34) and (57) to obtain

$$\begin{array}{ccc}\hfill |\dot{S}|& =& |\int dx\dot{p}lnp|\le \Gamma {\left[\int dxp{(\delta lnp)}^2\right]}^{1/2},\hfill \end{array}$$

(58)

$$\begin{array}{ccc}\hfill |\dot{Q}|& =& |\int dxV\dot{p}|\le \Gamma {\left[\int dxp{\left(\delta V\right)}^2\right]}^{1/2},\hfill \end{array}$$

(59)

$$\begin{array}{ccc}\hfill D{\dot{S}}_T& =& \left|\int dx\mu \dot{p}\right|\le \Gamma {\left[\int dxp{\left(\delta \mu \right)}^2\right]}^{1/2}.\hfill \end{array}$$

(60)

Equation (60) (Equation (59)) establishes the inequality between entropy production rate (heat flux) and the product of the RMS fluctuations of the microscopic free energy (potential energy) and $\Gamma$. Since $\delta \mu =\delta V+D(\delta lnp)$, we have

$$\begin{array}{ccc}\hfill \langle {\left(\delta \mu \right)}^2\rangle & =& \langle {\left(\delta V\right)}^2\rangle +D^2\langle {(\delta lnp)}^2\rangle +2D\langle \delta V\delta lnp\rangle .\hfill \end{array}$$

(61)

These relations are to be used in Section 5.2 below.

5.2. Applications to the Non-Autonomous O-U Process

For a linear O-U process with $V=\frac{\gamma }{2}{(x-v\left(t\right))}^2$ and $f=-\gamma (x-v(t\left)\right)$ in Equation (14), we use $\langle {\left(\delta x\right)}^2\rangle =\frac{1}{2\beta }$, $\langle {\left(\delta x\right)}^4\rangle =3\langle {\left(\delta x\right)}^2\rangle =\frac{3}{4{\beta }^2}$ and ${\partial }_t\langle x\rangle =-\gamma \langle x-v\left(t\right)\rangle$ to show

$$\begin{array}{ccc}\hfill \delta V& =& -{\partial }_t\langle x\rangle \delta x+\frac{\gamma }{2}\left[{\left(\delta x\right)}^2-\frac{1}{2\beta }\right]\hfill \\ \hfill \langle {\left(\delta V\right)}^2\rangle & =& {\left({\partial }_t\langle x\rangle \right)}^2{\sigma }^2+{\gamma }^2\frac{{\sigma }^4}{2},\hfill \\ \hfill \delta lnp& =& \frac{1}{2}-\beta {\left(\delta x\right)}^2,\hfill \\ \hfill \langle {(\delta lnp)}^2\rangle & =& \frac{1}{2},\hfill \\ \hfill \langle (\delta lnp)\left(\delta V\right)\rangle & =& -\frac{\gamma }{4\beta }.\hfill \end{array}$$

(62)

Using Equations (61)–(62) in Equations (58)–(60) together with $2D\beta -\gamma =-\frac{{\partial }_t\beta }{2\beta }$ and $\frac{{\partial }_t\beta }{\beta }=-2\frac{{\partial }_t\sigma }{\sigma }$ leads to

$$\begin{array}{ccc}\hfill |\dot{S}|& \le & \frac{1}{\sqrt{2}}\Gamma ,\hfill \\ \hfill |\dot{Q}|& \le & \Gamma \sigma {\left[{\left({\partial }_t\langle x\rangle \right)}^2+\frac{{\gamma }^2{\sigma }^2}{2}\right]}^{\frac{1}{2}},\hfill \\ \hfill D{\dot{S}}_T& \le & \Gamma \sigma {\left[{\left({\partial }_t\langle x\rangle \right)}^2+\frac{1}{2}{\left({\partial }_t\sigma \right)}^2\right]}^{\frac{1}{2}}.\hfill \end{array}$$

(63)

Finally, it is useful to examine the extreme cases of Equation (63). First, when ${\partial }_t\sigma =0$, Equation (63) holds as an equality as $D{\dot{S}}_T={\left({\partial }_t\langle x\rangle \right)}^2={\sigma }^2{\Gamma }^2$ (see Equation (27)), recovering Equation (28) with ${\partial }_t\sigma =0$. Second, when ${\partial }_t\langle x\rangle =0$, Equation (63) again holds as an equality since $D{\dot{S}}_T={\left({\partial }_t\sigma \right)}^2$ and $\Gamma \sqrt{\frac{1}{2}{\sigma }^2{\left({\partial }_t\sigma \right)}^2}={\left({\partial }_t\sigma \right)}^2$.

6. Geodesics, Control and Hyperbolic Geometry

The section aims to discuss geodesics in information geometry and its implications for self-organization and control. To illustrate the key concepts, we utilize an analytically solvable, generalized O-U process given by

$$\frac{dx}{dt}=-\gamma \left(t\right)[x-v\left(t\right)]+\xi ,$$

(64)

where $\gamma \left(t\right)>0$ is a damping constant; $v\left(t\right)$ is a deterministic force which determines the time evolution of the mean value of x; $\xi$ is a short (delta)-correlated noise with the time-dependent amplitude $D\left(t\right)$ in general, satisfying Equation (15).

For the initial condition in Equation (16), the mean value $\langle x\rangle \equiv y\left(t\right)$ and $\beta \left(t\right)$ are given by

$$\begin{array}{ccc}\hfill y\left(t\right)& =& \langle x\rangle =x_0{e}^{-{\int }_0^{t}d{t}_1\gamma \left(t_1\right)d{t}_1}+{\int }_0^{t}d{t}_1{e}^{-{\int }_0^{t}d{t}_1[\gamma \left(t_1\right)-\gamma \left(t\right)]d{t}_1}\gamma \left(t_1\right)f\left(t_1\right),\hfill \end{array}$$

(65)

$$\begin{array}{ccc}\hfill \frac{1}{2\beta \left(t\right)}& =& \langle {(x-\langle x\rangle )}^2\rangle =\frac{e^{-2{\int }_0^{t}d{t}_1\gamma \left(t_1\right)d{t}_1}}{2{\beta }_0}+{\int }_0^{t}d{t}_1{e}^{-2{\int }_0^{t}d{t}_1[\gamma \left(t_1\right)-\gamma \left(t\right)]d{t}_1}2D\left(t_1\right),\hfill \end{array}$$

(66)

where $x_0=\langle x(t=0)\rangle$.

6.1. Geodesics–Shortest-Distance Path

A geodesics between the two spatial locations is a unique path with the shortest distance. A similar concept can be applied to information geometry to define a unique evolution path between the two given PDFs, say, $p(x,t_1)$ and $p(x,t_2)$ in the statistical space. The Wootters’ distance in quantum mechanics in Equation (4) is such an example. For time-varying stochastic processes, there is an infinite number of different trajectories between the two PDFs at different times. The key question that we address in this section is how to find an exact time evolution of $p(x,t)$ when initial and final PDFs [15] are given. This is a much more difficult problem than finding a minimum distance between two PDFs (like the Wootter’s distance). In the following, we sketch some main steps needed for finding such a unique evolution path (the so-called geodesics) between given initial and final PDFs by minimizing $\mathcal{L}$ (see [15] for detailed steps).

For the O-U process in Equation (64), a geodesic solution does not exist for constant $\gamma$, $v\left(t\right)$ and D. Thus, finding a geodesic solution boils down to determing suitable functions of $\gamma \left(t\right)$, $v\left(t\right)$ or $D\left(t\right)$ [15]. To be specific, let $p(x,t_0)$ and $p(x,t_F)$, respectively, be the PDFs at the time $t=t_0$ and $t_F$ ($>t_0$) and find a geodesic solution by minimizing $\mathcal{L}\left(t\right)={\int }_{t_0}^{t_F}d{t}^{\prime }\Gamma \left(t^{\prime }\right)$. The latter is equivalent to minimizing ${\int }_{t_0}^{t_F}d{t}^{\prime }\mathcal{E}\left(t^{\prime }\right)$ and to keeping $\Gamma$ constant. (This geodesics is also called an optimal path (e.g., see [107]).) We rewrite $\mathcal{E}$ in Equation (27) for the O-U process in terms of $y=\langle x\rangle$

$$\mathcal{E}=\frac{1}{2{\beta }^2}{\left(\frac{d\beta }{dt}\right)}^2+2\beta {\left(\frac{dy}{dt}\right)}^2.$$

(67)

The Euler-Lagrange equation

$$\begin{array}{c}\hfill 0=\frac{d\mathcal{E}}{d\beta }-\frac{d}{dt}\frac{d\mathcal{E}}{d\dot{\beta }},0=\frac{d\mathcal{E}}{dy}-\frac{d}{dt}\frac{d\mathcal{E}}{d\dot{y}}\end{array}$$

(68)

($\dot{\beta }=\frac{d\beta }{dt}$ and $\dot{y}=\frac{dy}{dt}$) then gives us

$$\begin{array}{ccc}& & \frac{d^2\beta }{d{t}^2}-\frac{1}{\beta }{\left(\frac{d\beta }{dt}\right)}^2-2{\beta }^2{\left(\frac{dy}{dt}\right)}^2=0,\hfill \end{array}$$

(69)

$$\begin{array}{ccc}& & \frac{d}{dt}\left[\beta \frac{dy}{dt}\right]=0\to \beta \frac{dy}{dt}=c,\hfill \end{array}$$

(70)

where c is constant. An alternative method of obtaining Equations (69) and (70) is provided in Appendix C. The following equations are obtained from Equations (69) and (70) [15]

$$\begin{array}{ccc}\hfill {\left(\frac{d\beta }{dt}\right)}^2& =& -4c^2\beta +\alpha {\beta }^2,\hfill \end{array}$$

(71)

$$\begin{array}{ccc}\hfill {\Gamma }^2& =& \frac{1}{2{\beta }^2}{\dot{\beta }}^2+2c^2\beta =\frac{\alpha }{2},\hfill \end{array}$$

(72)

where $\alpha$ is another (integration) constant. General solutions to Equations (70) and (71) for $c\ne 0$ were found in terms of hyperbolic functions as [15]

$$\begin{array}{ccc}\hfill \beta \left(t\right)=\frac{4c^2}{\alpha }{cosh}^2\left[\frac{1}{2}\sqrt{\alpha }(t-A)\right],& & y\left(t\right)=\frac{\sqrt{\alpha }}{2c}tanh\left[\frac{1}{2}\sqrt{\alpha }(t-A)\right]-\frac{\sqrt{\alpha }}{2c}+B,\hfill \end{array}$$

(73)

where A and B are constant.

Equation (73) can be rewritten using $\sigma ={\left(2\beta \right)}^{-\frac{1}{2}}$ and $z=\frac{y}{\sqrt{2}}$ as follows

$${(z-z_c)}^2+{\sigma }^2=R^2,z_c=\frac{B}{\sqrt{2}}-sR,R=\frac{\Gamma }{2c},$$

(74)

where s denotes the sign of c so that $s=1$ when $c>0$ while $s=-1$ when $c<0$. Equation (74) is an equation of a circle for the variables z and $\sigma$ with the radius R and the center $z_c$, defined in the upper-half plane where $\sigma \ge 0$. Thus, geodesic motions occur along the portions of a circle as long as $c\ne 0$ (as can be seen in Figure 4). A geodesic moves on a circle with a larger radius for a larger information rate $\Gamma$ and speed and vice versus. This manifests the hyperbolic geometry in the upper half Poincaré model [13,117] where the half-plane represents z and $\sigma \ne 0$ (see also Appendix D). The constants $c,\alpha ,A$, and B determine the coordinate of the center and the radius of the circle R. These constants should be fixed by the fixed conditions at the initial $t=0$ and final time $t_F$.

Having found the most general form of the geodesic solution for $y\left(t\right)$ and $\beta$, the next steps require finding the values of constant values $c,\alpha ,A,B$ to satisfy the boundary conditions at $t=t_0$ and $t_F$, and then finding appropriate $\gamma \left(t\right)$, $D\left(t\right)$, and $v\left(t\right)$ that ensure the geodesic solutions. This means the O-U process should be controlled by $\gamma \left(t\right)$, $D\left(t\right)$ and $v\left(t\right)$ to ensure a geodesic solution.

Figure 4 shows an example of a geodesic solution in the upper half-plane y and ${\beta }^{-1/2}$ when $\gamma \left(t\right)=1$ is constant while $D\left(t\right)$ and $v\left(t\right)$ are time-dependent. The boundary conditions are chosen as $y\left(t_0\right)=y_0=\frac{5}{6}$ and $y\left(t_F\right)=y_F=\frac{1}{30}$ in all panels (a)–(d). $\beta \left(t_0\right)={\beta }_0=\beta \left(t_F\right)={\beta }_F=0.3$ in panels (a) and (b) while ${\beta }_0={\beta }_F=3$ in panels (c) and (d). Interestingly, circular-shape phase-portraits are seen in panels (b) and (d), reflecting hyperbolic geometry noted above (see also Appendix D) [13,117]. The speed at which the geodesic motion takes place in the phase portrait is determined by the constant value of $\Gamma =\sqrt{\frac{\alpha }{2}}$ (i.e., the larger $\alpha$, the faster time evolution).

Figure 5a,b are the corresponding PDF snapshots at different times (shown in different colors), demonstrating how the PDF evolves from the initial PDF in red to the final PDF in blue. In both cases, it is prominent that the PDF width ($\propto {\beta }^{-1/2}$) initially broadens and then becomes narrower.

6.2. Comments on Self-Organization and Control

Self-organization (also called homeostasis) is the novel phenomena where order spontaneously emerges out of disorder and is maintained by different feedbacks in complex systems [45,52,53,118,119,120,121,122,123]. The extremum principles of thermodynamics such as the minimum entropy production (e.g., [119,121]) or maximum entropy entropy production (e.g., [122,123]) have been proposed by considering a steady state or an instant time in different problems.

However, far from equilibrium, self-organization can be a time-varying non-equilibrium process involving perpetual or large fluctuations (e.g., see [52,53,54]). In this case, the extreme of entropy production should be on accumulative entropy production over time rather than at one instant time nor in a steady state. That is, we should consider the time-integral of the entropy production ${\dot{S}}_T$, or equivalently, the time-integral of $\sqrt{{\dot{S}}_T}$. As seen from Equations (24) and (53), for a linear O-U process with a constant variance, there is an exact proportionality between $\sqrt{{\dot{S}}_T}$ and $\Gamma$. In this case, the extreme of $\mathcal{L}\left(t\right)={\int }^{t}d{t}_1\Gamma \left(t_1\right)$ would be the same as the extreme of ${\int }^{t}d{t}_1\sqrt{{\dot{S}}_T}$. However, as noted previously, $\Gamma \propto \sqrt{{\dot{S}}_T}$ does not hold in general (e.g., see Equation (28)).

With these comments, we now look at the implications of a geodesic for self-organization, in particular, in biosystems. For the very existence and optimal functions of a living organiss, it is critical to minimize the dispersion of its physical states and to maintain its states within certain bounds upon changing conditions [124]. How fast its state changes in time can be quantified by the surprise rate ${\partial }_t[ln\left(p\right(x,t)]$. Since $\int dxp(x,t){\partial }_{t}ln\left(p\right(x,t\left)\right)=0$, we use its RMS value $\sqrt{\langle {({\partial }_{t}lnp)}^2\rangle }=\Gamma$ (see Equation (9)) and realize that the total change over a finite time interval $[t_0,t_F]$ is nothing more than ${\mathcal{L}}_{(}t)={\int }_{t_0}^{t_F}d{t}_1\Gamma \left(t_1\right)$. Thus, minimizing the accumulative/time-integral of the RMS surprise rate is equivalent to minimizing $\mathcal{L}$. Envisioning surprise rate as biological cost associated with changes (e.g., needed in updating the future prediction based on the current state [124,125]), we can then interpret $\mathcal{L}$ as an accumulative biological cost. Thus, geodesic would be an optimal path that minimizes such an accumulative biological cost.

Ref [15] addressed how to utilize this idea to control populations (tumors). Specifically, the results in Section 6.1 were applied to a nonlinear stochastic growth model (obtained by a nonlinear change of variables of the O-U process), and the geodesic solution in Equation (73) was used to find the optimal protocols $v\left(t\right)$ and $D\left(t\right)$ in reducing a large-size tumor to a smaller one. Here, in this problem, $D\left(t\right)$ represents the heterogeneity of tumor cells (e.g., larger D for metastatic tumor) that can be controlled by gene reprogramming while $v\left(t\right)$ models the effect of a drug or radiation that reduces the mean tumor population/size.

7. Discussions and Conclusions

There has been a growing interest in information geometry from theoretical and practical considerations. This paper discussed some recent developments in information geometric theory, focusing on time-dependent dynamic aspects of non-equilibrium processes (e.g., time-varying mean value, time-varying variance, or temperature) and their thermodynamic and physical/biological implications.

In Section 2 and Section 3, by utilizing a Langevin model of an over-damped stochastic process $x\left(t\right)$, we highlighted the importance of a path-dependent distance $\mathcal{L}$ in describing time-varying processes. In Section 4 and Section 5, we elucidated the thermodynamic meanings of the relative entropy and the information rate $\Gamma$ by relating them to the entropy production rate (${\dot{S}}_T$), $\dot{S}$, heat flux ($Q=D{\dot{S}}_m$), dissipated work (${\dot{W}}_D$), etc., and demonstrated the role of $\Gamma$ in determining bounds (or speed limit) on thermodynamical quantities.

Specifically, in the O-U process, we showed the exact relation $\Gamma =\sqrt{\frac{D}{{\sigma }^2}{\dot{S}}_T+{\dot{S}}^2}$ (Equation (28)), which is simplified as $\sigma \Gamma =\sqrt{D{\dot{S}}_T}$ when ${\partial }_t\sigma =0$ ($\sigma =\sqrt{\langle {\left(\delta x\right)}^2\rangle }$ is the standard deviation of x). Finally, Section 6 discussed geodesic and its implication for self-organization as well as the underlying hyperbolic geometry. It remains future works to explore the link between $\Gamma$ and the entropy production rate in other (e.g., nonlinear) systems consisting of three or more interacting components or data from self-organizing systems (e.g., normal brain).