A Technical Critique of Some Parts of the Free Energy Principle

Abstract

We summarize the original formulation of the free energy principle and highlight some technical issues. We discuss how these issues affect related results involving generalised coordinates and, where appropriate, mention consequences for and reveal, up to now unacknowledged, differences from newer formulations of the free energy principle. In particular, we reveal that various definitions of the “Markov blanket” proposed in different works are not equivalent. We show that crucial steps in the free energy argument, which involve rewriting the equations of motion of systems with Markov blankets, are not generally correct without additional (previously unstated) assumptions. We prove by counterexamples that the original free energy lemma, when taken at face value, is wrong. We show further that this free energy lemma, when it does hold, implies the equality of variational density and ergodic conditional density. The interpretation in terms of Bayesian inference hinges on this point, and we hence conclude that it is not sufficiently justified. Additionally, we highlight that the variational densities presented in newer formulations of the free energy principle and lemma are parametrised by different variables than in older works, leading to a substantially different interpretation of the theory. Note that we only highlight some specific problems in the discussed publications. These problems do not rule out conclusively that the general ideas behind the free energy principle are worth pursuing.

1. Overview

In [1], it was argued that the internal coordinates of an ergodic random dynamical system with a Markov blanket necessarily appear to engage in active Bayesian inference. Here, we reproduce the argument supporting this interpretation in detail and highlight at which points it faces technical issues. In the course of our critique, we also mention issues of some closely related alternative arguments. In cases where our results have clear consequences for the more recent related publications [2,3], we also mention those. In particular, we point out a conceptual difference in these latter works that has not previously been acknowledged. However, our analysis thereof does not go beyond a few remarks. In an additional section, we discuss the effect of our argument on [4]. The logical structure of the present paper is depicted in Figure 1. We note that the technical issues presented here do not affect the validity of approaches where a (expected) free energy minimizing agent is assumed a priori, as presented in, e.g., [5]. None of [1,2,3,4] make this assumption; they instead aim to identify the conditions under which such agents will emerge within a given stochastic process. We criticize specific formal issues in the latter publications but leave open whether they can be fixed. We now briefly introduce the setting of [1] and then sketch the content of this paper. We now briefly introduce the setting of [1] and then sketch the content of this paper.

The starting point is a random dynamical system whose evolution is governed by the stochastic differential equation:

$$\begin{array}{c}\hfill \dot{x}=f\left(x\right)+\omega ,\end{array}$$

(1)

where the system state x and vector field $f\left(x\right)$ are multi-dimensional and $\omega$ is a Gaussian noise term. There is an additional assumption that the system is ergodic, such that the steady state probability density $p^{*}\left(x\right)$ is well defined (In the original paper, the ergodic density is simply denoted $p\left(x\right)$. We here add a star to highlight that it is a time independent probability density.). In this case, $-ln{p}^{*}\left(x\right)$ plays the role of a potential function, in the sense that f can be formulated in terms of its gradients [6,7].

It is then assumed that there is a coordinate system $x=(\psi ,s,a,\lambda )$ with $\psi =({\psi }_1,...,{\psi }_{n_{\psi }})$, $s=(s_1,...,s_{n_s})$, $a=(a_1,...,a_{n_a})$, and $\lambda =({\lambda }_1,...,{\lambda }_{n_{\lambda }})$, referred to as external, sensory, active, and internal coordinates (these are called “states” in [1]), respectively, such that the following condition holds:

Condition 1.

The function $f\left(x\right)$ can be written as:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill f\left(x\right)& =\left(\begin{array}{c}{f}_{\psi }(\psi ,s,a)\\ f_s(\psi ,s,a)\\ f_a(s,a,\lambda )\\ f_{\lambda }(s,a,\lambda )\end{array}\right).\hfill \end{array}\end{array}$$

(2)

This particular structure is described as “[formalizing] the dependencies implied by the Markov blanket” [1]. In contrast, more recent works [2,3] formulated the Markov blanket in terms of the statistical dependencies of the ergodic density $p^{*}\left(x\right)=p^{*}(\psi ,s,a,\lambda )$. Specifically, the following condition is presented:

Condition 2.

The ergodic density factorises as:

$$\begin{array}{c}{p}^{*}(\psi ,s,a,\lambda )=p^{*}\left(\psi \right|s,a)p^{*}\left(\lambda \right|s,a)p^{*}(s,a).\end{array}$$

(3)

In other words, the internal and external coordinates are independently distributed when conditioned on the sensory and active coordinates. This means we have two different formal expressions of what constitutes a Markov blanket in these publications, and their relationship has not previously been established.

Taking Condition 1 to hold, the argument of [1] then proceeds along the following steps:

Step 2

Rewrite the vector field $f(\psi ,s,a,\lambda )$ describing the dynamics of the system in terms of the gradient of the negative logarithm of the ergodic density $p^{*}(\psi ,s,a,\lambda )$ of that system.

Step 3

Rewrite the components $f_{\lambda }(s,a,\lambda )$ and $f_a(s,a,\lambda )$ of the vector field $f(\psi ,s,a,\lambda )$ in terms of only partial gradients of the negative logarithm of $p^{*}(\psi ,s,a,\lambda )$.

Step 4

Assert (in the free energy lemma) the existence of a density $q\left(\psi \right|\lambda )$ over the external coordinates $\psi$ parameterized by the internal coordinates $\lambda$ and that $f(\psi ,s,a,\lambda )$ can again be rewritten, this time in terms of a free energy depending on $q\left(\mathsf{\Psi }\right|\lambda )$ (here, and whenever it would otherwise be ambiguous, we use a capitalized $\mathsf{\Psi }$ to indicate full distributions, rather than the probability density for a specific value of $\psi$).

Step 5

Claim that the equivalence of the equations of motion in Step 3 and Step 4 implies that certain partial gradients of the KL divergence between $q\left(\mathsf{\Psi }\right|\lambda )$ and the conditional ergodic density $p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )$ must vanish.

Step 6

Claim that it follows from Step 5 that $q\left(\mathsf{\Psi }\right|\lambda )$ and $p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )$ are “rendered” equal.

Step 7

Interpret:

• $p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )$ as a posterior over external coordinates given particular values of sensor, active, and internal coordinates,
• $q\left(\mathsf{\Psi }\right|\lambda )$ as encoding Bayesian beliefs about the external coordinates by the internal coordinates, and
• their equality as the internal coordinates appearing to “solve the problem of Bayesian inference”.

In the present paper, we make the following main observations:

• The re-expression of Equation (1) in the form chosen in Step 2 is derived under restrictive assumptions, including that the system is subject to Gaussian and Markov noise.
• Conditions 1 and 2 are independent of each other.
• Conditions 1 and 3 together lead to a system where the interpretation of s and a as sensory and active coordinates is questionable.
• Under both Conditions 1 and 2, the expressions of $f_{\lambda }(s,a,\lambda )$ and $f_a(s,a,\lambda )$ resulting from Step 3 are not as general as those contained in the result of Step 2. The more general alternative expression derived in [2] remains insufficiently general.
• Under both Conditions 1 and 2, the free energy lemma, when taken at face value, is wrong and cannot be salvaged by using alternatives in Step 3.
• Under both Conditions 1 and 2, contrary to Step 6, the vanishing of the gradient of the KL divergence does not imply the equality of $q\left(\mathsf{\Psi }\right|\lambda )$ and $p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )$.
• As a consequence, the basic preconditions for the interpretations in Step 7 are not implied by either of the two proposed Markov blankets Conditions 1 and 2.

The latter [4] presents an argument almost identical to the one in the original [1]. In Section 8, we discuss how our observations apply to this publication.

2. Expression via the Gradient of the Ergodic Density

Here, we introduce the expression of the system’s dynamics Equation (1) in the form used for the free energy lemma (Lemma 2.1 in [1]). This form expresses the dynamics of the internal and active coordinates of the given ergodic random dynamical system in terms of the gradient of the ergodic density $p^{*}\left(x\right)$. In accordance with the results of [7], $f\left(x\right)$ is rewritten as (see Equation (2.5) in [1]):

$$\begin{array}{cc}\hfill f\left(x\right)& =(\mathsf{\Gamma }+R)·\nabla ln{p}^{*}\left(x\right),\hfill \end{array}$$

(4)

where $\mathsf{\Gamma }$ is the diffusion matrix, which we will take to be block diagonal (in [1], and later work such as [2], $\mathsf{\Gamma }$ is taken to be proportional to the identity matrix), and R is an antisymmetric matrix, defined through the relation:

$$\begin{array}{c}MR+R{M}^T=M\mathsf{\Gamma }-\mathsf{\Gamma }{M}^T,\end{array}$$

(5)

with

$$\begin{array}{c}{M}_{ij}={\nabla }_j{f}_i\left(x\right).\end{array}$$

(6)

Here, and in all of [1,2,3,4], both $\mathsf{\Gamma }$ and R are assumed constant. We emphasise here that, for general nonlinear models, these matrices can vary with the coordinates and Equation (5) holds only approximately [8,9] (the exact conditions under which these matrices can be chosen to be constant can be found in [9,10] and, for the discrete state case, [11]). Moreover, Equation (4) is derived in the literature under the explicit assumption that the fluctuations $\omega$ be Gaussian and Markov [6,7]. For the counterexamples we present here, we restrict ourselves to the class of Ornstein–Uhlenbeck processes, for which R and $\mathsf{\Gamma }$ are always constant, and the ergodic density $p^{*}\left(x\right)=p^{*}(\psi ,s,a,\lambda )$ is necessarily a multivariate Gaussian with zero mean. Specifically, following [7],

$$\begin{array}{c}\hfill p^{*}(\psi ,s,a,\lambda ):=\frac{1}{Z}exp\left[-\frac{1}{2}(\psi ,s,a,\lambda )U{(\psi ,s,a,\lambda )}^{\top }\right],\end{array}$$

(7)

where $(\psi ,s,a,\lambda )$ is a row vector and Z is a suitable normalisation constant. From Equation (4), it can be seen that,

$$\begin{array}{c}\hfill U=-{(\mathsf{\Gamma }+R)}^{-1}M;\end{array}$$

(8)

though we emphasise here that strict relations between M and U can only be made because of the assumption that $\mathsf{\Gamma }$ and R are coordinate independent [12]. This concludes Step 2.

Before moving on to Step 3, we note that, under the assumptions implicit in Step 2, we can express Conditions 1 and 2 in terms of the matrices M and U (in the nonlinear case, these matrices can still be defined in terms of the derivatives of the force vector field and potential, respectively; however, they will be generally coordinate-dependent, even when $\mathsf{\Gamma }$ and R are not [8]). Firstly, since it effectively states that ${\nabla }_{\psi }{f}_a\left(x\right)={\nabla }_{\psi }{f}_{\lambda }\left(x\right)={\nabla }_{\lambda }{f}_s\left(x\right)={\nabla }_{\lambda }{f}_{\psi }\left(x\right)=0$,

$$\begin{array}{c}\mathrm{Condition}1\iff M_{a\psi }=M_{\lambda \psi }=M_{s\lambda }=M_{\psi \lambda }=0,\end{array}$$

(9)

with $M_{\alpha \beta }$ a block sub-matrix of M in general. Secondly, because of the multivariate Gaussian nature of $p^{*}(\psi ,s,a,\lambda )$, the dependencies of conditional distributions are encoded in the inverse U of the covariance matrix; we therefore have that:

$$\begin{array}{c}\mathrm{Condition}2\iff U_{\psi \lambda }=U_{\lambda \psi }=0,\end{array}$$

(10)

where $U_{\alpha \beta }$ is a block sub-matrix of U. These implications bring us to our first observation:

Observation 1.

Neither Condition 1 (the vector field dependency structure) nor Condition 2 (conditional independence in the ergodic distribution) imply the other:

$$\begin{array}{cc}\hfill & \mathrm{Condition}1\nRightarrow \mathrm{Condition}2\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill & \mathrm{Condition}1\nLeftarrow \mathrm{Condition}2.\hfill \end{array}$$

(12)

Proof. 

In Appendix A, we provide direct counterexamples, using the equivalent constraints on the matrices M and U in Equations (9) and (10), for the implication in either direction. That is, there exists a system obeying Condition 1 that does not obey Condition 2 (proving Equation (11)), and there exists one obeying Condition 2 that does not obey Condition 1 (proving Equation (12)). □

Henceforth, unless otherwise stated, we will assume both Conditions 1 and 2. Any implications that fail to hold in this special case cannot hold generally.

3. Re-Expression Using Only Partial Gradients

For Step 3, we focus on the components $f_{\lambda }=(f_{{\lambda }_1},\dots ,f_{n_{\lambda }})$ and $f_a=(f_{a_1},\dots ,f_{n_a})$ of f. Without loss of generality, we can rewrite them from Equation (4) as:

$$\begin{array}{cc}{f}_a(s,a,\lambda )=\hfill & \left(R_{a\psi }·{\nabla }_{\psi }+R_{as}·{\nabla }_s+({\mathsf{\Gamma }}_{aa}+R_{aa})·{\nabla }_a\right.\hfill \\ & \left.+R_{a\lambda }·{\nabla }_{\lambda }\right)ln{p}^{*}(\psi ,s,a,\lambda ),\hfill \end{array}$$

(13)

$$\begin{array}{cc}{f}_{\lambda }(s,a,\lambda )=\hfill & \left(R_{\lambda \psi }·{\nabla }_{\psi }+R_{\lambda s}·{\nabla }_s+({\mathsf{\Gamma }}_{\lambda \lambda }+R_{\lambda \lambda })·{\nabla }_{\lambda }\right.\hfill \\ & \left.+R_{\lambda a}·{\nabla }_a\right)ln{p}^{*}(\psi ,s,a,\lambda ),\hfill \end{array}$$

(14)

where ${\mathsf{\Gamma }}_{nm}$ ($R_{nm}$) is the block of $\mathsf{\Gamma }$ (R) connecting derivatives with respect to the m coordinates to the time derivatives of the n coordinates. The expectation value with respect to $p^{*}\left(\psi \right|s,a,\lambda )$ leaves the left-hand side of these equations unchanged. A few manipulations ([2] cf. Equation (12.14), p. 129) reveal that, on the right-hand side, this leads to the ergodic density $p^{*}(\psi ,s,a,\lambda )$ being replaced by the marginalised ergodic density $p^{*}(s,a,\lambda )$ so that we get:

$$\begin{array}{cc}{f}_a(s,a,\lambda )=\hfill & \left(R_{a\psi }·{\nabla }_{\psi }+R_{as}·{\nabla }_s+({\mathsf{\Gamma }}_{aa}+R_{aa})·{\nabla }_a\right.\hfill \\ & \left.+R_{a\lambda }·{\nabla }_{\lambda }\right)ln{p}^{*}(s,a,\lambda )\hfill \end{array}$$

(15)

$$\begin{array}{cc}{f}_{\lambda }(s,a,\lambda )=\hfill & \left(R_{\lambda \psi }·{\nabla }_{\psi }+R_{\lambda s}·{\nabla }_s+({\mathsf{\Gamma }}_{\lambda \lambda }+R_{\lambda \lambda })·{\nabla }_{\lambda }\right.\hfill \\ & \left.+R_{\lambda a}·{\nabla }_a\right)ln{p}^{*}(s,a,\lambda ).\hfill \end{array}$$

(16)

Since ${\nabla }_{\psi }ln{p}^{*}(s,a,\lambda )=0$, the terms involving ${\nabla }_{\psi }$ drop out:

$$\begin{array}{cc}\hfill f_a(s,a,\lambda )& =\left(R_{as}·{\nabla }_s+({\mathsf{\Gamma }}_{aa}+R_{aa})·{\nabla }_a+R_{a\lambda }·{\nabla }_{\lambda }\right)ln{p}^{*}(s,a,\lambda ),\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill f_{\lambda }(s,a,\lambda )& =\left(R_{\lambda s}·{\nabla }_s+R_{\lambda a}·{\nabla }_a+({\mathsf{\Gamma }}_{\lambda \lambda }+R_{\lambda \lambda })·{\nabla }_{\lambda }\right)ln{p}^{*}(s,a,\lambda ).\hfill \end{array}$$

(18)

We are not aware of how to further simplify this equation without additional assumptions. However, in (Equations (2.5) and (2.6) of [1]), all of the off-diagonal terms are implicitly assumed to vanish, i.e., Equation (4) is equated with:

$$\begin{array}{cc}\hfill f_a(s,a,\lambda )& =({\mathsf{\Gamma }}_{aa}+R_{aa})·{\nabla }_{a}ln{p}^{*}(s,a,\lambda ),\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill f_{\lambda }(s,a,\lambda )& =({\mathsf{\Gamma }}_{\lambda \lambda }+R_{\lambda \lambda })·{\nabla }_{\lambda }ln{p}^{*}(s,a,\lambda ).\hfill \end{array}$$

(20)

This equation is the result of Step 3.

More recently (Appendix B of [2]), a more detailed discussion of Equation (4) was presented, where it was claimed that Condition 1 implies Condition 2 (cf. our Observation 1) along with the following simplification of Equations (17) and (18) ([2], Equations (12.8)–(12.11), (12.15), pp. 126–129):

$$\begin{array}{cc}\hfill f_a(s,a,\lambda )& =\left(({\mathsf{\Gamma }}_{aa}+R_{aa})·{\nabla }_a+R_{a\lambda }·{\nabla }_{\lambda }\right)ln{p}^{*}(s,a,\lambda ),\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill f_{\lambda }(s,a,\lambda )& =\left(R_{\lambda a}·{\nabla }_a+({\mathsf{\Gamma }}_{\lambda \lambda }+R_{\lambda \lambda })·{\nabla }_{\lambda }\right)ln{p}^{*}(s,a,\lambda ).\hfill \end{array}$$

(22)

However, Equations (21) and (22) are still provably less general than Equations (13) and (14), even when both Conditions 1 and 2 are satisfied.

Observation 2.

Given a random dynamical system obeying Equation (1), ergodicity, and both Conditions 1 and 2, none of Equations (19)–(22) generally hold.

Proof. 

By counterexample, see Appendix B. There, we show explicitly that a model satisfying the above assumptions does not satisfy the equations in question. □

In order to arrive at Equations (21) and (22) from Equations (17) and (18) in general, one must remove the offending “solenoidal flow” terms by fiat. That is, one assumes $R_{as}=R_{\lambda s}=0$. In [2], Equation (12.4), the following, even stronger, condition was assumed as an alternative starting point (along with Condition 2):

Condition 3.

The blocks of the R matrix appearing in Equation (4) coupling $(s,a)$ coordinates to λ and ψ coordinates and ψ coordinates to λ coordinates vanish, i.e.,

$$\begin{array}{c}{R}_{\psi s}=R_{\psi a}=R_{\psi \lambda }=R_{s\lambda }=R_{a\lambda }=0.\end{array}$$

(23)

This is claimed to imply $M_{\psi \lambda }=M_{\lambda \psi }=0$, but not the full Condition 1. However, in [3], both Conditions 1 and 3 were assumed (along with $R_{as}=0$). This prompts our next observation.

Observation 3.

In a system satisfying both Conditions 1 and 3, the internal coordinates cannot be directly influenced by the sensory coordinates: $f_{\lambda }(s,a,\lambda )=f_{\lambda }(a,\lambda )$, and the external coordinates cannot be directly influenced by the active coordinates: $f_{\psi }(\psi ,s,a)=f_{\psi }(\psi ,s)$.

Proof. 

From Equation (5), it follows that:

$$\begin{array}{c}M=(\mathsf{\Gamma }+R)M^T{(\mathsf{\Gamma }-R)}^{-1},\end{array}$$

(24)

with the inverse replaced by a pseudoinverse if $\mathsf{\Gamma }-R$ is not invertible. Therefore, if ${\mathsf{\Gamma }}_{\alpha \beta }={\delta }_{\alpha \beta }{\mathsf{\Gamma }}_{\alpha \alpha }$ and $R_{\alpha \beta }={\delta }_{\alpha \beta }{R}_{\alpha \alpha }$ for blocks of coordinates labelled by $\alpha$ and $\beta$, then:

$$\begin{array}{c}{M}_{\alpha \beta }=({\mathsf{\Gamma }}_{\alpha \alpha }+R_{\alpha \alpha })M_{\beta \alpha }^T{({\mathsf{\Gamma }}_{\beta \beta }-R_{\beta \beta })}^{-1},\end{array}$$

(25)

and $M_{\beta \alpha }=0\Rightarrow M_{\alpha \beta }=0$.

Condition 3 implies that only the nonzero blocks of R are $R_{\psi \psi }$, $R_{ss}$, $R_{sa}$, $R_{as}$, $R_{aa}$, and $R_{\lambda \lambda }$, and $\mathsf{\Gamma }$ is assumed to be block diagonal. As noted in Equation (9), Condition 1 requires that $M_{a\psi }=M_{\lambda \psi }=M_{s\lambda }=M_{\psi \lambda }=0$. Through Equation (25), these together imply that $M_{\lambda s}=M_{\psi a}=0$, and hence that:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill f\left(x\right)& =\left(\begin{array}{c}{f}_{\psi }(\psi ,s)\\ f_s(\psi ,s,a)\\ f_a(s,a,\lambda )\\ f_{\lambda }(a,\lambda )\end{array}\right),\hfill \end{array}\end{array}$$

(26)

as shown. □

In this case, the four sets of coordinates interact in a chain, and it is questionable whether the s and a coordinates can be meaningfully interpreted, respectively, as sensory inputs to the internal coordinates or their boundary-mediated influence on the external coordinates.

4. Free Energy Lemma

The relation of the dynamics of the internal coordinates to Bayesian beliefs is made by introducing a density (called the variational density) $q\left(\mathsf{\Psi }\right|\lambda )$ that is then interpreted as encoding a Bayesian belief. It is parameterized by the internal coordinates $\lambda$ and claimed to be “arbitrary”. We take this “at face value” and consider $q\left(\mathsf{\Psi }\right|\lambda )$ to be parameterized only by $\lambda$ and, therefore, to be independent of $(s,a)$. (We note that there is a convention in the literature on variational Bayesian inference, e.g., in [13], to drop the observed variables/data in the variational density. It is possible that in [1], $(s,a)$ was seen as observed variables and dropped from the variational density $q\left(\mathsf{\Psi }\right|\lambda )$ as in this convention. However, the reason that dropping the observed variables is justified in the established convention is that those observed variables are fixed throughout the minimization of the variational free energy and the parameters of the variational density do not influence the observed data in any way. In other words, the variational density is optimized for a single data point. In [1], the data point was continuously changing and partially doing so with dependence on the parameter $\lambda$ as $\dot{a}=f_a(s,a,\lambda )$. These differences and their consequences are non-trivial and beyond the scope of this paper, so we assume that the variational density does not depend on $(s,a)$.) If $q\left(\mathsf{\Psi }\right|\lambda )$ is allowed to depend on $(s,a)$, Observation 4 does not apply, and the free energy lemma is made trivially true by setting $q\left(\psi \right|s,a,\lambda ):=p^{*}\left(\psi \right|s,a,\lambda )$. The existence of the variational density $q\left(\mathsf{\Psi }\right|\lambda )$ is asserted by the free energy lemma (see Lemma 2.1 in [1]) (Explicitly, the free energy lemma asserts the existence of a free energy $F(s,a,\lambda )$ in terms of which $f(\psi ,s,a,\lambda )$ can be expressed and not the existence of $q\left(\mathsf{\Psi }\right|\lambda )$. However, since the free energy is defined as a functional of $q\left(\mathsf{\Psi }\right|\lambda )$, it exists if and only if a suitable $q\left(\mathsf{\Psi }\right|\lambda )$ exists.).

More precisely, the free energy lemma (and Step 4) asserts that for every ergodic density (equivalently as expressed in [1], for every Gibbs energy $G\left(x\right):=-ln{p}^{*}(\psi ,s,a,\lambda )$) $p^{*}(\psi ,s,a,\lambda )$ of a system obeying Equations (19) and (20), there is a free energy $F(s,a,\lambda )$, defined as:

$$\begin{array}{cc}\hfill F(s,a,\lambda ):& =-ln{p}^{*}(s,a,\lambda )+\int q\left(\psi \right|\lambda )ln\frac{q\left(\psi \right|\lambda )}{p^{*}\left(\psi \right|s,a,\lambda )}\mathrm{d}\psi \hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill & =-ln{p}^{*}(s,a,\lambda )+{\mathrm{D}}_{\mathrm{KL}}\left[q\left(\mathsf{\Psi }\right|\lambda )\right||p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )],\hfill \end{array}$$

(28)

in terms of the “posterior density” $p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )$ (here, we keep the conditioning argument $\lambda$, as in [1], and do not explicitly assume Condition 2, though our conclusions are unaffected by it), such that Equations (19) and (20) can be rewritten as:

$$\begin{array}{cc}\hfill f_a(s,a,\lambda )& =-{(\mathsf{\Gamma }+R)}_{aa}·{\nabla }_{a}F(s,a,\lambda ),\hfill \end{array}$$

(29)

$$\begin{array}{cc}\hfill f_{\lambda }(s,a,\lambda )& =-{(\mathsf{\Gamma }+R)}_{\lambda \lambda }·{\nabla }_{\lambda }F(s,a,\lambda ).\hfill \end{array}$$

(30)

It is worth considering what a proof of the free energy lemma could look like. A proof of the existence of a free energy (and therefore of the free energy lemma) would need to show that, for every system satisfying the given assumptions, there always exists a $q\left(\mathsf{\Psi }\right|\lambda )$ such that the right-hand sides of Equations (29) and (30) are equal to the right-hand sides of Equations (19) and (20). Expanding Equations (29) and (30) using (28) leads to:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill f_a(s,a,\lambda )& ={(\mathsf{\Gamma }+R)}_{aa}·{\nabla }_{a}ln{p}^{*}(s,a,\lambda )\hfill \\ \hfill & \phantom{{(\mathsf{\Gamma }+R)}_{aa}}-{(\mathsf{\Gamma }+R)}_{aa}·{\nabla }_a{\mathrm{D}}_{\mathrm{KL}}\left[q\left(\mathsf{\Psi }\right|\lambda )\right||p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )],\hfill \end{array}\end{array}$$

(31)

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill f_{\lambda }(s,a,\lambda )& ={(\mathsf{\Gamma }+R)}_{\lambda \lambda }·{\nabla }_{\lambda }ln{p}^{*}(s,a,\lambda )\hfill \\ \hfill & \phantom{{(\mathsf{\Gamma }+R)}_{\lambda \lambda }}-{(\mathsf{\Gamma }+R)}_{\lambda \lambda }·{\nabla }_{\lambda }{\mathrm{D}}_{\mathrm{KL}}\left[q\left(\mathsf{\Psi }\right|\lambda )\right||p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )].\hfill \end{array}\end{array}$$

(32)

For the equality of the right-hand sides to those of Equations (19) and (20), we need:

$$\begin{array}{cc}\hfill {(\mathsf{\Gamma }+R)}_{aa}·{\nabla }_a{\mathrm{D}}_{\mathrm{KL}}\left[q\left(\mathsf{\Psi }\right|\lambda )\right||p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )]& =0\hfill \end{array}$$

(33)

$$\begin{array}{cc}\hfill {(\mathsf{\Gamma }+R)}_{\lambda \lambda }·{\nabla }_{\lambda }{\mathrm{D}}_{\mathrm{KL}}\left[q\left(\mathsf{\Psi }\right|\lambda )\right||p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )]& =0.\hfill \end{array}$$

(34)

In other words, these equations say that the free energy lemma holds if any of the following three conditions (of strictly increasing strengths) are given:

• There is a $q\left(\mathsf{\Psi }\right|\lambda )$ such that the partial gradients ${\nabla }_a$ and ${\nabla }_{\lambda }$ of the KL divergence between the variational density and the conditional ergodic density are elements of the nullspaces of ${(\mathsf{\Gamma }+R)}_{aa}$ and ${(\mathsf{\Gamma }+R)}_{\lambda \lambda }$, respectively.
• There is a $q\left(\mathsf{\Psi }\right|\lambda )$ such that the gradients of the KL divergence to $p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )$ are equal to the nullvector:

  $$\begin{array}{c}\hfill {\nabla }_a{\mathrm{D}}_{\mathrm{KL}}\left[q\left(\mathsf{\Psi }\right|\lambda )\right||p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )]=0,\end{array}$$

  (35)

  $$\begin{array}{c}\hfill {\nabla }_{\lambda }{\mathrm{D}}_{\mathrm{KL}}\left[q\left(\mathsf{\Psi }\right|\lambda )\right||p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )]=0,\end{array}$$

  (36)
  Then, they are always elements of the nullspaces of ${(\mathsf{\Gamma }+R)}_{aa}$ and ${(\mathsf{\Gamma }+R)}_{\lambda \lambda }$, respectively.
• There is a $q\left(\mathsf{\Psi }\right|\lambda )$ such that $q\left(\mathsf{\Psi }\right|\lambda )=p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )$ (and hence, $p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )=p^{*}\left(\mathsf{\Psi }\right|\lambda )$), which implies that the KL divergence to $p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )$ vanishes for all $a,\lambda$ and the two partial gradients are always nullvectors and therefore elements of the according nullspaces.

The free energy lemma can then be proven by showing that one of these three cases follows from the conditions of the lemma. However, no attempt was made in [1] to establish this. Instead, the given proof discusses the purported consequences of the existence of a suitable $q\left(\mathsf{\Psi }\right|\lambda )$. These will be discussed in Steps 5 and 6.

Even if the free energy lemma does not hold for systems obeying Equations (19) and (20), one might expect that the systems instead only satisfy the more general Equations (21) and (22) or the most general Equations (17) and (18). For these systems, the free energy lemma would require that there is a $q\left(\mathsf{\Psi }\right|\lambda )$ such that:

$$\begin{array}{cc}\hfill f_a(s,a,\lambda )& =\left(({\mathsf{\Gamma }}_{aa}+R_{aa})·{\nabla }_a+R_{a\lambda }·{\nabla }_{\lambda }\right)F(s,a,\lambda ),\hfill \end{array}$$

(37)

$$\begin{array}{cc}\hfill f_{\lambda }(s,a,\lambda )& =\left(R_{\lambda a}·{\nabla }_a+({\mathsf{\Gamma }}_{\lambda \lambda }+R_{\lambda \lambda })·{\nabla }_{\lambda }\right)F(s,a,\lambda ).\hfill \end{array}$$

(38)

or:

$$\begin{array}{cc}\hfill f_a(s,a,\lambda )& =\left(R_{as}·{\nabla }_s+({\mathsf{\Gamma }}_{aa}+R_{aa})·{\nabla }_a+R_{a\lambda }·{\nabla }_{\lambda }\right)F(s,a,\lambda ),\hfill \end{array}$$

(39)

$$\begin{array}{cc}\hfill f_{\lambda }(s,a,\lambda )& =\left(R_{\lambda s}·{\nabla }_s+R_{\lambda a}·{\nabla }_a+({\mathsf{\Gamma }}_{\lambda \lambda }+R_{\lambda \lambda })·{\nabla }_{\lambda }\right)F(s,a,\lambda ),\hfill \end{array}$$

(40)

hold, respectively. However, we find this not to be the case in general.

Observation 4.

Given a random dynamical system obeying Equation (1), ergodicity, Conditions 1 and 2, there need not exist a free energy expressed in terms of a variational density $q\left(\mathsf{\Psi }\right|\lambda )$ such that:

(i) 

Equations (29) and (30) hold if Equations (19) and (20) do;

(ii) 

Equations (37) and (38) hold if Equations (19) and (20) do not hold, but Equations (21) and (22) do;

(iii) 

Equations (39) and (40) hold if neither Equations (19) and (20) nor Equations (21) and (22) hold, but Equations (17) and (18) do.

Proof. 

In Appendix C, we derive a set of conditions on the R and U matrices and on the putative variational density $q\left(\mathsf{\Psi }\right|\lambda )$, which follow from each of the pairs of equations in Cases (i–iii). We show that, in general, each pair leads to a contradiction, and in each case, we provide a counterexample that falls into the according system class. □

Before proceeding, we note that later works presented an alternative version of the free energy lemma, where the conditioning argument of $q\left(\mathsf{\Psi }\right|\lambda )$ was replaced by the most likely value of $\lambda$ conditional on the $(s,a)$ coordinates [2,3]. We here concern ourselves with the version apparent in [1], where $q\left(\mathsf{\Psi }\right|\lambda )$ is parametrised by the internal states themselves, but we briefly comment on the interpretation of the alternative approach in Step 7.

5. Vanishing Gradients

As mentioned in Step 4, the proof of the free energy lemma in [1] only discussed its consequences. The first proposed consequence is that expressing the vector field in terms of a free energy as in Equations (29) and (30) “requires” that the gradients with respect to a and $\lambda$ of the KL divergence vanish, i.e., that Equations (35) and (36) hold.

We mentioned in Step 4 that the implication in the opposite direction holds. This can be seen from Equations (33) and (34). However, if the nullspace of ${(\mathsf{\Gamma }+R)}_{aa}$ or ${(\mathsf{\Gamma }+R)}_{\lambda \lambda }$ is non-trivial, then the gradient may be a non-zero element of this subspace and Equations (29) and (30) will still hold. In that case, the vanishing gradients would not be necessary for the free energy lemma.

The conditions under which a non-trivial nullspace exists were discussed in [7]. In short, the nullspace is guaranteed to be trivial in the special case where $\mathsf{\Gamma }$ is positive definite. Whether or not ergodic systems with a Markov blanket can ever admit a non-trivial nullspace, and hence divergences in Equations (31) and (32) with non-vanishing gradients, is not immediately clear. However, in order to establish the necessity of Equations (35) and (36), this remains to be proven.

6. Equality of $\mathbf{Q}\left(\mathsf{\Psi }\right|\lambda )$ and $P^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )$

The proof of the free energy lemma in [1] also proposes that the vanishing of the gradients of the KL divergence, of the variational density $q\left(\mathsf{\Psi }\right|\lambda )$ from the conditional ergodic density $p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )$, implies the equality of these densities. We mentioned in Equations (5) that the implication in the opposite direction holds. This can also be seen from Equations (33) and (34). Concerning the implication in the direction proposed by [1], let us now assume that for a given system of Equations (19) and (20) holds, a variational density $q\left(\mathsf{\Psi }\right|\lambda )$ does exist, and the gradients of the KL divergence of the variational and ergodic densities vanish, i.e., Equations (35) and (36) hold. Then, consider the argument by [1] in this direct quote (comments in square brackets by us):

“However, Equation (2.6) [Equations (19) and (20) above] requires the gradients of the divergence to be zero [Equations (35) and (36)], which means the divergence must be minimized with respect to internal states. This means that the variational and posterior densities must be equal:

$$\begin{array}{c}\hfill q\left(\psi \right|\lambda )=p^{[*]}\left(\psi \right|s,a,\lambda )\Rightarrow {\mathrm{D}}_{\mathrm{KL}}=0\Rightarrow \left\{\begin{array}{cc}\hfill & (\mathsf{\Gamma }+R)·{\nabla }_{\lambda }{\mathrm{D}}_{\mathrm{KL}}=0,\hfill \\ \hfill & (\mathsf{\Gamma }+R)·{\nabla }_a{\mathrm{D}}_{\mathrm{KL}}=0.\hfill \end{array}\right.\end{array}$$

In other words, the flow of internal and active states minimizes free energy, rendering the variational density equivalent to the posterior density over external states.”

The first problem in the above quote is that the minimization of the divergence does not follow from the vanishing gradients. On the contrary, since Equations (35) and (36) must hold for all $(s,a,\lambda )$, the KL divergence:

$${\mathrm{D}}_{\mathrm{KL}}\left[q\left(\mathsf{\Psi }\right|\lambda )\right||p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )]$$

cannot depend on $(\lambda ,a)$; it therefore has no extremum (and thus no minimum) with respect to either of these coordinates.

The second problem pertains to the identification of the two distributions at a minimum. In general, if we try to find the minimum of a KL divergence between a given probability density $p_1\left(Y\right)$ and a family of densities $p_2\left(Y\right|\theta )$ parameterized by $\theta$, then the lowest possible value of zero is achieved only if there is a parameter ${\theta }_1$ such that $p_2\left(Y\right|{\theta }_1)=p_1\left(Y\right)$. If there is no such ${\theta }_1$, then the minimum value will be larger than zero. Therefore, even if the divergence were minimized, it would not need to be zero. More generally, the divergence $K\left(s\right)$ need not be zero for any value of s.

There is therefore no satisfactory reason given why the variational density $q\left(\mathsf{\Psi }\right|\lambda )$ and the posterior density $p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )$ should be equal or have low KL divergence. In fact, they need not be (Note that, since any $q\left(\mathsf{\Psi }\right|\lambda )$ that does not depend on $(s,a)$ is an element of the set of those that do, Observation 5 remains true for the case where we allow this dependence. In that case, the free energy lemma holds because we can set $q\left(\mathsf{\Psi }\right|s,a,\lambda ):=p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )$, and thus, a q exists for which the densities are actually equal. However, the claim here is that for every q that obeys the conditions in Observation 5, we must have equality.).

Observation 5.

Given a random dynamical system obeying Equation (1), ergodicity, Conditions 1 and 2. Then if, additionally,

(i) 

Equations (19) and (20) hold and the free energy lemma holds, i.e., there exists a probability density $q\left(\mathsf{\Psi }\right|\lambda )$ such that Equations (29) and (30) hold, or

(ii) 

Equations (21) and (22) hold and there exists $q\left(\mathsf{\Psi }\right|\lambda )$ such that Equations (37) and (38) hold, or

(iii) 

Equations (17) and (18) hold and there exists $q\left(\mathsf{\Psi }\right|\lambda )$ such that Equations (39) and (40) hold,

then there is no $c\ge 0$ for which it can be guaranteed that:

$${\mathrm{D}}_{\mathrm{K}\mathrm{L}}\left[q\left(\mathsf{\Psi }\right|\lambda )\right||p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )]<c.$$

(41)

In particular, it does not follow from these conditions that:

$$\begin{array}{c}\hfill q\left(\mathsf{\Psi }\right|\lambda )=p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda ).\end{array}$$

(42)

Proof. 

By example, see Appendix D. To show that the implication does not generally hold for a given system and densities $q\left(\mathsf{\Psi }\right|\lambda )$ that obey Equations (19), (20), (29), and (30), Equations (21), (22), (37), and (38), or Equations (17), (18), (39), and (40), we only have to consider a system that obeys all three pairs of equations, Equations (19) and (20), Equations (21) and (22), and Equations (21) and (22), and for which a suitable $q\left(\mathsf{\Psi }\right|\lambda )$ exist. For this system, we then need to show that the $q\left(\mathsf{\Psi }\right|\lambda )$ that obey Equations (29) and (30) are not necessarily equal (or similar) to $p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )$.

We use a variant of the model used in Appendix B as such a counterexample. This system obeys all three of Equations (19) and (20), Equations (21) and (22), and Equations (21) and (22), and the nullspace of the associated $\mathsf{\Gamma }+R$ is trivial. We identify a set of possible $q\left(\mathsf{\Psi }\right|\lambda )$ satisfying Equations (29) and (30), which implies that the gradients of the KL divergence between those $q\left(\mathsf{\Psi }\right|\lambda )$ and $p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )$ vanish, i.e., Equations (35) and (36) hold. We then demonstrate that for the $q\left(\mathsf{\Psi }\right|\lambda )$ in this set, the value of the KL divergence to $p^{*}\left(\mathsf{\Psi }\right|s,a,\lambda )$ can be arbitrarily large. □

7. Interpretation

Finally, we turn our attention to the interpretation in terms of Bayesian inference, i.e., Step 7. We again quote directly from [1]:

Because (by Gibbs inequality) this divergence [D_KL$\left[q\left(\psi \right|\lambda )\right|\left|p^{*}\left(\psi \right|s,a,\lambda )\right]$] cannot be less than zero, the internal flow will appear to have minimized the divergence between the variational and posterior density. In other words, the internal states will appear to have solved the problem of Bayesian inference by encoding posterior beliefs about hidden (external) states, under a generative model provided by the Gibbs energy.

We showed that, in general, there is no suitable variational density that is only parameterized by the internal coordinate $\lambda$. We then showed that, even if there is a suitable variational density (including those parameterized by all of $(s,a,\lambda )$), it can be arbitrarily different from the posterior density. Since the arguments for the internal flow appearing to minimize the divergence between variational and posterior density are therefore incorrect, there is no reason why the internal states should appear to have solved the problem of Bayesian inference.

As mentioned in Step 4, some newer works (e.g., [2,3]) formulated a different free energy principle, where the variational density of beliefs is parametrised not by the internal coordinates $\lambda$, but by $\overline{\lambda }(s,a)={arg\; max}_{\lambda }{p}^{*}\left(\lambda \right|s,a)$, the most likely value of the internal coordinates given the sensory and active ones. In this case, Observations 4 and 5 do not apply. However, the new parameters $\overline{\lambda }(s,a)$ are strictly a function of the sensory and active coordinates. This means we have a Markov chain (with capitalisations indicating random variables associated with the corresponding lower case coordinates (or functions of coordinates)) $\mathsf{\Lambda }\to (S,A)\to \overline{\mathsf{\Lambda }}$ and, by the data processing inequality [14], the mutual information between the both sensory and active coordinates and the belief parameter $\overline{\lambda }$ upper bounds that are between the internal coordinates and the belief parameter. It is therefore not clear to what extent the internal coordinates $\lambda$, rather than the active and sensory coordinates $(s,a)$ themselves, can be said to be encoding beliefs about the external coordinates. Note also that, on any given trajectory, unless the distribution $p^{*}\left(\lambda \right|s,a)$ is sufficiently peaked and unimodal, the internal coordinates are not guaranteed to spend most of their time close to their most likely conditional value, and (by definition if Condition 2 holds) they will not be better predictors of the external coordinates than those in the Markov blanket.

Generally, $\lambda \ne \overline{\lambda }$, and $\overline{\lambda }$ is the solution to an optimization problem that is assumed to be solved in these later works. Using this optimized variable to parametrise beliefs is therefore a considerable departure from [1]. Contrary to the impression created by the way it was referenced in [2,3], the older theory in [1] should be clearly distinguished from the newer ones in these more recent papers.

8. Consequences for Friston, K. et al. 2014

Reference [4] argued for the same interpretation as [1], but there were some differences in the argument.

The differences were the following:

• In [4], Equation (1) was formulated for “generalized states”, which we refer to here as generalized coordinates. This means that the variable x is replaced by a multidimensional variable denoted $\tilde{x}=(x,x^{\prime },x^{″},...)$.
• The Markov blanket structure was not explicitly defined via Equation (2). Formally, it was introduced directly (see [4] Equation (10)) in a less general form corresponding to Equations (19) and (20) (at the same time, [1] is referenced in connection to the Markov blanket so there seems to be no intention to replace the original definition with the stronger one). Therefore, our observations concerning Steps 2 to 4 are not directly relevant to this paper.
• The internal coordinate $\lambda$ was renamed to r, and the role of matrix R was played by the matrix $-Q$.
• The proof of the free energy lemma given in [4] was different. It (implicitly) suggested setting the variational density equal to the ergodic conditional posterior.
• The proof of the free energy lemma no longer contained the proposition that the gradient of the KL divergence of the variational density and the ergodic conditional density vanish, i.e., Step 5.
• The proof also no longer contained the claim that the vanishing gradients of the KL divergence of the variational density and the ergodic conditional density imply the equality of those densities, i.e., Step 6 was not present.

The interpretation in terms of Bayesian inference was unchanged and still relied on the equality of the variational and the ergodic conditional density.

Since there were no explicit generalized coordinate versions of Steps 2, 3, 5 and 6 in [4], we do not discuss those steps here. We only disprove the free energy lemma and the claim that when the free energy lemma holds, the variational and ergodic conditional density become equal. For this, we present a way to translate the counterexamples used in Observations 4 and 5 into counterexamples in generalized coordinates. The interpretation in terms of Bayesian inference given in [4] is therefore equally as unjustified as the one in [1].

For completeness, we first state the generalized coordinate versions of the stochastic differential Equation (1):

$$\begin{array}{c}\hfill \dot{\tilde{x}}=f\left(\tilde{x}\right)+\tilde{\omega },\end{array}$$

(43)

the less general version of the Markov blanket structure Equation (2):

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill f_{\tilde{\psi }}(\tilde{\psi },\tilde{s},\tilde{a})& ={(\mathsf{\Gamma }-Q)}_{\tilde{\psi }\tilde{\psi }}{\nabla }_{\tilde{\psi }}ln{p}^{*}(\tilde{\psi },\tilde{s},\tilde{a},\tilde{r})\hfill \\ \hfill f_{\tilde{s}}(\tilde{\psi },\tilde{s},\tilde{a})& ={(\mathsf{\Gamma }-Q)}_{\tilde{s}\tilde{s}}{\nabla }_{\tilde{s}}ln{p}^{*}(\tilde{\psi },\tilde{s},\tilde{a},\tilde{r})\hfill \\ \hfill f_{\tilde{a}}(\tilde{s},\tilde{a},\tilde{r})& ={(\mathsf{\Gamma }-Q)}_{\tilde{a}\tilde{a}}{\nabla }_{\tilde{a}}ln{p}^{*}(\tilde{\psi },\tilde{s},\tilde{a},\tilde{r})\hfill \\ \hfill f_{\tilde{r}}(\tilde{s},\tilde{a},\tilde{r})& ={(\mathsf{\Gamma }-Q)}_{\tilde{r}\tilde{r}}{\nabla }_{\tilde{r}}ln{p}^{*}(\tilde{\psi },\tilde{s},\tilde{a},\tilde{r}),\hfill \end{array}\end{array}$$

(44)

the expression of the $\tilde{a}$ and $\tilde{r}$ components of the vector field in terms of the marginalised ergodic density Equations (19) and (20):

$$\begin{array}{cc}\hfill f_{\tilde{a}}(\tilde{s},\tilde{a},\tilde{r})& ={(\mathsf{\Gamma }-Q)}_{\tilde{a}\tilde{a}}·{\nabla }_{\tilde{a}}ln{p}^{*}(\tilde{s},\tilde{a},\tilde{r}),\hfill \end{array}$$

(45)

$$\begin{array}{cc}\hfill f_{\tilde{r}}(\tilde{s},\tilde{a},\tilde{r})& ={(\mathsf{\Gamma }-Q)}_{\tilde{r}\tilde{r}}·{\nabla }_{\tilde{r}}ln{p}^{*}(\tilde{s},\tilde{a},\tilde{r}),\hfill \end{array}$$

(46)

and in terms of free energy Equations (29) and (30):

$$\begin{array}{cc}\hfill f_{\tilde{a}}(\tilde{s},\tilde{a},\tilde{r})& ={(Q-\mathsf{\Gamma })}_{\tilde{a}\tilde{a}}·{\nabla }_{\tilde{a}}F(\tilde{s},\tilde{a},\tilde{r}),\hfill \end{array}$$

(47)

$$\begin{array}{cc}\hfill f_{\tilde{r}}(\tilde{s},\tilde{a},\tilde{r})& ={(Q-\mathsf{\Gamma })}_{\tilde{r}\tilde{r}}·{\nabla }_{\tilde{r}}F(\tilde{s},\tilde{a},\tilde{r}).\hfill \end{array}$$

(48)

The free energy lemma then requires that there exists $q\left(\tilde{\mathsf{\Psi }}\right|\tilde{r})$ such that the KL divergence between $p^{*}\left(\tilde{\mathsf{\Psi }}\right|\tilde{s},\tilde{a},\tilde{r})$ vanishes. Without going into further details of the difference between the proof in [4] and that in [1], we can prove the former wrong by translating the counterexample used for the latter into generalised coordinates.

Observation 6.

There is a general way to translate a system in ordinary coordinates into a system of generalised coordinates that corresponds to an infinite number of independent copies of the original system. This means all properties of the original system (e.g., linearity, ergodicity, the Gaussian and Markovian property of the noise, Conditions 1 and 2, the properties of $\mathsf{\Gamma },R,U$) are preserved during this translation.

Proof. 

By construction, see Appendix E. □

This implies that the counterexamples used in proving Observations 4 and 5 directly translate to the setting of the generalised coordinates. The free energy lemma is therefore also wrong for generalised coordinates, and the variational density $q\left(\tilde{\mathsf{\Psi }}\right|\tilde{r})$ is not “ensured” [4] to be equal to the conditional ergodic density $p^{*}\left(\tilde{\mathsf{\Psi }}\right|\tilde{s},\tilde{a},\tilde{r})$.

9. Conclusions

We find that the two different Markov blanket conditions proposed in [1,2,3] are independent of each other. We then show that under both of those Markov blanket conditions, among the six steps contained in the argument in [1], three do not hold independently of each other. We also show that fixing the second of those steps (Step 3) does not provide a valid alternative. The line of reasoning of [1] therefore does not support its claim that the internal coordinates of a Markov blanket “appear to have solved the problem of Bayesian inference by encoding posterior beliefs about hidden (external) [coordinates], …”. We also show that using generalised coordinates as in [4] does not remedy the situation. Additionally, we identify a technical error in [2] and an interpretational issue resulting from possibly too strong assumptions (both Conditions 1 and 3) in [3]. We also highlight that the latter publications both argued that it is the most likely internal coordinates given sensory and active coordinates that encode posterior beliefs about external states instead of the internal coordinates themselves. The resulting free energy principle and lemma are therefore a different proposal. This is not subject to our technical critique.