Conceptual Inadequacy of the Shore and Johnson Axioms for Wide Classes of Complex Systems

Abstract

It is by now well known that the Boltzmann-Gibbs-von Neumann-Shannon logarithmic entropic functional (S_BG) is inadequate for wide classes of strongly correlated systems: see for instance the 2001 Brukner and Zeilinger’s Conceptual inadequacy of the Shannon information in quantum measurements, among many other systems exhibiting various forms of complexity. On the other hand, the Shannon and Khinchin axioms uniquely mandate the BG form $S_{BG}=-k{\displaystyle {\sum }_i{p}_i}\ln p_i$; the Shore and Johnson axioms follow the same path. Many natural, artificial and social systems have been satisfactorily approached with nonadditive entropies such as the $S_q=k\frac{1-{\displaystyle {\sum }_i{p}_i^q}}{q-1}$ one (q ∈ $ℛ$; S₁ = S_BG), basis of nonextensive statistical mechanics. Consistently, the Shannon 1948 and Khinchine 1953 uniqueness theorems have already been generalized in the literature, by Santos 1997 and Abe 2000 respectively, in order to uniquely mandate S_q. We argue here that the same remains to be done with the Shore and Johnson 1980 axioms. We arrive to this conclusion by analyzing specific classes of strongly correlated complex systems that await such generalization.

In classical mechanics we are allowed to simultaneously measure a coordinate and its conjugate momentum. Not so in quantum mechanics, where these two variables appear as correlated in a well known subtle manner. Planck constant h precisely plays the role of characterizing this correlation. It is in this sense that the formal limit h → 0 makes quantum mechanics to recover the classical one. Similarly, space and time play independent roles in classical mechanics. Not so in special relativity, where they mutually influence each other through the Lorentz transformation. The vacuum light velocity c precisely plays the role of characterizing this nontrivial mutual influence. It is in this sense that the formal limit 1/c → 0 makes relativistic mechanics to recover the classical one. In the theory of probabilities and statistical mechanics, a totally analogous situation occurs. Indeed, if systems A (with probabilities $\left\{p_i^A\right\},i=1,2,\dots ,W_A$) and B (with probabilities $\left\{p_j^B\right\},j=1,2,\dots W_B$) are probabilistically independent, the joint probabilities of the system (A + B) are simply given by $p_{ij}^{A+B}=p_i^A{p}_j^B$. If, however, correlations exist between A and B, this lack of independence can be characterized by some parameter(s) (from now on noted κ), which can always be defined in such a way that independence is recovered when κ vanishes. In such situation we have that $p_{ij}^{A+B}=f_{ij}(p_i^A,p_j^B;\kappa )$ where f_ij is a function typically satisfying f_ij(x, y; 0) = xy. All these rather elementary remarks apply to all kinds of approaches in theoretical physics and elsewhere: related arguments can be seen, for instance, in [1].

In a recent paper [2] (see also [3]), Pressé et al., claim that they proved that nonadditive entropies like those yielding nonextensive statistical mechanics [4–7] (as well as many others such as the Anteneodo-Plastino entropy, Curado entropy, Borges-Roditi entropy, Kaniadakis entropy, among others) are logically inadmissible because they would introduce biases that are not warranted by the data. They base their arguments on the Shore and Johnson (SJ) set of axioms. This set is consistent with the Boltzmann-Gibbs (BG) entropy, which, as well known, is additive in the Penrose sense [8] (i.e., additive with regard to probabilistically independent subsystems).

Here we emphasize that nonadditive entropies emerge from strong correlations (among the random variables of the system) which are definitively out of the SJ hypothesis. Indeed, the interesting SJ axiomatic framework addresses essentially systems for which it is possible to provide information about their subsystems without providing information about the interactions between the subsystems, which clearly is not the case where nonadditive entropies are to be used. Let us be more specific. The SJ axioms focus on systems whose total number W (N) of microscopic possibilities exponentially increases with N in the N → ∞ limit, i.e., when W (N) ∼ µ^N (µ > 1) (hereafter referred to as the independent or exponential class), or equivalently when W (N + 1) ∼ W (N) × µ, where µ essentially corresponds to the nonvanishing Lebesgue measure of the one-particle phase space. The entropy to be used for such systems clearly is (in its discrete version) the BG one, $S_{BG}=k{\displaystyle {\sum }_{i=1}^W{p}_i}\ln \frac{1}{p_i}(\text{with}{\displaystyle {\sum }_{i=1}^W{p}_i=1})$, which scales with N like its equal-probability particular instance, namely S_BG = k ln W. Indeed, for the so called exponential class (which includes strict independence as the particular case W (N) = µ^N, with µ > 1)), we have S(N) = k ln W (N) ∼ N, thus providing an extensive entropy as thermodynamically required.

In relevant contrast with the above case, strong correlations between the N elements do exist in great variety of natural, artificial and social systems. For example, it might be W (N) ∼ N^ρ (ρ > 0) (referred to as the power-law class), hence $W(N+1)\sim W(N){(\frac{N+1}{N})}^{\rho }\sim W(N)(1+\frac{\rho }{N})$. The entropy corresponding to this case is $S_q=k\frac{1-{\displaystyle {\sum }_{i=1}^W{p}_i^q}}{q-1}=k{\displaystyle {\sum }_{i=1}^W{p}_i}{\ln }_q\frac{1}{p_i}=-k{\displaystyle {\sum }_{i=1}^W{p}_i^q}{\ln }_q{p}_i=-k{\displaystyle {\sum }_{i=1}^W{p}_i}{\ln }_{2-q}{p}_i$ with ${\ln }_{q}z\equiv \frac{z^{1-q}-1}{1-q}(z>0;{\ln }_{1}z=\ln z)$ and $q=1-\frac{1}{\rho }$. Indeed, the extremal S_q (occurring for equal probabilities) is given by S_q = k ln_q W, and, for that specific value of q, we verify that S_q=1−1/ρ(N) ∝ N, thus once again providing an extensive entropy as thermodynamically required. Another example of strong correlations between the N elements of the system occurs when $W(N)\sim {\nu }^{N^{\gamma }}$ (ν > 1; 0 < γ < 1) (referred to as the stretched-exponential class), hence $W(N+1)\sim W(N)\frac{{\nu }^{{(N+1)}^{\gamma }}}{{\nu }^{N^{\gamma }}}\sim W(N)(1+\frac{\gamma \ln \nu }{N^{1-\gamma }})$ An entropy corresponding to this case is $S_{\delta }=k{\displaystyle {\sum }_{i=1}^W{p}_i}{(\ln \frac{1}{p_i})}^{\delta }$ with δ = 1/γ (footnote in page 69 of [6]; see also [9]). Indeed, the extremal S_δ (occurring for equal probabilities) is given by S_δ = k(ln W)^δ, and, for that specific value of δ, we verify that S_δ=1/γ(N) ∝ N, thus once again providing an extensive entropy as thermodynamically required. Notice that, in the limit N → ∞, 1 ≪ N^ρ ≪ ${\nu }^{N^{\gamma }}$ ≪ µ^N, which means that the phase-space Lebesgue measure corresponding to the power-law and stretched-exponential classes vanishes, whereas that corresponding to the exponential class remains finite. This is the deep reason why, in the presence of strong correlations (e.g., the power-law and the stretched-exponential classes), the BG entropy is generically not physically appropriate anymore since it violates the thermodynamical requirement of entropic extensivity (the reasons why entropy is in all cases required to be thermodynamically extensive remains outside the scope of the present paper; the interested reader can however refer to [10–14]). The above cases are summarized in

Table 1. Additive and nonadditive entropic functionals and classes of systems for which the entropy is extensive. W(N) is the number of admissible microscopic configurations of a system with N elements; only configurations with nonvanishing occurrence probability are considered admissible.

	ENTROPY
W(N) (N → ∞)	SBG (ADDITIVE)	Sq (q ≠ 1) (NONADDITIVE)	Sδ (δ ≠ 1) (NONADDITIVE)
∼ µN (µ > 1)	EXTENSIVE	NONEXTENSIVE	NONEXTENSIVE
∼ Nρ (ρ > 0)	NONEXTENSIVE	EXTENSIVE (q = 1 − 1/ρ)	NONEXTENSIVE
$\sim {\nu }^{N^{\gamma }}(\nu >1;0<\gamma <1)$	NONEXTENSIVE	NONEXTENSIVE	EXTENSIVE (δ = 1/γ)

. We see therefore that the SJ set of axioms, demanding, as they do, system independence, are not applicable unless we have indisputable reasons to believe that the data that we are facing correspond to a case belonging to the exponential class, and by no means correspond to strongly correlated cases such as those belonging to the power-law or stretched-exponential classes, or even others [15–17]. Even if we were not particularly interested in thermodynamics, it is clear that the scenario of system independence must be assumed for the data in order to justify the use of the SJ framework. If we may insist, the actual data must be minimally consistent with a scenario of system independence; if they are not, the whole inference procedure needs by all means to follow a different path. This central point is missed in [2]. Incidentally, there are in [2] a few more inadvertences or inadequacies, the most prominent of which we highlight now.

(a) On the nature of the index q

It is emphasized in [2] that q would be obtained not from first principles but only as a fitting parameter. To support such a viewpoint, the authors quote a nontechnical article published over a decade ago (their Ref. [25] in [2]). They seem to be unaware of the many examples where q has in fact been obtained from (dynamical) first principles. Let us quote here some of them, among many others: (i) The value of q (both for the sensitivity to the initial conditions and the entropy production per unit time) at the Feingenbaum point of the logistic map is [18,19] $q=1-\frac{\ln 2}{\ln {\alpha }_F}=0.2444877013\dots$. (1018 exact digits are presently known: see for instance [6]; α_F is the Feigenbaum universal constant); (ii) The value of q for the q-Gaussian distribution of velocities of cold atoms in optical lattices is given [20–22] by q = 1 + 44E_R/U₀, where E_R is the recoil energy and U₀ the potential depth; (iii) The value of q in order to have an extensive block entropy in the class of one-dimensional systems at a quantum critical point characterized by the central charge c is given by $q=\frac{\sqrt{9+c^2}-3}{c}$ (c > 0; notice that c → ∞ yields q → 1) [23]; (iv) The index q for the stationary-state distribution in space (and, in fact, also in momenta) in overdamped systems of the type-II superconductors in the presence of an external confining potential is given [24] by a q-exponential of the potential with q = 0, which, within the variational framework which uses linear constraints, corresponds to an entropic index q = 2 − 0.

However, even if it is definitively wrong that q cannot be derived from first principles, it is certainly true that, in the literature, it does frequently play the role of a fitting parameter. This is a natural consequence of the simple fact that the calculation of q from first principles demands the exact knowledge of the microscopic dynamics, which is very rarely available (and, even when it is available, it frequently involves mathematically intractable calculations). This has no deeper epistemological significance than the well known fact that the precise orbits of the planets of our planetary system are not in practice obtained from first principles but rather from astronomical observations and fittings. This by no means implies that Newtonian mechanics is not a first-principle theory, but it just exhibits that its full calculation is virtually impossible because the exact knowledge of the initial conditions of all the masses of the planetary system is required. In spite of that gigantic difficulty, classical mechanics is nevertheless capable of easily establishing the (approximate) form of all those orbits, namely the Keplerian elliptic form. Similarly, even when q is not analytically accessible, basic properties for wide classes of complex systems can be satisfactorily fitted with the q-exponential and q-Gaussian forms that are implied by the extremization of the S_q entropy.

(b) On nonindependent probabilities

We read in [2] that “the Tsallis entropy can only be justified if events i and j were to have the following joint probability, $p_{ij}^{q-1}=(u_i^{q-1}+v_j^{q-1}-1)$”, and “We apply SJ’s approach to derive what joint probability for states of two systems would be required to justify the form of the Tsallis entropy”. In the notation currently used in q-entropies and nonextensive statistical mechanics, these statements correspond to 1/p_ij = (1/u_i) ⊗_q (1/v_j), or alternatively to p_ij = u_i ⊗_2−q v_j, where x ⊗_q y ≡ [x^1−q + x^1−q −1]^1/1−q (x ⊗₁ y = xy) is referred to as the q-product [25,26]. There is in [2] some degree of confusion with respect to this point. Let us be precise. If we have the particular instance of equal probabilities, then S_q = k ln_q W, hence indeed we have the extensive-like equality S_q(W_u ⊗_q W_v) = S_q(W_u) + S_q(W_v), to be contrasted with the nonadditive expression $\frac{S_q(W_u\times W_v)}{k}=\frac{S_q(W_u)}{k}+\frac{S_q(W_v)}{k}+(1-q)\frac{S_q(W_u)}{k}\frac{S_q(W_v)}{k}$, W_u and W_v being the total number of states of the systems with probabilities {u_i} and {v_j} respectively. But, for the generic case (i.e., when we have not necessarily equal probabilities), we do not have the simple expression indicated in [2] for the composition of q-entropies. Indeed, $S_q(\{u_i{\otimes }_{2-q}{v}_j\})=-k{\displaystyle {\sum }_{ij}(u_i{\otimes }_{2-q}{v}_j)}{\ln }_{2-q}(u_i{\otimes }_{2-q}{v}_j)=-k{\displaystyle {\sum }_{ij}(u_i{\otimes }_{2-q}{v}_j)}({\ln }_{2-q}{u}_i+{\ln }_{2-q}{v}_j)\ne -k{\displaystyle {\sum }_{ij}{u}_i{v}_j}({\ln }_{2-q}{u}_i+{\ln }_{2-q}{v}_j)=-k{\displaystyle {\sum }_{i=1}^W{u}_i}{\ln }_{2-q}{u}_i-k{\displaystyle {\sum }_{j=1}^W{v}_j}{\ln }_{2-q}{v}_j=S_q(\{u_i\})+S_q(\{v_j\})$. This is an interesting and nontrivial consequence of this class of correlations (curiously enough, referred in [2] to as “spurious correlations”) between probabilistic events.

(c) On unconventional averages

We read in [2] that “Furthermore, unconventional averages must often be used to constrain nonadditive entropies (their Refs. [28–33]) to assure the convexity of those functions if they are to be used to infer a unique model.” This claim appears to be kind of unclear since, for example, the nonadditive entropic functional S_q is itself concave for all q > 0 (in contrast, for instance, with the additive Renyi entropic functional, which is concave only for 0 < q ≤ 1). When we want to go one step further, and produce a statistical mechanics, then constraints must indeed be added, and they naturally play a relevant role. From an information-theory standpoint, constraints must be simple and robust, such as mean values and widths. These quantities are typically taken to be 〈x〉 and 〈x²〉, since they simply characterize the location and the width (given by $\sqrt{\langle x^2\rangle -{\langle x\rangle }^2}$) of the distribution of probabilities p(x) of the random variable noted x. These quantities are mathematically admissible as long as they are finite (which typically occurs if p(x) decays quickly enough, for example exponentially quick, at large values of |x|). When we have very-long-tailed power-laws these quantities diverge, and we must therefore replace the conventional constraints by mathematically appropriate ones. For example, the mean value 〈x〉 for the q-exponential distribution $p(x)\propto e_q^{-\beta x}$ (with β > 0 and x ≥ 0; $e_q^z\equiv {[1+(1-q)z]}^{1/(1-q)}$, and $e_1^z=e^z$) diverges for q ≥ 3/2, whereas appropriately q-generalized mean values remain finite as long as the distribution p(x) remains normalizable (in this example, for q < 2). Another example: if we had $p(x)\propto e_q^{-\beta x^2}(\beta >0;-\infty <x<\infty )$, then 〈x²〉 diverges for q ≥ 5/3, whereas the appropriately q-generalized variance remains finite as long as the distribution p(x) remains normalizable (in this example, for q < 3). These important mathematical issues are illustrated in [27] and analytically discussed in [28]; see also [29] for transformations associated with constraints of different kinds within nonextensive statistical mechanics. In the case of power-laws, the authors of [2] offer as alternative to use S_BG with logarithmic constraints: “We conclude by adding that it is possible to infer power laws within a principle of maximizing the BG entropy by constraining just one average: Mandelbrot (their Ref. [35]) showed this by invoking logarithmic constraints 〈ln x〉 [to avoid confusion, we have noted x, and not k the random variable]”. If we do this, we obtain p(x) ∝ 1/x^η (η ∈ $ℛ$; 0 < x < ∞), which is not normalizable for any value of η. In fact, given any admissible entropy and any distribution of probabilities, it is always possible to find constraints such that the extremization of that entropy yields precisely that distribution. Clearly, it is not in this manner that theory of information is to be used. Another most relevant aspect which also justifies the use, within a variational framework, of both additive and nonadditive entropies is the following. If we consider the standard (homogeneous and linear) Fokker-Planck equation in the presence of an external confining potential V (x), we obtain for its unique stationary state p(x) ∝ e^{−βV (x)}, which precisely coincides with the distribution which uniquely optimizes S_BG with fixed 〈V (x)〉. If we consider now quite general inhomogeneous and/or nonlinear Fokker-Planck equations still in the presence of the confining potential V (x), its unique stationary state once again precisely coincides with the distribution extremizing specific generalized entropies satisfying the H-theorem, and whose functional form is mandated by the specific inhomogeneity and/or nonlinearity [30–35].

Last but not least, a considerable amount of predictions, verifications and applications of nonadditive entropies are today available in the literature which are useful in theoretical, experimental, observational and computational approaches of a wide variety of systems. Among many others, we may illustrate the existing bibliography [36] with: (i) The velocities of cold atoms in dissipative optical lattices [21]; (ii) The velocities of particles in quasi-two dimensional dusty plasma [37]; (iii) Single ions in radio frequency traps interacting with a classical buffer gas [38]; (iv) The relaxation curves of RKKY spin glasses, like CuMn and AuFe [39]; (v) Hadronic transverse momenta distributions at LHC experiments [40]; (vi) Long-ranged-interacting many-body classical Hamiltonians [41,42]; (vii) Nonlinear generalizations of the Schroedinger, Klein-Gordon and Dirac equations through q-plane waves [43,44]; (viii) The black-hole entropy and the so called area-law in quantum systems [10–13]; (ix) The (area-preserving) standard map [45]. Also q-generalized forms, still under progress, of the Central Limit Theorem have emerged from this theory [46,47].

For better understanding the domain of validity of the SJ System Independence Axiom (Axiom III of Ref. [14] of [2]), let us focus on a specific case. Consider, for example, a one-dimensional system at zero temperature, where it displays a quantum critical point, say an Ising ferromagnetic chain in the presence of a transverse magnetic field at its critical value. Let us assume that the chain has N first-neighbor-connected spins, and let us focus on a subsystem of it made of L successive spins, see for instance [23]. Let us note ρ_L ≡ Tr_N−LρN the density matrix of the subset, which is obtained by tracing over (N − L) spins the density matrix ρ_N of the total chain. We define the block q-entropy as follows: $S_q(L,N)=k\frac{1-Tr{(\rho L)}^q}{q-1}$, S1(L, N) = −kTrρ_L ln ρ_L being of course the block Boltzmann-Gibbs-von Neumann entropy. The entropy of the total chain is given by S_q(N, N). For the strongly quantum-entangled state that we are considering here the system is in a pure state (i.e., $Tr{\rho }_N^2=1$), therefore S_q(N, N) vanishes for all N and all q, in particular for q = 1. Consequently the asymptotic entropy per particle ${\lim }_{N\to \infty }\frac{S_q(N)}{N}$ also vanishes, ∀q. In remarkable contrast, the block of L spins is in a very different state, namely a statistical mixture (i.e., $Tr{\rho }_L^2<1$). Consequently, the block entropy per particle has a very different behavior than that of the full entropy per particle. More precisely, ${\lim }_{L\to \infty }\frac{S_q(L,\infty )}{L}$ vanishes for q > q*, diverges for q < q*, and is finite for q = q*, where q* < 1, and S_q(L, ∞) ≡ lim_N→∞ S_q(L, N) [23]. All these somewhat counterintuitive facts come from the fact that, if we divide the full chain into say an L-block and an (N − L)-block, it is not possible to provide informations on each of the two blocks that would not interact, in the sense that doing measurements on one of them would not affect the other one. This fact does not accommodate with the SJ set of axioms (particularly the System Independence Axiom). The same happens for the power-law and stretched-exponential classes discussed earlier in this paper.

The well known set of axioms of Shannon [48] and of Khinchin [49] mandate a unique entropic functional, namely S_BG. They have both been q-generalized (in [50] and [51] respectively), and the unique entropic functional that is then mandated is S_q. Let us conclude by mentioning that we believe that, similarly, the Shore-Johnson axioms might be generalizable in order to cover S_q or even other nonadditive entropies. Undoubtedly, such a generalization would definitively be welcome.