Next Article in Journal
Entropy, Age and Time Operator
Previous Article in Journal
Tsallis Distribution Decorated with Log-Periodic Oscillation
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Letter

On an Objective Basis for the Maximum Entropy Principle

Department of Electrical Engineering, The Pennsylvania State University, University Park, PA 16802, USA
*
Author to whom correspondence should be addressed.
Submission received: 26 September 2014 / Revised: 12 January 2015 / Accepted: 15 January 2015 / Published: 19 January 2015
(This article belongs to the Section Information Theory, Probability and Statistics)

Abstract

:
In this letter, we elaborate on some of the issues raised by a recent paper by Neapolitan and Jiang concerning the maximum entropy (ME) principle and alternative principles for estimating probabilities consistent with known, measured constraint information. We argue that the ME solution for the “problematic” example introduced by Neapolitan and Jiang has stronger objective basis, rooted in results from information theory, than their alternative proposed solution. We also raise some technical concerns about the Bayesian analysis in their work, which was used to independently support their alternative to the ME solution. The letter concludes by noting some open problems involving maximum entropy statistical inference.

1. Introduction

In a recent paper, “A note of caution on maximizing entropy” [1], the authors considered the problem of estimating a probability mass function given supplied constraint information. They identified as “problematic” the maximum entropy solution for the example of a 3-sided die, where the given constraint information is that the mean die value is two. For this example, maximum entropy (ME) solves the following problem:
p ¯ ME = arg max p ¯ i = 1 3 p i log p i
subject to i = 1 3 i p i = 2 i = 1 3 p i = 1
In this case, one can easily show that the maximum entropy solution, consistent with the given constraints, is the uniform distribution p i = 1 3 , i = 1 , 2 , 3.
In their paper, the authors propose an alternative “objectively-based” approach for solving this problem. Specifically, they suppose that the probabilities are random variables, uniformly distributed over their ranges, which are prescribed by the given constraints, i.e., p1 = p3 [0, 0.5] and p2 [0, 1]. Accordingly, they choose, as their estimated probabilities, the expected values of these (uniformly distributed) random variables: p ¯ = ( p 1 , p 2 , p 3 ) = ( 1 4 , 1 2 , 1 4 ). The authors argue on intuitive grounds that their solution may be preferable to the ME solution, as they state: “p2 could be as high as 1, while the other probabilities are bounded above by 0.5….[so] we may be inclined to bet on 2. Once the information gives us reason to prefer one alternative over the others, it is troublesome to claim that the probabilities…are equal.” They then also consider a Bayesian learning setting and show, under particular stated assumptions, that Bayesian updating is consistent with their ( 1 4 , 1 2 , 1 4 ) solution. Beyond identifying what the authors call a “problematic” example for the maximum entropy principle, their paper gives historical background on the interpretation of probability, including excerpts of Jaynes’ views on maximum entropy and some of the multiple senses in which, based on Jaynes’ writings, one can construe that the maximum entropy principle gives “objective” probabilities.
In this letter, we do not attempt to elucidate or specifically articulate Jaynes’ understanding of the maximum entropy principle. The purpose of this letter is to elaborate further on the 3-sided die problem from [1] (as well as related problems, where ME is often applied) in order to further understand and explicate several statistically objective bases for preferring one set of probability assignments over another. In so doing, we will argue that there is strong, objective support for the ME solution, as opposed to the alternative solution proposed by Neapolitan and Jiang. We also identify some open problems in maximum entropy statistical inference.

2. On Objective Bases For Preferring One Probability Assignment Over Another

2.1. “Most Probable” Interpretation of Maximum Entropy

In [2], Jaynes does provide a principled basis for preferring the maximum entropy solution over alternative probability assignments. Specifically, let N be the number of repeated trials of an experiment with K possible outcomes {ω1, ω2, …, ωK}, and with some constraint information, such as the mean die value measured based on these N repeated trials. Note that the outcomes of the individual trials are not known. Nor is it known the number of occurrences (Nk) of each distinct outcome, ωk. However, suppose that we did know Nk, k = 1, …, K. For large N, by the weak law of large numbers, e.g., [3], we know that N k N p k with probability 1, where pk, k = 1, …, K are the true probabilities. Thus, if (N1, N2, …, NK) were known, a good choice for the probability assignments would be the frequency counts ( N 1 N , N 2 N , , N K N ). Accordingly, estimating p ¯ = ( p 1 , p 2 , , p K ) amounts to estimating (N1, …, NK). Let (x1, x2, …, xN), xi {ω1, …, ωK}, be a particular N-trial realization sequence (microstate) for the experiment, with associated macrostate (counts) ( N 1 , N 2 , , N K ) that agrees with the given constraint information. Suppose all such microstates are a priori equally likely. Then the probability of macrostate (N1, N2, …, NK) is:
P ( N 1 , , N K ) = 1 K N ( N 1 , , N K N ) ,
where the multinomial ( N 1 , , N K N ) is the number of distinct microstates that are consistent with the (constraint-achieving) macrostate. Since, for any given realization sequence, with macrostate (N1, …, NK), we would form the probability estimate as p ¯ = ( N 1 N , , N K N ), P (N1, …, NK) is also the probability that we form the probability assignment ( N 1 N , , N K N ). Thus, if we choose (N1, …, NK) to maximize (2), we are determining the probability mass function p 1 , p 2 , , p K = ( N 1 N , , N K N ) that we are most likely to produce as an estimate, given the specified constraint information and the number of die tosses. To maximize (2), one maximizes the multinomial coefficient ( N 1 , , N K N ). Based on Stirling’s approximation [4], ( N 1 , , N K N ) ~ e N H ( N 1 N , , N K N ), with H(·) Stirling’s entropy function. Accordingly, one can closely approximate maximizing (2) subject to, e.g., a mean value constraint i = 1 K i N i = μ by maximizing Shannon’s entropy function H ( N 1 N , , N K N ) subject to the constraint. Allowing unconstrained probabilities, rather than fractions of N, this amounts to solving:
max p 1 , p 2 , , p K i = 1 K p i log p i
subject to i = 1 K i p i = μ and i = 1 K p i = 1.
Note, too, that since (2) or its approximate
P ( p 1 , , p K ) = e N H ( p 1 , , p K ) K N
is the probability of forming the estimate (p1, …, pK), we can use (4) to evaluate the relative likelihoods of producing different candidate probability assignments, P ( p 1 , , p K ) P ( p 1 , , p K ) = e N ( H ( p 1 , , p K ) H ( p 1 , , p K ) ) Thus, the likelihood of the maximum entropy solution, relative to an alternative distribution, grows exponentially with the entropy difference.
In [1], they acknowledged this statistically-based justification for the maximum entropy distribution, as applied to the 6-sided Brandeis die problem. Moreover, they motivated the 3-sided die problem by stating that “suppose a friend later tossed the die many times…”. Thus, their 3-sided die problem genuinely does consider the scenario where the constraint information was accurately measured, based on many repeated die tosses. Accordingly, the above interpretation should be applicable to their 3-sided die problem, just as it is to the Brandeis die problem. For the 3-sided die problem, we have P ( 1 3 , 1 3 , 1 3 ) P ( 1 4 , 1 2 , 1 4 ) = 2 N ( log 2 ( 3 ) 1.5 ) ~ 2 0.08 N. For example, if N = 1000, the ME distribution is more than 1024 times more likely to be produced as the estimate than the proposed distribution from [1]. The only real assumption in this analysis is that realization sequences (microstates) consistent with a given macrostate are equally likely.
In [1], they also stated that “since there is no reason to believe the die is fair, we could consider all probability distributions that satisfy the constraints in the problem equally probable. That is, we apply the principle of indifference to the probability values themselves”. While it is true that a priori there is no reason to believe the die is fair, there is also no evidence to support that all probability distributions are equally probable given that our only knowledge about the die is the mean value constraint. Applying a principle of indifference in this way, by [1], imposes a further (rather strong and unjustified) constraint which, as shown above, leads to a solution 1024 times less probable than the solution built on only the available evidence. The “least presumptuous” strategy should only consider the mean value constraint and not impose further constraints. Nevertheless, a different distribution than the ME solution may be achieved if the principle of indifference is applied correctly; i.e., if it is supported by some evidence or by available knowledge about the distribution for a particular application.

2.2. Asymptotic Equipartition Principle (AEP) Interpretation

Another related albeit somewhat different vantage point from which to evaluate the two candidate distributions ( 1 3 , 1 3 , 1 3 ) and ( 1 4 , 1 2 , 1 4 ) is with respect to the asymptotic equipartition principle (AEP) [4]. The AEP considers N independent trials of random draws from a given distribution (p1, p2, …, pK). It defines the ϵ-typical set A ϵ ( N ) as the set of sequences (x1, …, xN) such that 2−N(H(X)+ϵ)P (x1, x2, …, xN) 2−N(H(X)−ϵ). One can show the following [4]:
  • The cardinality of this set is approximately 2NH(p1, p2, …, pK).
  • Each sequence in this set is approximately equally likely.
  • The typical set accounts for nearly all the probability, i.e., the joint pmf P (x1, x2, …, xN) is very nearly approximated as a uniform distribution on all typical sequences, with a zero probability assignment on all non-typical sequences.
Note that not all sequences that are typical will precisely achieve the mean value constraint. However, their deviation from the constraint is made arbitrarily small as N gets large. From the perspective of the AEP, according to the ( 1 4 , 1 2 , 1 4 ) model, there are roughly 21.5N equally likely realizations (each with self-information very close to ( 1 4 , 1 2 , 1 4 ). On the other hand, according to the ME distribution ( 1 3 , 1 3 , 1 3 ), there are 2 N log 2 ( 3 ) equally likely realizations (typical sequences). Note that the sequences that are typical for ( 1 4 , 1 2 , 1 4 ) are not typical for ( 1 3 , 1 3 , 1 3 ), and vice versa. Thus, in choosing between the distributions, one is either rejecting 21.5N realizations typical of ( 1 4 , 1 2 , 1 4 ) or 2 N log 2 ( 3 ) realizations typical of ( 1 3 , 1 3 , 1 3 ). Again, for N = 1000, choosing ( 1 4 , 1 2 , 1 4 ) means rejecting more than 1024 times the number of realizations than if one chooses ( 1 3 , 1 3 , 1 3 ). In the absence of additional information, it seems one should reject the hypothesis that contains (far) fewer realizations that are (approximately) consistent with the measured constraints. Again, from this vantage, the ME solution is preferred.

2.3. The Bayesian Learning Analysis from [1]

The authors in [1] sought to independently validate the ( 1 4 , 1 2 , 1 4 ) distribution by considering a Bayesian learning setting, starting from an uninformative Dirichlet prior on the probabilities. Here, they imposed the mean value constraint, fixed N1 = N3, and assumed all (N1, N2) configurations consistent with satisfying the mean value constraint to be equally likely. They then let N → ∞ and evaluated the conditional probability of die value 2, which they found to be 0.5. While this analysis does appear to support the proposed ( 1 4 , 1 2 , 1 4 ) distribution, a concern with this analysis is the consistency between letting N → ∞ and the assumption that all (N1, N2) configurations meeting the constraint are equally likely. Specifically, by the weak law of large numbers, as N , N 1 N P 1 and N 2 N P 2 with probability 1, i.e., the assumption of equally likely macrostate pairs appears to be incompatible with letting N → ∞.

2.4. Encoding Additional Constraints

Objectively, it is of course possible that the true distribution (used to randomly generate N realizations and achieve a mean value of 2) is the ( 1 4 , 1 2 , 1 4 ) distribution. This would be revealed within the ME framework if we imposed one additional constraint. Specifically, if, in addition to the mean value constraint, we imposed the second moment constraint, i = 1 3 i 2 p i = 4.5 the (unique) ME distribution is indeed the ( 1 4 , 1 2 , 1 4 ) distribution. Thus, ME is not incompatible with the ( 1 4 , 1 2 , 1 4 ) distribution. It is simply producing the “best” distribution given whatever accurate constraint information is made available.

3. Open Problems for Maximum Entropy

While we believe (as exposited here) that ME is an objectively supported method for estimating probabilities given supplied constraints, one open problem is perhaps how to properly account for inaccuracies that may exist in the measured constraints. Constraints in the ME framework are usually in the form of expectations, but since in practice the true values of expectations are not known, they are often estimated by sample averages from (observed) data. For example, in the problem discussed in this letter, μ = E[X] is replaced by its sample-based estimate μ ^ = i i N i N. That is, expectations taken with respect to the ME distribution are constrained to agree with empirical averages, rather than with the true expectations. This may cause overfitting if the sample size is not large enough and, thus, if the expectations are poorly estimated. [5] proposed a framework to address this problem by relaxing strict constraint satisfaction. This approach may be problematic for applications where multiple constraints of different orders are imposed, with each constraint estimated based on a different sample size. For example, for a die experiment, the mean constraint is estimated based on the full set of N trials, whereas a conditional probability constraint such as P (X = 6|X ≥ 3) would be based on a smaller sample size. Consider also [6], where joint probability constraints, from pairwise probabilities up to fifth order, were encoded, in learning ME conditional probability models. It appears that a principled framework, properly accounting for varying degrees of constraint inaccuracy (based, e.g., on different sample sizes used to measure the constraints) is still needed.
Another open problem for maximum entropy is to determine a systematic procedure to search over all possible constraints and impose the most relevant ones for any particular problem. Especially for high-dimensional problems and domains where there is a lack of expert knowledge of the most suitable constraints, we need to objectively determine the relevant constraints to impose. There have been some efforts to address this issue. For instance, [7] described an iterative procedure to decide which constraints to impose, applied to a natural language processing task. Also, [6] used the Kullback distance and the Bayesian Information Criterion to choose relevant constraints and applied this approach to the analysis of genome-wide association study. Nevertheless, it may be fruitful to further investigate alternative approaches for this problem.

4. Conclusions

In this letter, we elaborated on some of the issues raised by a recent paper [1] concerning the maximum entropy (ME) principle and alternative principles for estimating probabilities consistent with known, measured constraint information. We have argued that the ME solution for the “problematic” example introduced in [1] has stronger objective basis than their alternative proposed solution. We also noted some open problems involving maximum entropy statistical inference.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Neapolitan, R.E.; Jiang, X. A note of caution on maximizing entropy. Entropy 2014, 16, 4004–4014. [Google Scholar]
  2. Jaynes, E. T. Papers on Probability, Statistics and Statistical Physics; Rosenkrantz, R.D., Ed.; D. Reidel: Dordrecht, The Netherlands, 1982. [Google Scholar]
  3. Rotar, V. Probability and Stochastic Modeling; CRC Press: Boca Raton, FL, USA, 2013. [Google Scholar]
  4. Cover, T.M.; Thomas, J. Elements of Information Theory; Wiley: New York, NY, USA, 2006. [Google Scholar]
  5. Dudik, M.; Phillips, S.J.; Schapire, R.E. Performance guarantees for regularized maximum entropy density estimation, Proceedings of the 17th Annual Conference on Computational Learning Theory, Banff, Canada, 1–4 July 2004; pp. 472–486.
  6. Miller, D.J.; Zhang, Y.; Yu, G.; Liu, Y.; Chen, L.; Langefeld, C.D.; Herrington, D.; Wang, Y. An algorithm for learning maximum entropy probability models of disease risk that efficiently searches and sparingly encodes multilocus genomic interactions. Bioinformatics 2009, 25, 2478–2485. [Google Scholar]
  7. Berger, A.L.; Pietra, V.J.D.; Pietra, S.A.D. A maximum entropy approach to natural language processing. Comput. Linguist 1996, 22, 39–71. [Google Scholar]

Share and Cite

MDPI and ACS Style

Miller, D.J.; Soleimani, H. On an Objective Basis for the Maximum Entropy Principle. Entropy 2015, 17, 401-406. https://0-doi-org.brum.beds.ac.uk/10.3390/e17010401

AMA Style

Miller DJ, Soleimani H. On an Objective Basis for the Maximum Entropy Principle. Entropy. 2015; 17(1):401-406. https://0-doi-org.brum.beds.ac.uk/10.3390/e17010401

Chicago/Turabian Style

Miller, David J., and Hossein Soleimani. 2015. "On an Objective Basis for the Maximum Entropy Principle" Entropy 17, no. 1: 401-406. https://0-doi-org.brum.beds.ac.uk/10.3390/e17010401

Article Metrics

Back to TopTop