entropy-logo

Journal Browser

Journal Browser

Maximum Entropy and Bayes Theorem

A special issue of Entropy (ISSN 1099-4300).

Deadline for manuscript submissions: closed (30 June 2013) | Viewed by 47240

Special Issue Editor

Mathematics and Computer Science, Clarkson University, Potsdam, NY 13699, USA
Interests: Bayesian inference and maximum entropy methods; data fusion; multi-scale dynamical systems; machine learning; probability theory
Special Issues, Collections and Topics in MDPI journals

Special Issue Information

Dear Colleagues,

The probabilistic version of entropy originated with Boltzmann. Much later, Shannon (1948) created his information “entropy” using an axiomatic approach. Finally it was Jaynes (1957) who put the two together and showed that by maximizing Shannon’s entropy, along with specific information from physics  in the form of expectation values, one gets Boltzmann’s and more importantly Gibbs’ statistical mechanics.

Before Shannon, Jeffreys (1939) revitalized the Bayesian (Laplacian) view of probabilities (among others) and examined the ratio of a posterior versus a prior, specifically the log of this ratio. He used this ratio as a kind of measure to compare distributions with a focus on finding an invariant prior. This concept was expanded by others such as Kullback and Leibler (1951), Shore and Johnson (1980), Skilling (1988) and Caticha and Giffin (2006) by using the mean of this ratio as a measure to examine the “distance” between distributions. This mean has a form similar to Shannon’s entropy and is sometimes called the cross entropy or relative entropy.

For the purposes of inference, the goal of both Bayes Theorem and Maximum Entropy is to determine a probability distribution based on certain information. However, even though the two are indubitably linked, the use of each is controversial, particularly Maximum Entropy. In this volume we seek contributions that will shed light on their use, their connections and their controversies. The scope of the contributions will be very broad to include philosophical discussions, theoretical derivations and explicit, practical solutions to current problems. Such problems may come from, but by no means are limited to, topics from fields such as data mining, dynamical systems, ecology, economics, image analysis and computer vision, machine learning, signal analysis, physics and other scientific areas of inquiry. Papers demonstrating that Bayesian or Maximum Entropy methods yielding unique solutions that cannot be produced otherwise are especially welcome.

Dr. Adom Giffin
Guest Editor

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Entropy is an international peer-reviewed open access monthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 2600 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

Keywords

  • cross entropy
  • relative entropy
  • information theory
  • inference
  • maximum entropy
  • maximum relative entropy
  • Bayes theorem
  • Bayes rule
  • Bayesian
  • machine learning
  • data mining
  • image analysis
  • computer vision
  • machine learning
  • signal analysis
  • data fusion
  • multi-scale dynamical systems
  • distributed computing
  • probability theory

Published Papers (7 papers)

Order results
Result details
Select all
Export citation of selected articles as:

Research

Jump to: Review

387 KiB  
Article
The Kalman Filter Revisited Using Maximum Relative Entropy
by Adom Giffin and Renaldas Urniezius
Entropy 2014, 16(2), 1047-1069; https://0-doi-org.brum.beds.ac.uk/10.3390/e16021047 - 19 Feb 2014
Cited by 17 | Viewed by 9521
Abstract
In 1960, Rudolf E. Kalman created what is known as the Kalman filter, which is a way to estimate unknown variables from noisy measurements. The algorithm follows the logic that if the previous state of the system is known, it could be used [...] Read more.
In 1960, Rudolf E. Kalman created what is known as the Kalman filter, which is a way to estimate unknown variables from noisy measurements. The algorithm follows the logic that if the previous state of the system is known, it could be used as the best guess for the current state. This information is first applied a priori to any measurement by using it in the underlying dynamics of the system. Second, measurements of the unknown variables are taken. These two pieces of information are taken into account to determine the current state of the system. Bayesian inference is specifically designed to accommodate the problem of updating what we think of the world based on partial or uncertain information. In this paper, we present a derivation of the general Bayesian filter, then adapt it for Markov systems. A simple example is shown for pedagogical purposes. We also show that by using the Kalman assumptions or “constraints”, we can arrive at the Kalman filter using the method of maximum (relative) entropy (MrE), which goes beyond Bayesian methods. Finally, we derive a generalized, nonlinear filter using MrE, where the original Kalman Filter is a special case. We further show that the variable relationship can be any function, and thus, approximations, such as the extended Kalman filter, the unscented Kalman filter and other Kalman variants are special cases as well. Full article
(This article belongs to the Special Issue Maximum Entropy and Bayes Theorem)
318 KiB  
Article
Consistency and Generalization Bounds for Maximum Entropy Density Estimation
by Shaojun Wang, Russell Greiner and Shaomin Wang
Entropy 2013, 15(12), 5439-5463; https://0-doi-org.brum.beds.ac.uk/10.3390/e15125439 - 09 Dec 2013
Cited by 4 | Viewed by 5476
Abstract
We investigate the statistical properties of maximum entropy density estimation, both for the complete data case and the incomplete data case. We show that under certain assumptions, the generalization error can be bounded in terms of the complexity of the underlying feature functions. [...] Read more.
We investigate the statistical properties of maximum entropy density estimation, both for the complete data case and the incomplete data case. We show that under certain assumptions, the generalization error can be bounded in terms of the complexity of the underlying feature functions. This allows us to establish the universal consistency of maximum entropy density estimation. Full article
(This article belongs to the Special Issue Maximum Entropy and Bayes Theorem)
Show Figures

Figure 1

732 KiB  
Article
Objective Bayesianism and the Maximum Entropy Principle
by Jürgen Landes and Jon Williamson
Entropy 2013, 15(9), 3528-3591; https://0-doi-org.brum.beds.ac.uk/10.3390/e15093528 - 04 Sep 2013
Cited by 23 | Viewed by 6387
Abstract
Objective Bayesian epistemology invokes three norms: the strengths of our beliefs should be probabilities; they should be calibrated to our evidence of physical probabilities; and they should otherwise equivocate sufficiently between the basic propositions that we can express. The three norms are sometimes [...] Read more.
Objective Bayesian epistemology invokes three norms: the strengths of our beliefs should be probabilities; they should be calibrated to our evidence of physical probabilities; and they should otherwise equivocate sufficiently between the basic propositions that we can express. The three norms are sometimes explicated by appealing to the maximum entropy principle, which says that a belief function should be a probability function, from all those that are calibrated to evidence, that has maximum entropy. However, the three norms of objective Bayesianism are usually justified in different ways. In this paper, we show that the three norms can all be subsumed under a single justification in terms of minimising worst-case expected loss. This, in turn, is equivalent to maximising a generalised notion of entropy. We suggest that requiring language invariance, in addition to minimising worst-case expected loss, motivates maximisation of standard entropy as opposed to maximisation of other instances of generalised entropy. Our argument also provides a qualified justification for updating degrees of belief by Bayesian conditionalisation. However, conditional probabilities play a less central part in the objective Bayesian account than they do under the subjective view of Bayesianism, leading to a reduced role for Bayes’ Theorem. Full article
(This article belongs to the Special Issue Maximum Entropy and Bayes Theorem)
Show Figures

Figure 1

289 KiB  
Article
Relative Entropy Derivative Bounds
by Pablo Zegers, Alexis Fuentes and Carlos Alarcón
Entropy 2013, 15(7), 2861-2873; https://0-doi-org.brum.beds.ac.uk/10.3390/e15072861 - 23 Jul 2013
Cited by 4 | Viewed by 5760
Abstract
We show that the derivative of the relative entropy with respect to its parameters is lower and upper bounded. We characterize the conditions under which this derivative can reach zero. We use these results to explain when the minimum relative entropy and the [...] Read more.
We show that the derivative of the relative entropy with respect to its parameters is lower and upper bounded. We characterize the conditions under which this derivative can reach zero. We use these results to explain when the minimum relative entropy and the maximum log likelihood approaches can be valid. We show that these approaches naturally activate in the presence of large data sets and that they are inherent properties of any density estimation process involving large numbers of random variables. Full article
(This article belongs to the Special Issue Maximum Entropy and Bayes Theorem)
193 KiB  
Article
Classification of Knee Joint Vibration Signals Using Bivariate Feature Distribution Estimation and Maximal Posterior Probability Decision Criterion
by Yunfeng Wu, Suxian Cai, Shanshan Yang, Fang Zheng and Ning Xiang
Entropy 2013, 15(4), 1375-1387; https://0-doi-org.brum.beds.ac.uk/10.3390/e15041375 - 17 Apr 2013
Cited by 36 | Viewed by 6993
Abstract
Analysis of knee joint vibration or vibroarthrographic (VAG) signals using signal processing and machine learning algorithms possesses high potential for the noninvasive detection of articular cartilage degeneration, which may reduce unnecessary exploratory surgery. Feature representation of knee joint VAG signals helps characterize the [...] Read more.
Analysis of knee joint vibration or vibroarthrographic (VAG) signals using signal processing and machine learning algorithms possesses high potential for the noninvasive detection of articular cartilage degeneration, which may reduce unnecessary exploratory surgery. Feature representation of knee joint VAG signals helps characterize the pathological condition of degenerative articular cartilages in the knee. This paper used the kernel-based probability density estimation method to model the distributions of the VAG signals recorded from healthy subjects and patients with knee joint disorders. The estimated densities of the VAG signals showed explicit distributions of the normal and abnormal signal groups, along with the corresponding contours in the bivariate feature space. The signal classifications were performed by using the Fisher’s linear discriminant analysis, support vector machine with polynomial kernels, and the maximal posterior probability decision criterion. The maximal posterior probability decision criterion was able to provide the total classification accuracy of 86.67% and the area (Az) of 0.9096 under the receiver operating characteristics curve, which were superior to the results obtained by either the Fisher’s linear discriminant analysis (accuracy: 81.33%, Az: 0.8564) or the support vector machine with polynomial kernels (accuracy: 81.33%, Az: 0.8533). Such results demonstrated the merits of the bivariate feature distribution estimation and the superiority of the maximal posterior probability decision criterion for analysis of knee joint VAG signals. Full article
(This article belongs to the Special Issue Maximum Entropy and Bayes Theorem)
Show Figures

Figure 1

540 KiB  
Article
Experimental Assessment of a 2-D Entropy-Based Model for Velocity Distribution in Open Channel Flow
by Nicola Fontana, Gustavo Marini and Francesco De Paola
Entropy 2013, 15(3), 988-998; https://0-doi-org.brum.beds.ac.uk/10.3390/e15030988 - 06 Mar 2013
Cited by 22 | Viewed by 6165
Abstract
Velocity distribution in an open channel flow can be very useful to model many hydraulic phenomena. Among the others, several 1D models based on the concept of entropy are available in the literature, which allow estimating the velocity distribution by measuring velocities only [...] Read more.
Velocity distribution in an open channel flow can be very useful to model many hydraulic phenomena. Among the others, several 1D models based on the concept of entropy are available in the literature, which allow estimating the velocity distribution by measuring velocities only in a few points. Nevertheless, since 1D models have often a limited practical use, a 2D entropy based model was recently developed. The model provides a reliable estimation of the velocity distribution for open channel flow with a rectangular cross section, if the maximum velocity and the average velocity are known. In this paper results from the proposed model were compared with measured velocities carried out from laboratory experiments. Calculated values were also compared with results inferred from a 2D model available in the literature, resulting in a greater ease of use and a more reliable estimate of the velocity profile. Full article
(This article belongs to the Special Issue Maximum Entropy and Bayes Theorem)
Show Figures

Figure 1

Review

Jump to: Research

592 KiB  
Review
Quantum Models of Classical World
by Petr Hájíček
Entropy 2013, 15(3), 789-925; https://0-doi-org.brum.beds.ac.uk/10.3390/e15030789 - 27 Feb 2013
Cited by 2 | Viewed by 4892
Abstract
This paper is a review of our recent work on three notorious problems of non-relativistic quantum mechanics: realist interpretation, quantum theory of classical properties, and the problem of quantum measurement. A considerable progress has been achieved, based on four distinct new ideas. First, [...] Read more.
This paper is a review of our recent work on three notorious problems of non-relativistic quantum mechanics: realist interpretation, quantum theory of classical properties, and the problem of quantum measurement. A considerable progress has been achieved, based on four distinct new ideas. First, objective properties are associated with states rather than with values of observables. Second, all classical properties are selected properties of certain high entropy quantum states of macroscopic systems. Third, registration of a quantum system is strongly disturbed by systems of the same type in the environment. Fourth, detectors must be distinguished from ancillas and the states of registered systems are partially dissipated and lost in the detectors. The paper has two aims: a clear explanation of all new results and a coherent and contradiction-free account of the whole quantum mechanics including all necessary changes of its current textbook version. Full article
(This article belongs to the Special Issue Maximum Entropy and Bayes Theorem)
Back to TopTop