Nonparametric Estimation of Entropy and Mutual Information
A special issue of Entropy (ISSN 1099-4300). This special issue belongs to the section "Information Theory, Probability and Statistics".
Deadline for manuscript submissions: closed (28 April 2022) | Viewed by 3854
Special Issue Editor
Interests: natural language processing; machine learning; information theory; representation learning; information extraction; unsupervised learning; deep learning; spectral methods
Special Issue Information
Dear Colleagues,
Entropy and mutual information are fundamental quantities in information theory with broad application. One prominent recent application is the use of mutual information as a loss function in training deep neural networks. Here, the model has sampling access to an unknown population distribution over a pair of observed variables (e.g., related images or passages) and maximizes the mutual information between their encodings. Since the population distribution is unknown, this requires nonparametric estimation. This has motivated a surge of recent works on developing effective variational estimators of lower (or upper) bounds on mutual information. At the same time, formal limitations on certain approaches have been established such as the impossibility of high-confidence, distribution-free lower bounds on entropy (and therefore mutual information) larger than the log of the sample size.
In this Special Issue, we invite contributions on improving, or better understanding the limits of, nonparametric estimation of entropy and mutual information. We welcome unpublished original papers and comprehensive reviews on a wide spectrum of relevant topics, such as (but are not limited to) developing new nonparametric estimators, analyzing new theoretical properties of an existing estimator, drawing new connections between training objectives for neural networks and mutual information, and novel applications and experimental designs on nonparametric estimation.
Dr. Karl Stratos
Guest Editor
Manuscript Submission Information
Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.
Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Entropy is an international peer-reviewed open access monthly journal published by MDPI.
Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 2600 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.
Keywords
- mutual information
- entropy
- nonparametric estimation
- variational models
- deep learning
- optimization
- representation learning
