Submit to Entropy Review for Entropy Propose a Special Issue

Journal Browser

► Journal Browser

Information Transfer in Multilayer/Deep Architectures

Print Special Issue Flyer
Special Issue Editors
Special Issue Information
Published Papers

A special issue of Entropy (ISSN 1099-4300).

Deadline for manuscript submissions: closed (15 July 2020) | Viewed by 39394

Share This Special Issue

Special Issue Editors

Prof. Dr. Vincent Vigneron

E-Mail Website
Guest Editor

Informatique Biologie Intégrative et Systèmes Complexes, Université d'Évry-Val-d'Essonn, Évry, France
Interests: machine learning; blind source separation; image processing

Prof. Dr. Hichem Maaref

E-Mail Website
Guest Editor

Informatique Biologie Intégrative et Systèmes Complexes, Université d'Évry-Val-d'Essonn, Évry, France
Interests: machine learning; image processing; robotic

Special Issue Information

Dear Colleagues,

The renewal of research interest in machine learning came with the emergence of the concept of big data during the late 2000s.
Schematically, families of deep learning networks (DLN) emerged with industrial ambitions, taking advantage of the development of graphics cards (GPUs) to construct prediction models from massive amounts of collected and stored data and substantial means of calculation. It is illusory to want to learn a deep network involving millions of parameters without very large databases. We tend to think that more data lead to more information.
In addition, the core of learning is all but a problem of data representation, not in the ‘data compression’ sense. For instance, in DLN, one representation (input layer) is replaced by a cascade of many representations (hidden layers), which means an increase of information (entropy). However, some questions remain:
How does information spread in these inflationary networks? Is information transform conservative through the DLN? Can information theory quantify the learning capacity of these networks? How do generative models convert information from the observed space to the hidden space?

Foreseen contributions include the following:

- high-dimension feature selection and pattern correlations
- information entropy in large data representation
- information gain in decision trees
- between layer dependencies
- auto-encoding
- network capacity and information loss
- etc.

This Special Issue has the ambition to collect responses to these questions from the theorical and applicative points of view.

Prof. Vincent Vigneron
Prof. Hichem Maaref
Guest Editors

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Entropy is an international peer-reviewed open access monthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 2600 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

Published Papers (5 papers)

Download All Papers

Research

17 pages, 2265 KiB

Open AccessArticle

A New Pooling Approach Based on Zeckendorf’s Theorem for Texture Transfer Information

by Vincent Vigneron, Hichem Maaref and Tahir Q. Syed

Entropy 2021, 23(3), 279; https://0-doi-org.brum.beds.ac.uk/10.3390/e23030279 - 25 Feb 2021

Cited by 3 | Viewed by 1728

Abstract

The pooling layer is at the heart of every convolutional neural network (CNN) contributing to the invariance of data variation. This paper proposes a pooling method based on Zeckendorf’s number series. The maximum pooling layers are replaced with Z pooling layer, which capture texels from input images, convolution layers, etc. It is shown that Z pooling properties are better adapted to segmentation tasks than other pooling functions. The method was evaluated on a traditional image segmentation task and on a dense labeling task carried out with a series of deep learning architectures in which the usual maximum pooling layers were altered to use the proposed pooling mechanism. Not only does it arbitrarily increase the receptive field in a parameterless fashion but it can better tolerate rotations since the pooling layers are independent of the geometric arrangement or sizes of the image regions. Different combinations of pooling operations produce images capable of emphasizing low/high frequencies, extract ultrametric contours, etc. Full article

(This article belongs to the Special Issue Information Transfer in Multilayer/Deep Architectures)

► Show Figures

Figure 1

17 pages, 3630 KiB

Open AccessArticle

On the Use of Concentrated Time–Frequency Representations as Input to a Deep Convolutional Neural Network: Application to Non Intrusive Load Monitoring

by Sarra Houidi, Dominique Fourer and François Auger

Entropy 2020, 22(9), 911; https://0-doi-org.brum.beds.ac.uk/10.3390/e22090911 - 19 Aug 2020

Cited by 24 | Viewed by 2892

Abstract

Since decades past, time–frequency (TF) analysis has demonstrated its capability to efficiently handle non-stationary multi-component signals which are ubiquitous in a large number of applications. TF analysis us allows to estimate physics-related meaningful parameters (e.g.,

F_{0}

, group delay, etc.) and can [...] Read more.

F_{0}

, group delay, etc.) and can provide sparse signal representations when a suitable tuning of the method parameters is used. On another hand, deep learning with Convolutional Neural Networks (CNN) is the current state-of-the-art approach for pattern recognition and allows us to automatically extract relevant signal features despite the fact that the trained models can suffer from a lack of interpretability. Hence, this paper proposes to combine together these two approaches to take benefit of their respective advantages and addresses non-intrusive load monitoring (NILM) which consists of identifying a home electrical appliance (HEA) from its measured energy consumption signal as a “toy” problem. This study investigates the role of the TF representation when synchrosqueezed or not, used as the input of a 2D CNN applied to a pattern recognition task. We also propose a solution for interpreting the information conveyed by the trained CNN through different neural architecture by establishing a link with our previously proposed “handcrafted” interpretable features thanks to the layer-wise relevant propagation (LRP) method. Our experiments on the publicly available PLAID dataset show excellent appliance recognition results (accuracy above 97%) using the suitable TF representation and allow an interpretation of the trained model. Full article

(This article belongs to the Special Issue Information Transfer in Multilayer/Deep Architectures)

► Show Figures

Figure 1

22 pages, 2072 KiB

Open AccessEditor’s ChoiceArticle

Deep Learning for Stock Market Prediction

by M. Nabipour, P. Nayyeri, H. Jabani, A. Mosavi, E. Salwana and Shahab S.

Entropy 2020, 22(8), 840; https://0-doi-org.brum.beds.ac.uk/10.3390/e22080840 - 30 Jul 2020

Cited by 185 | Viewed by 27785

Abstract

The prediction of stock groups values has always been attractive and challenging for shareholders due to its inherent dynamics, non-linearity, and complex nature. This paper concentrates on the future prediction of stock market groups. Four groups named diversified financials, petroleum, non-metallic minerals, and basic metals from Tehran stock exchange were chosen for experimental evaluations. Data were collected for the groups based on 10 years of historical records. The value predictions are created for 1, 2, 5, 10, 15, 20, and 30 days in advance. Various machine learning algorithms were utilized for prediction of future values of stock market groups. We employed decision tree, bagging, random forest, adaptive boosting (Adaboost), gradient boosting, and eXtreme gradient boosting (XGBoost), and artificial neural networks (ANN), recurrent neural network (RNN) and long short-term memory (LSTM). Ten technical indicators were selected as the inputs into each of the prediction models. Finally, the results of the predictions were presented for each technique based on four metrics. Among all algorithms used in this paper, LSTM shows more accurate results with the highest model fitting ability. In addition, for tree-based models, there is often an intense competition between Adaboost, Gradient Boosting, and XGBoost. Full article

(This article belongs to the Special Issue Information Transfer in Multilayer/Deep Architectures)

► Show Figures

Figure 1

16 pages, 1514 KiB

Open AccessArticle

Relative Distribution Entropy Loss Function in CNN Image Retrieval

by Pingping Liu, Lida Shi, Zhuang Miao, Baixin Jin and Qiuzhan Zhou

Entropy 2020, 22(3), 321; https://0-doi-org.brum.beds.ac.uk/10.3390/e22030321 - 11 Mar 2020

Cited by 5 | Viewed by 3105

Abstract

Convolutional neural networks (CNN) is the most mainstream solution in the field of image retrieval. Deep metric learning is introduced into the field of image retrieval, focusing on the construction of pair-based loss function. However, most pair-based loss functions of metric learning merely take common vector similarity (such as Euclidean distance) of the final image descriptors into consideration, while neglecting other distribution characters of these descriptors. In this work, we propose relative distribution entropy (RDE) to describe the internal distribution attributes of image descriptors. We combine relative distribution entropy with the Euclidean distance to obtain the relative distribution entropy weighted distance (RDE-distance). Moreover, the RDE-distance is fused with the contrastive loss and triplet loss to build the relative distributed entropy loss functions. The experimental results demonstrate that our method attains the state-of-the-art performance on most image retrieval benchmarks. Full article

(This article belongs to the Special Issue Information Transfer in Multilayer/Deep Architectures)

► Show Figures

Figure 1

27 pages, 6642 KiB

Open AccessArticle

Emergence of Network Motifs in Deep Neural Networks

by Matteo Zambra, Amos Maritan and Alberto Testolin

Entropy 2020, 22(2), 204; https://0-doi-org.brum.beds.ac.uk/10.3390/e22020204 - 11 Feb 2020

Cited by 7 | Viewed by 3255

Abstract

Network science can offer fundamental insights into the structural and functional properties of complex systems. For example, it is widely known that neuronal circuits tend to organize into basic functional topological modules, called network motifs. In this article, we show that network science tools can be successfully applied also to the study of artificial neural networks operating according to self-organizing (learning) principles. In particular, we study the emergence of network motifs in multi-layer perceptrons, whose initial connectivity is defined as a stack of fully-connected, bipartite graphs. Simulations show that the final network topology is shaped by learning dynamics, but can be strongly biased by choosing appropriate weight initialization schemes. Overall, our results suggest that non-trivial initialization strategies can make learning more effective by promoting the development of useful network motifs, which are often surprisingly consistent with those observed in general transduction networks. Full article

(This article belongs to the Special Issue Information Transfer in Multilayer/Deep Architectures)

► Show Figures

Figure 1

Journal Menu

Journal Browser

Information Transfer in Multilayer/Deep Architectures

Share This Special Issue

Special Issue Editors

Special Issue Information

Published Papers (5 papers)

Research

Further Information

Guidelines

MDPI Initiatives

Follow MDPI