An Information-Theoretic Perspective on Proper Quaternion Variational Autoencoders

Abstract

Variational autoencoders are deep generative models that have recently received a great deal of attention due to their ability to model the latent distribution of any kind of input such as images and audio signals, among others. A novel variational autoncoder in the quaternion domain $\mathbb{H}$, namely the QVAE, has been recently proposed, leveraging the augmented second order statics of $\mathbb{H}$-proper signals. In this paper, we analyze the QVAE under an information-theoretic perspective, studying the ability of the $\mathbb{H}$-proper model to approximate improper distributions as well as the built-in $\mathbb{H}$-proper ones and the loss of entropy due to the improperness of the input signal. We conduct experiments on a substantial set of quaternion signals, for each of which the QVAE shows the ability of modelling the input distribution, while learning the improperness and increasing the entropy of the latent space. The proposed analysis will prove that proper QVAEs can be employed with a good approximation even when the quaternion input data are improper.

1. Introduction

Generative learning models have been recently gained considerable attention due to their surprising performance in producing highly realistic signals of various types [1,2,3,4]. They have been successfully employed in a wide variety of applications, such as image-to-image translation [5], image fusion [6], face de-identification [7], natural language generation [8], data augmentation on ancient handwritten characters [9], MRI super-resolution [10], brain tumor growth prediction [11], generative modeling of structured-data [12].

Among such generative methods, the variational autoencoders (VAEs) have been proven to perform stochastic variational inference and learning even for large datasets and intractable posterior distributions [13]. The main advantages of the VAEs rely on their capability of learning smooth latent representations of the input data. This has led VAEs to be used in several fields of applications, including high-quality image generation [14], speech enhancement [15], music style transfer [16], data augmentation [17], 3D scene generation [18], gesture similarity analysis [19], text generation [20] and sequential recommendation [21], among others.

Recent advances on VAEs focus both on theoretical aspects, such as the improvement of the stochastic inference approach [22], and on architectural aspects, such as the use of different types of latent variables to learn local and global structures [23], the definition of hierarchical schemes [24], rather than the use of a multi-agent generator [25]. Among the most recent VAE models, we focus on the quaternion-valued variational autoencoder (QVAE), which exploits the properties of quaternion algebra to improve performance, on one hand, and to significantly reduce the overall number of network parameters [26], on the other hand.

The basic approach on which relies the QVAE is the learning in the quaternion domain, which results in significant advantages in the presence of multidimensional input data (mainly 3D and 4D data) showing some inter-channel correlations. This properties have been widely exploited in shallow learning models, such as linear and nonlinear adaptive filters [27,28,29,30,31,32,33,34]. Another fundamental property of quaternion-valued learning is the Hamilton product, which has recently favored the proliferation of convolutional neural networks in the quaternion domain [35,36,37,38]. Due to their capabilities, quaternion-valued learning methods have been applied in several applications, including spoken language understanding [39], color image processing [40,41], 3D audio [42,43], speech recognition [44], image generation [45], quantum mechanics [46], risk diversification [47], gait data analysis [48].

As with any other complex-valued and hypercomplex-valued algebra, the quaternion-valued learning can rely on the properties of the second-order statistics, which are fundamental to characterize the input data based on their correlation. A characterization of correlation is achieved in terms of covariance and pseudo-covariance. In particular, random variables and processes with a vanishing pseudo-covariance are called proper [49,50,51]. Properness is preserved under affine transformations. Moreover, it should be considered that the multivariate Gaussian density assumes a natural form only for proper random variables. Properness in the quaternion domain, also denoted as $\mathbb{H}$-properness, is based on the vanishing of three different complementary covariance matrices [51]. In the case of possibly improper random variables, we need to take into account the quaternion conjugate and augmented second-order statistics [51,52,53]. The degree of improperness of a quaternion random vector can be measured in several ways. An interesting approach is to use an improperness measure based on the Kullback–Leibler (KL) divergence between the augmented covariance matrix and its closest proper version in the KL sense [51]. Quaternion second-order statistics have been used in several applications, from independent component analysis to canonical transform [54,55], from linear to nonlinear adaptive filtering [56,57,58].

Recently, augmented quaternion second-order statistics have been also exploited for the first time for the development of a deep learning model, the QVAE [26]. Deep VAEs should model long-range correlations in data [59] and quaternion layers allow to leverage internal latent relations between input features. The QVAE also leads to a significant reduction in the number of parameters, which may benefit the decoder. A full improper QVAE should be developed to completely exploit correlated multidimensional input data, such as color images. However, on one hand an improper QVAE may be too complex due to the sophisticated structure of the complete covariance matrix. On the other hand, the proper QVAEs may represent a good tradeoff between complexity and performance, even when input data are improper. To this end, an information-theoretic analyses, thanks to its relation with improperness measures, may help in understanding how good such approximation may be in terms of properness [60,61].

In order to understand and demystify the theoretical capabilities of generating data with a proper model when the input data is either proper or improper, in this paper we investigate the proper QVAE from an information-theoretic perspective. To this end, we show how the KL divergence may establish a relation between the loss function of the QVAE and the improperness measure. We exploit the KL-based measure to show the entropy loss due to the improperness of the quaternion random vector input of the QVAE, or equivalently, the mutual information among the quaternion random vector and its involutions over three pure unit quaternions [51,52]. We show how the entropy in input data plays an important role in the generation of improper signals. Furthermore, we illustrate how the improperness can be calibrated in the latent space in order to enhance its differential entropy. Simulation results on artificial signals will prove the ability of the proper QVAE to be adopted with a good approximation for the generation of improper signals. Moreover, results will show how the properness is preserved by the model, even if it is composed of several nonlinearities.

The paper is organized as follows. In Section 2, we briefly introduce the main properties of quaternion algebra that are exploited in quaternion-valued processing. In Section 3, we analyze the quaternion properness and the augmented second-order statistics of quaternion-valued signals, which are then exploited to define the quaternion variational autoencoders. In particular, the proper QVAE is described in Section 4, while in Section 5 we define the relation between the loss function of the QVAE and the properness measures of the involved signals by using the quaternion-valued entropy. Experimental simulations are shown in Section 6, while final conclusions are derived in Section 7.

2. Fundamental Properties of Quaternion Algebra

Quaternions are hypercomplex numbers consisting of four components, a scalar part and three imaginary ones, and elements of ${\mathbb{R}}^4$: The quaternion domain $\mathbb{H}$ is an extension of the complex domain $\mathbb{C}$. A quaternion is identified as

$$q=q_0+q_1\widehat{\mathit{ı}}+q_2\mathit{ĵ}+q_3\widehat{\kappa }=q_0+\mathbf{q}$$

(1)

in which $q_0,q_1,q_2,q_3$ are real coefficients and $\widehat{\mathit{ı}},\mathit{ĵ},\widehat{\kappa }$ imaginary units. A quaternion without the scalar part $q_0$ is named pure quaternion and represents a vector in ${\mathbb{R}}^3$ since the imaginary units play the role of orthonormal basis, being $\widehat{\mathit{ı}}=(1,0,0),\mathit{ĵ}=(0,1,0),\widehat{\kappa }=(0,0,1)$. The quaternions’ peculiarity is the relation among imaginary components which comply with the following properties:

$$\begin{array}{c}{\widehat{\mathit{ı}}}^2={\mathit{ĵ}}^2={\widehat{\kappa }}^2=-1\\ \widehat{\mathit{ı}}\mathit{ĵ}=\widehat{\mathit{ı}}\times \mathit{ĵ}=\widehat{\kappa };\mathit{ĵ}\widehat{\kappa }=\mathit{ĵ}\times \widehat{\kappa }=\widehat{\mathit{ı}};\widehat{\kappa }\widehat{\mathit{ı}}=\widehat{\kappa }\times \widehat{\mathit{ı}}=\mathit{ĵ}\\ \widehat{\mathit{ı}}\mathit{ĵ}=-\mathit{ĵ}\widehat{\mathit{ı}};\mathit{ĵ}\widehat{\kappa }=-\widehat{\kappa }\mathit{ĵ};\widehat{\kappa }\widehat{\mathit{ı}}=-\widehat{\mathit{ı}}\widehat{\kappa }.\end{array}$$

(2)

The fundamental properties of quaternion operations are then described below.

Addition and subtraction of two quaternions. Addition and subtraction among quaternions q and p are performed component-wise:

$$q\pm p=q_0\pm p_0+(q_1\pm p_1)\widehat{\mathit{ı}}+(q_2\pm p_2)\mathit{ĵ}+(q_3\pm p_3)\widehat{\kappa }.$$

(3)

Scalar product of two quaternions. As for addition and subtraction, the scalar product is also an element-wise operation:

$$q·p=q_0{p}_0+q_1{p}_1+q_2{p}_2+q_3{p}_3.$$

(4)

Vector product of two quaternions. Due to the relations among imaginary units in (2) and the non commutative properties of vector product in the hypercomplex domain, a new kind of vector multiplication has to be introduced. The Hamilton product, denoted by ⊗, is at the core of quaternion neural networks and it is described by the following equation:

$$\begin{array}{cc}\hfill q\otimes p& =\left(q_0+q_1\widehat{\mathit{ı}}+q_2\mathit{ĵ}+q_3\widehat{\kappa }\right)\left(p_0+p_1\widehat{\mathit{ı}}+p_2\mathit{ĵ}+p_3\widehat{\kappa }\right)\hfill \\ \hfill & =\left(q_0{p}_0-q_1{p}_1-q_2{p}_2-q_3{p}_3\right)\hfill \\ \hfill & +\left(q_0{p}_1+q_1{p}_0+q_2{p}_3-q_3{p}_3\right)\widehat{\mathit{ı}}\hfill \\ \hfill & +\left(q_0{p}_2-q_1{p}_3+q_2{p}_0+q_3{p}_2\right)\mathit{ĵ}\hfill \\ \hfill & +\left(q_0{p}_3+q_1{p}_2-q_2{p}_1+q_3{p}_0\right)\widehat{\kappa }.\hfill \end{array}$$

(5)

The Hamilton product can also be written in concise form:

$$q\otimes p=q_0{p}_0-\mathbf{q}·\mathbf{p}+q_0\mathbf{p}+p_0\mathbf{q}+\mathbf{q}\times \mathbf{p},$$

from which it is straightforward obtaining the Hamilton product for pure quaternions, by fixing $q_0=p_0=0$:

$$\mathbf{q}\otimes \mathbf{p}=-\mathbf{q}·\mathbf{p}+\mathbf{q}\times \mathbf{p}.$$

Conjugate of a quaternion. The conjugate of a quaternion number is obtained by subtracting the imaginary units instead of adding them to the scalar part:

$$q^{*}=q_0-q_1\widehat{\mathit{ı}}-q_2\mathit{ĵ}-q_3\widehat{\kappa }=q_0-\mathbf{q}.$$

(6)

Norm of a quaternion. The norm of a quaternion is simply the euclidean norm in ${\mathbb{R}}^4$, thus $\left|q\right|=\sqrt{q{q}^{*}}=\sqrt{q_0^2+q_1^2+q_2^2+q_3^2}$.

Quaternion polar form. Quaternions admit an Euler representation, so considering a quaternion q, an angle $\theta \in \mathbb{R}$ and a pure unit quaternion $\nu =\mathbf{q}/∥\mathbf{q}∥$, the polar form is:

$$q=\left|q\right|(cos\left(\theta \right)+\nu sin\left(\theta \right))=\left|q\right|e^{\nu \theta }$$

in this setting, $cos\left(\theta \right)=q_0/∥q∥$ and $sin\left(\theta \right)=∥\mathbf{q}∥/∥q∥$.

Quaternion involutions. A quaternion involution is generally defined as a mapping which is its own inverse or self-inverse mapping. Quaternions have infinite involutions that can be generalized by [62]:

$$q^{\nu }=-\nu q\nu$$

(7)

in which q is the quaternion to be involved and $\nu$ is the axis of the involution. Interestingly, the scalar part of the quaternion is invariant to any kind of involution while the imaginary parts are reflected across the axis of the involution. Consequently, the conjugate of a quaternion is an involution since the scalar part remains unchanged while the imaginary units are reflected through the axis. Together with the conjugate, there are three crucial involutions we have to consider, which are perpendicular involutions:

$$\begin{array}{cc}\hfill & q^{\widehat{\mathit{ı}}}=-\widehat{\mathit{ı}}q\widehat{\mathit{ı}}=q_0+q_1\widehat{\mathit{ı}}-q_2\mathit{ĵ}-q_3\widehat{\kappa }\hfill \\ \hfill & q^{\mathit{ĵ}}=-\mathit{ĵ}q\mathit{ĵ}=q_0-q_1\widehat{\mathit{ı}}+q_2\mathit{ĵ}-q_3\widehat{\kappa }\hfill \\ \hfill & q^{\widehat{\kappa }}=-\widehat{\kappa }q\widehat{\kappa }=q_0-q_1\widehat{\mathit{ı}}-q_2\mathit{ĵ}+q_3\widehat{\kappa }\hfill \end{array}$$

(8)

In the perpendicular involutions, the quaternion is involved across its imaginary units.

3. Augmented Second-Order Statistics and Quaternion Properness

3.1. Statistics for Quaternion Random Signals

In order to understand the behavior of a signal, it is often crucial to study its statistics. A generic Gaussian signal can be completely described by the mean and the covariance matrix. For quaternion random variables, the mean can be easily derived by (3), resulting in another quaternion:

$${\mu }_q={\overline{q}}_0+{\overline{q}}_1\widehat{\mathit{ı}}+{\overline{q}}_2\mathit{ĵ}+{\overline{q}}_3\widehat{\kappa }$$

(9)

in which ${\overline{q}}_{\delta },\delta =\{0,1,2,3\}$ is the average of each quaternion component. Conversely, deriving second-order statistics for quaternion signals requires more detailed computations due to the interactions among components. As previously demonstrated for complex numbers [49], the covariance matrix is not able to describe the second-order information and the complementary covariance matrix needs to be considered too. Consequently, taking the standard covariance matrix of a quaternion signal into account is not sufficient to cover the complete second-order statistics; thus, the information is augmented through three complementary covariance matrices [52]. In order to build this complete statistical description, we need to introduce the information brought by the perpendicular involutions ${\mathbf{q}}^{\widehat{\mathit{ı}}},{\mathbf{q}}^{\mathit{ĵ}},{\mathbf{q}}^{\widehat{\kappa }}$ introduced in (8). The matrices to be considered are then:

$$\begin{array}{c}{\mathbf{C}}_{\mathbf{q}\mathbf{q}}=E\left\{\mathbf{q}{\mathbf{q}}^{\mathrm{H}}\right\},{\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\widehat{\mathit{ı}}}}=E\left\{\mathbf{q}{\mathbf{q}}^{{\widehat{\mathit{ı}}}^{\mathrm{H}}}\right\},{\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\mathit{ĵ}}}=E\left\{\mathbf{q}{\mathbf{q}}^{{\mathit{ĵ}}^{\mathrm{H}}}\right\},{\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\widehat{\kappa }}}=E\left\{\mathbf{q}{\mathbf{q}}^{{\widehat{\kappa }}^{\mathrm{H}}}\right\}\end{array}$$

(10)

which are, respectively, the standard covariance matrix and the three complementary matrices. Note that ${(·)}^{\mathrm{H}}$ is the conjugate transpose operator. These matrices are still quaternions, so they comprise a real part and three imaginary units. As an example, we analyze the first matrix ${\mathbf{C}}_{\mathbf{q}\mathbf{q}}$. It is composed by the covariance matrices of the quaternion components ${\mathbf{q}}_0,{\mathbf{q}}_1,{\mathbf{q}}_2,{\mathbf{q}}_3$ as follows:

$$\begin{array}{cc}\hfill {\mathbf{C}}_{\mathbf{q}\mathbf{q}}& =\left({\mathbf{C}}_{{\mathbf{q}}_0}+{\mathbf{C}}_{{\mathbf{q}}_1}+{\mathbf{C}}_{{\mathbf{q}}_2}+{\mathbf{C}}_{{\mathbf{q}}_3}\right)\hfill \\ \hfill & +\left({\mathbf{C}}_{{\mathbf{q}}_1{\mathbf{q}}_0}-{\mathbf{C}}_{{\mathbf{q}}_0{\mathbf{q}}_1}+{\mathbf{C}}_{{\mathbf{q}}_3{\mathbf{q}}_2}-{\mathbf{C}}_{{\mathbf{q}}_2{\mathbf{q}}_3}\right)\widehat{\mathit{ı}}\hfill \\ \hfill & +\left({\mathbf{C}}_{{\mathbf{q}}_2{\mathbf{q}}_0}-{\mathbf{C}}_{{\mathbf{q}}_0{\mathbf{q}}_2}+{\mathbf{C}}_{{\mathbf{q}}_1{\mathbf{q}}_3}-{\mathbf{C}}_{{\mathbf{q}}_3{\mathbf{q}}_1}\right)\mathit{ĵ}\hfill \\ \hfill & +\left({\mathbf{C}}_{{\mathbf{q}}_3{\mathbf{q}}_0}-{\mathbf{C}}_{{\mathbf{q}}_0{\mathbf{q}}_3}+{\mathbf{C}}_{{\mathbf{q}}_2{\mathbf{q}}_1}-{\mathbf{C}}_{{\mathbf{q}}_1{\mathbf{q}}_2}\right)\widehat{\kappa }.\hfill \end{array}$$

(11)

Similarly, also the matrices ${\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\widehat{\mathit{ı}}}},{\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\mathit{ĵ}}},{\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\widehat{\kappa }}}$ can be determined and written in the above quaternion form.

The composition and the combination of the standard covariance matrix and of the complementary covariance matrices give rise to the augmented covariance matrix ${\tilde{\mathbf{C}}}_{\mathbf{q}\mathbf{q}}$ which recovers the complete second-order information of the augmented quaternion vector $\tilde{\mathbf{q}}=\left[\mathbf{q},{\mathbf{q}}^{\widehat{\mathit{ı}}},{\mathbf{q}}^{\mathit{ĵ}},{\mathbf{q}}^{\widehat{\kappa }}\right]$[52]. The matrices in (10) have interesting properties: each component of the matrix is symmetric but the component corresponding to the involved imaginary unit is, instead, skew-symmetric. These characteristics are included in the Hermitian property, according to which ${\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\widehat{\mathit{ı}}}}$ is $\widehat{\mathit{ı}}$-Hermitian, ${\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\mathit{ĵ}}}$ is $\mathit{ĵ}$-Hermitian and ${\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\widehat{\kappa }}}$ is $\widehat{\kappa }$-Hermitian. The derived augmented covariance matrix follows.

$${\tilde{\mathbf{C}}}_{\mathbf{q}\mathbf{q}}=E\left\{\tilde{\mathbf{q}}{\tilde{\mathbf{q}}}^{\mathrm{H}}\right\}=\left[\begin{array}{cccc}{\mathbf{C}}_{\mathbf{q}\mathbf{q}}& {\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\widehat{\mathit{ı}}}}& {\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\mathit{ĵ}}}& {\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\widehat{\kappa }}}\\ {\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\widehat{\mathit{ı}}}}^{\mathrm{H}}& {\mathbf{C}}_{{\mathbf{q}}^{\widehat{\mathit{ı}}}{\mathbf{q}}^{\widehat{\mathit{ı}}}}& {\mathbf{C}}_{{\mathbf{q}}^{\widehat{\mathit{ı}}}{\mathbf{q}}^{\mathit{ĵ}}}& {\mathbf{C}}_{{\mathbf{q}}^{\widehat{\mathit{ı}}}{\mathbf{q}}^{\widehat{\kappa }}}\\ {\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\mathit{ĵ}}}^{\mathrm{H}}& {\mathbf{C}}_{{\mathbf{q}}^{\mathit{ĵ}}{\mathbf{q}}^{\widehat{\mathit{ı}}}}& {\mathbf{C}}_{{\mathbf{q}}^{\mathit{ĵ}}{\mathbf{q}}^{\mathit{ĵ}}}& {\mathbf{C}}_{{\mathbf{q}}^{\mathit{ĵ}}{\mathbf{q}}^{\widehat{\kappa }}}\\ {\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\widehat{\kappa }}}^{\mathrm{H}}& {\mathbf{C}}_{{\mathbf{q}}^{\widehat{\kappa }}{\mathbf{q}}^{\widehat{\mathit{ı}}}}& {\mathbf{C}}_{{\mathbf{q}}^{\widehat{\kappa }}{\mathbf{q}}^{\mathit{ĵ}}}& {\mathbf{C}}_{{\mathbf{q}}^{\widehat{\kappa }}{\mathbf{q}}^{\widehat{\kappa }}}\end{array}\right].$$

(12)

By means of the mean and augmented covariance matrix and following [52], it is now possible to define a generic quaternion Gaussian distribution $p\left(\tilde{\mathbf{q}}\right)=p\left(\mathbf{q},{\mathbf{q}}^{\widehat{\mathit{ı}}},{\mathbf{q}}^{\mathit{ĵ}},{\mathbf{q}}^{\widehat{\kappa }}\right)$, which is Gaussian if its components are jointly normal:

$$p\left(\tilde{\mathbf{q}}\right)=\frac{1}{{\left(\pi /2\right)}^{2N}det{\left({\tilde{\mathbf{C}}}_{\mathbf{q}\mathbf{q}}\right)}^{1/2}}exp\left\{-\frac{1}{2}{\left(\tilde{\mathbf{q}}-{\tilde{\mu }}_q\right)}^{\mathrm{H}}{\tilde{\mathbf{C}}}_{\mathbf{q}\mathbf{q}}^{-1}\left(\tilde{\mathbf{q}}-{\tilde{\mu }}_q\right)\right\},$$

(13)

where the mean ${\tilde{\mu }}_q$ is the mean of the augmented quaternion $\tilde{\mathbf{q}}$ and N the number of samples.

3.2. $\mathbb{H}$-Properness

In the previous subsection, we describe the complete second-order information of a quaternion signal by means of the augmented covariance matrix which is comprised of the covariance matrices of the quaternion with its perpendicular involutions. Due to the four degrees of freedom in quaternions, the $\mathbb{H}$ domain includes manifold properness definitions that are employed to characterize the relation of $\mathbf{q}$ with ${\mathbf{q}}^{\widehat{\mathit{ı}}},{\mathbf{q}}^{\mathit{ĵ}},{\mathbf{q}}^{\widehat{\kappa }}$ in some particular cases.

The lighter condition is described in the $\mathbb{R}$-properness, which holds for a signal that is uncorrelated with one of its perpendicular involutions, leading to the cancellation of either one of the complementary covariance submatrices ${\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\widehat{\mathit{ı}}}},{\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\mathit{ĵ}}},{\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\widehat{\kappa }}}$. A slightly harder constraint is brought by the $\mathbb{C}$-properness which claims the nullification of two of the three complementary covariance submatrices. A composition of the two already defined properties gives rise to the strongest properness condition, the $\mathbb{H}$-properness. The latter holds when each complementary covariance matrix vanishes, that is ${\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\widehat{\mathit{ı}}}}={\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\mathit{ĵ}}}={\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\widehat{\kappa }}}=\mathbf{0}$. However, according to [52], an $\mathbb{H}$-proper variable has also several other features that are reported in

Table 1. Fundamental properties of $\mathbb{H}$-proper random variables.

$E\left\{{\mathbf{q}}_{\delta }^2\right\}=E\left\{{\mathbf{q}}_{ϵ}^2\right\}={\sigma }^2$	$\forall \delta ,ϵ=\left\{0,1,2,3\right\}$
$E\left\{{\mathbf{q}}_{\delta }{\mathbf{q}}_{ϵ}\right\}=\mathbf{0}$	$\forall \delta ,ϵ=\left\{0,1,2,3\right\}$, $\delta \ne ϵ$
$E\left\{\mathbf{q}\mathbf{q}\right\}=-2{\sigma }^2$	$\forall \delta =\left\{0,1,2,3\right\}$
$E\left\{{\left|\mathbf{q}\right|}^2\right\}=4{\sigma }^2$	$\forall \delta =\left\{0,1,2,3\right\}$

. An $\mathbb{H}$-proper signal is then uncorrelated with its perpendicular involutions and, according to the second property in Table 1, it also has uncorrelated components ${\mathbf{q}}_0,{\mathbf{q}}_1,{\mathbf{q}}_2{\mathbf{q}}_3$ which have equal variance ${\sigma }^2$. From these assumptions, we can derive the corresponding augmented covariance matrix for $\mathbb{H}$-proper signals which will be symmetric and positive definite:

$${\widehat{\mathbf{C}}}_{\mathbf{q}\mathbf{q}}=E\left\{\tilde{\mathbf{q}}{\tilde{\mathbf{q}}}^{\mathrm{H}}\right\}=\left[\begin{array}{cccc}{\mathbf{C}}_{\mathbf{q}\mathbf{q}}& \mathbf{0}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& {\mathbf{C}}_{{\mathbf{q}}^{\widehat{\mathit{ı}}}{\mathbf{q}}^{\widehat{\mathit{ı}}}}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& {\mathbf{C}}_{{\mathbf{q}}^{\mathit{ĵ}}{\mathbf{q}}^{\mathit{ĵ}}}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& \mathbf{0}& {\mathbf{C}}_{{\mathbf{q}}^{\widehat{\kappa }}{\mathbf{q}}^{\widehat{\kappa }}}\end{array}\right]=4{\sigma }^2\mathbf{I}.$$

(14)

The structure of the matrix ${\widehat{\mathbf{C}}}_{\mathbf{q}\mathbf{q}}$ is invariant to any linear transformation $\nu ={\mathbf{T}}^{{}^{\mathrm{H}}}\mathbf{q}$ where $\mathbf{T}$ is a quaternion weight matrix [54], thus ensuring that the quaternion properness is preserved by linear layers in neural networks.

As for improper signals in Section 3.1, a probability density function can be derived for $\mathbb{H}$-proper signals too. In this framework, the distribution is described by the covariance matrix in (14) which is equal to $4{\sigma }^2\mathbf{I}$ and by the mean of the quaternion component in (9):

$$p\left(\tilde{\mathbf{q}}\right)=\frac{1}{{\left(2\pi {\sigma }^2\right)}^{2N}}exp\left\{-\frac{1}{2{\sigma }^2}{\left(\mathbf{q}-{\mu }_{\mathbf{q}}\right)}^{\mathrm{H}}\left(\mathbf{q}-{\mu }_{\mathbf{q}}\right)\right\}$$

(15)

In order to evaluate the properness of an augmented zero-mean quaternion random vector, a simple and fast coefficient was proposed in [32]. It indicates the correlation of the signal with its perpendicular involutions through the complementary covariance matrices. Note that the coefficient is bounded in $[0,1]$ and it is 0 in the case of $\mathbb{H}$-proper signals, while it is equal to 1 for improper ones. It is defined as

$$I_{\alpha }=\left|\frac{{\mathbf{C}}_{\mathbf{q}{\mathbf{q}}^{\alpha }}}{{\mathbf{C}}_{\mathbf{q}\mathbf{q}}}\right|,\alpha =\widehat{\mathit{ı}},\mathit{ĵ},\widehat{\kappa }.$$

(16)

In our experiments, since we are dealing with $\mathbb{H}$-proper signals, we can just report the average of the three coefficients, simply denoted as I. Conversely, if we evaluate $\mathbb{R}$ or $\mathbb{C}$ proper signals, we should consider each coefficient independently.

4. The $\mathbb{H}$-Proper Variational Autoencoder

Originally introduced in [26] (the implementation of the QVAE is available online at: https://github.com/eleGAN23/QVAE, accessed on 1 July 2021), the quaternion variational autoencoder (QVAE) is a generative model which learns and controls the latent distribution of an n-dimensional $\mathbb{H}$-proper quaternion input. Once the distribution is learnt, then the model is able to generate new samples from it thanks to the probabilistic relation between the two quaternion spaces of the input and of the latent distribution.

Similar to the variational autoencoder (VAE) for real-valued distributions in [13], the QVAE grasps the mapping of the original quaternion input $\mathbf{x}\in \mathbb{H}$ into the latent space described by a prior distribution $p_{\theta }\left(ϵ\right)$, with $ϵ\in \mathbb{H}$. Then, it learns the conditional distribution $p_{\theta }\left(\mathbf{x}\right|\mathbf{z})$ of the input with respect to the latent vector $\mathbf{z}\in \mathbb{H}$, that is the prior distribution adjusted with the statistics learnt by the encoder. However, due to the intractability of the mapping distribution $p_{\theta }\left(\mathbf{z}\right|\mathbf{x})$, the QVAE introduces an approximation $q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})$ to encode the input into the latent vector, as in its real-valued counterpart [13]. The parameters of this recognition model (or inference model) are then optimized to match the true intractable conditional distribution as $q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})\approx p_{\theta }\left(\mathbf{z}\right|\mathbf{x})$. By employing this approximation, it is possible to express the marginal likelihood

$$\begin{array}{cc}\hfill log\left(p_{\theta }\left(\mathbf{x}\right)\right)& =\lambda {\mathcal{D}}_{\mathrm{KL}}\left(q_{\varphi }\left(\mathbf{z}|\mathbf{x}\right)\parallel p_{\theta }\left(\mathbf{z}|\mathbf{x}\right)\right)+{\mathcal{L}}_{\theta ,\varphi }\left(\mathbf{x}\right)\hfill \\ \hfill & =-\lambda {\mathcal{D}}_{\mathrm{KL}}\left(q_{\varphi }\left(\mathbf{z}|\mathbf{x}\right)\parallel p_{\theta }\left(ϵ\right)\right)+E_q\left\{log\left(p_{\theta }\left(\mathbf{x}|\mathbf{z}\right)\right)\right\}\hfill \end{array}$$

(17)

where ${\mathcal{L}}_{\theta ,\varphi }\left(\mathbf{x}\right)$ is the so called evidence lower bound (ELBO) [13] with respect to the parameters $\varphi$ and $\theta$, which is a lower bound on the log-likelihood of the data distribution [63]. ${\mathcal{D}}_{\mathrm{KL}}(·)$ is the KL divergence between the $\mathbb{H}$-proper centered isotropic Gaussian variable $p_{\theta }\left(ϵ\right)\sim \mathcal{N}(\mathbf{0},{\widehat{\mathbf{C}}}_{ϵϵ})$ and the $\mathbb{H}$-proper Gaussian $q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})\sim \mathcal{N}({\mu }_{\mathbf{z}},{\widehat{\mathbf{C}}}_{\mathbf{z}\mathbf{z}})$, with ${\widehat{\mathbf{C}}}_{\mathbf{z}\mathbf{z}}$ taking the form of (14) and ${\widehat{\mathbf{C}}}_{ϵϵ}=4\mathbf{I}{\sigma }^2$. The KL divergence in the quaternion domain takes the following form:

$$\begin{array}{cc}\hfill {\mathcal{D}}_{\mathrm{KL}}\left(q_{\varphi }\left(\mathbf{z}|\mathbf{x}\right)\parallel p_{\theta }\left(ϵ\right)\right)=& \frac{1}{2}\left(Tr\left\{{\widehat{\mathbf{C}}}_{ϵϵ}^{-\frac{1}{2}}{\widehat{\mathbf{C}}}_{\mathbf{z}\mathbf{z}}{\widehat{\mathbf{C}}}_{ϵϵ}^{-\frac{1}{2}}\right\}\right.\hfill \\ \hfill & +\left.{\mu }_{\mathbf{z}}^{\mathrm{H}}{\widehat{\mathbf{C}}}_{ϵϵ}^{-1}{\mu }_{\mathbf{z}}-N+log\left(\frac{det\left({\widehat{\mathbf{C}}}_{ϵϵ}\right)}{det\left({\widehat{\mathbf{C}}}_{\mathbf{z}\mathbf{z}}\right)}\right)\right).\hfill \end{array}$$

(18)

The optimization of the QVAE cost function (17) goes through the minimization of the KL divergence, through which the QVAE forces the recognition model to be as close as possible to the prior distribution by operating on the statistics ${\mu }_{\mathbf{z}}$ and ${\widehat{\mathbf{C}}}_{\mathbf{z}\mathbf{z}}$.

5. Kullback–Leibler Divergence as Entropy Loss Due to Improperness

The KL divergence in (17) can be indeed seen as a measure of the improperness of the recognition model. As defined in [51], the measure is described by:

$$\mathcal{P}=\underset{{\widehat{\mathbf{C}}}_{ϵϵ}\in \mathcal{C}}{min}{\mathcal{D}}_{\mathrm{KL}}\left({\widehat{\mathbf{C}}}_{\mathbf{z}\mathbf{z}}\parallel {\widehat{\mathbf{C}}}_{ϵϵ}\right)$$

(19)

where $\mathcal{C}$ is the set of all the augmented covariance matrices respecting one of the quaternion propernesses introduced in Section 3.2. This measure estimates the distance of the recognition model $q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})$ with an augmented covariance matrix ${\widehat{\mathbf{C}}}_{\mathbf{z}\mathbf{z}}$ from the $\mathbb{H}$-proper isotropic Gaussian variable $p_{\theta }\left(ϵ\right)$ described by the augmented covariance matrix ${\widehat{\mathbf{C}}}_{ϵϵ}$, by assuming ${\mu }_{\mathbf{z}}=0$.

Since we are considering distributions with zero mean, the first term in the second line of the KL Equation (17) vanishes. Furthermore, Equation (18) minimizes the divergence over the set $\mathcal{C}$; thus, in the $\mathbb{H}$-proper case, the matrix which minimizes (18) is the diagonal covariance matrix defined in (14). Employing these considerations and with some computations, we can reduce the improperness measure to:

$$\mathcal{P}=-\frac{1}{2}log\left(det\left\{{\widehat{\mathbf{C}}}_{ϵϵ}^{-\frac{1}{2}}{\widehat{\mathbf{C}}}_{\mathbf{z}\mathbf{z}}{\widehat{\mathbf{C}}}_{ϵϵ}^{-\frac{1}{2}}\right\}\right).$$

(20)

If the signal for which we want to measure the improperness, i.e., the recognition model $q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})$, is $\mathbb{H}$-proper, then the measure will be close to 0 since ${\widehat{\mathbf{C}}}_{\mathbf{z}\mathbf{z}}$ will be close to ${\widehat{\mathbf{C}}}_{ϵϵ}$ in the KL sense. Conversely, if the signal is improper, $\mathcal{P}$ will be far from 0.

By defining the improperness measure through the KL, it is possible to leverage the relation among the divergence and the entropy and exploring the mutual information of the signal. Therefore, we can use the definition of the differential entropy for a generic quaternion vector [51,52] to compute the entropy of the recognition model $q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})$:

$$\begin{array}{cc}\hfill \mathrm{H}\left(q_{\varphi }\left(\mathbf{z}|\mathbf{x}\right)\right)& =2Nlog\left(\frac{\pi e}{2}\right)+\frac{1}{2}log\left(det\left({\widehat{\mathbf{C}}}_{\mathbf{z}\mathbf{z}}\right)\right)\hfill \\ \hfill & =log\left({\left(\frac{\pi e}{2}\right)}^{2N}det{\left({\widehat{\mathbf{C}}}_{\mathbf{z}\mathbf{z}}\right)}^{\frac{1}{2}}\right).\hfill \end{array}$$

(21)

The entropy expression can be simplified for isotropic $\mathbb{H}$-proper variables such as the prior distribution $p_{\theta }\left(ϵ\right)$, by replacing the value of the determinant from (14), which is equal to ${\left(4{\sigma }^2\right)}^{4N}$.

$$\begin{array}{cc}\hfill \mathrm{H}\left(p_{\theta }\left(ϵ\right)\right)& =log\left({\left(\frac{\pi e}{2}\right)}^{2N}{\left({\left(4{\sigma }^2\right)}^{4N}\right)}^{\frac{1}{2}}\right)\hfill \\ \hfill & =2N\left(log\left(2\pi e{\sigma }^2\right)\right).\hfill \end{array}$$

(22)

In [52], the authors proved that the differential entropy is maximum for $\mathbb{H}$-proper signals. Thus, (21) is an upper bound for the general entropy definition (20), hence: $\mathrm{H}\left(q_{\varphi }\left(\mathbf{z}|\mathbf{x}\right)\right)\le \mathrm{H}\left(p_{\theta }\left(ϵ\right)\right)$.

Then, according to [51], the improperness measure $\mathcal{P}$ can be rewritten through the difference between the entropy of the generic augmented quaternion and the entropy of the isotropic variable:

$${\mathcal{P}}_{\mathrm{H}}=\mathrm{H}\left(p_{\theta }\left(ϵ\right)\right)-\mathrm{H}\left(q_{\varphi }\left(\mathbf{z}|\mathbf{x}\right)\right).$$

(23)

This formulation elegantly evaluates the loss of entropy that is due to the improperness of the recognition model. Therefore, we can measure the properness of the recognition model and compute the QVAE entropy loss due to the measure of properness (or improperness) of $q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})$. Thus, when optimizing the QVAE cost function during training, together with the ELBO minimization, through the KL divergence we are jointly minimize also the difference between the Gaussian distributions $p_{\varphi }\left(ϵ\right)$ and $q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})$ and the entropy loss of the latter.

Since the KL under the information theoretic perspective allows the QVAE to deal also with improper signals, although the QVAE is originally set up for $\mathbb{H}$-proper distributions, by means of an approximation it can be employed also for improper random variables. Let us consider an incoming improper signal $\mathbf{x}\in \mathbb{H}$, so both the recognition model $q_{\varphi }\left(\mathbf{z}\right|\mathbf{x})$ and the generative model $p_{\theta }\left(\mathbf{x}\right|\mathbf{z})$ are represented by improper distributions too. We leave instead unchanged the prior distribution $p_{\theta }\left(ϵ\right)$, which is the centered isotropic $\mathbb{H}$-proper distribution. On the training stage, the KL divergence of the QVAE will move the improper distribution towards the $\mathbb{H}$-proper prior distribution. While the distribution learning is well performed by the model, the KL takes care of increasing the entropy by slightly reducing the improperness. However, the overall original (improper) structure of the signal is preserved by the QVAE which just aims at enhancing the entropy of the recognition model, while leaving free the generative model.

6. Experimental Results

In this section, we report the sets of experiments we performed for testing the reliability of the improperness measures and the performances of the $\mathbb{H}$-proper variational autoencoder in learning proper as well as improper distributions.

6.1. Proper and Improper Test Signals

We consider two proper signals and two improper signals and we evaluate the improperness coefficient (16), the differential entropy (20), the improperness measure (18) and the entropy loss (22). We expect that the proper signals have improperness measure and coefficients close to 0, as well as the entropy loss. On the contrary, the entropy should be greater than for the corresponding improper signals.

As input signals, we consider:

• $\mathbb{H}$-proper signal $\mathbf{x}$ with independent Gaussian components defined as:

  $$\mathrm{P}1:{\mathbf{x}}_{\delta }\sim \mathcal{N}(0.03,1)$$

  with $\delta =\{0,1,2,3\}$.
• $\mathbb{H}$-proper filtered signal $\mathbf{x}$ from a colored Gaussian noise $\eta \left[n\right]$ as [31]:

  $$\mathrm{P}2:\mathbf{x}\left[n\right]=b\mathbf{x}[n-1]+\frac{\sqrt{1-b^2}}{\sqrt{4}}\eta \left[n\right]$$

  with $b=0.5$.
• Improper signal $\mathbf{x}$ from a quaternion $\mathbf{q}$ with components:

  $${\mathbf{q}}_0=0,{\mathbf{q}}_1\sim \mathcal{N}(0,1),{\mathbf{q}}_2\sim \mathcal{N}(0,1),{\mathbf{q}}_3\sim \mathcal{N}(0,1)$$

  and composed as:

  $$\mathrm{I}1:\mathbf{x}=\mathbf{z}{e}^{\mathbf{q}},\mathrm{with}\mathbf{z}\sim \mathcal{N}(0.03,1)$$
• Improper filtered signal $\mathbf{x}$ from a Gaussian noise $\eta \left[n\right]$ as:

  $$\mathrm{I}2:{\mathbf{x}}_{\delta }\left[n\right]=\sqrt{1-b_{\delta }^2}+(1-b_{\delta })\eta \left[n\right]$$

  with $b=[0.75,0.78,0.82,0.85]$ and $\delta =\{0,1,2,3\}$.

Mild improper signals (i.e., with $I_{\alpha }$ different from 0 and to 1) can be generated by varying the coefficient b in the improper signal I2. As an example, by setting $b=[0.01,0.2,0.75,0.99]$, it is possible to obtain a mild-improper signal with $0.4<I_{\alpha }<0.7$. However, we want to stress the $\mathbb{H}$-proper QVAE with completely improper cases that are the extreme and most difficult cases. For this reason, we only consider improper signals with $I_{\alpha }=1$.

We compute the measures defined in Section 5 for each of these signal by using the Quaternion Toolbox for MATLAB${}^{\circledR }$ (S. J. Sangwine and N. Le Bihan. Quaternion Toolbox for MATLAB${}^{\circledR }$, Version 2 with support for Octonions, 2013. First public release 2005. Software library available online at: http://qtfm.sourceforge.net/, accessed on 1 July 2021) and report the results in

Table 2. In order to prove the measures and the assumptions presented in Section 5, we compute the average improperness coefficient I (16), the differential entropy $\mathrm{H}$ (20), the improperness measure $\mathcal{P}$ (19) and entropy loss ${\mathcal{P}}_{\mathrm{H}}$ (22) for different kinds of $\mathbb{H}$-proper and improper signals. The entropy $\mathrm{H}$ is highest for the $\mathbb{H}$-proper signals, while the improperness measure and coefficients are close to 0. As  is clear from $\mathcal{P}$ and ${\mathcal{P}}_{\mathrm{H}}$, there is a low entropy loss due to the improperness in the first two signals, while it is high for the last two, improper, random vectors.

Signal	I	$\mathbf{H}$	$\mathcal{P}$	${\mathcal{P}}_{\mathbf{H}}$
P1. (proper)	0.0393	4.0177	0.0049	0.0049
P2. (proper)	0.0528	5.0174	0.0103	0.0103
I1. (improper)	1.0000	−51.6040	50.9298	51.0691
I2. (improper)	0.9999	−45.4454	51.3121	51.1843

. As we expect, the proper signals have low improperness coefficients, meaning that the complementary covariances are sparse, while the coefficients for the improper signals are reasonably equal to 1. On the other hand, the differential entropy is low and negative for the improper signals, while it is highest for the proper ones, proving that (21) is an upper bound for the generic differential entropy. Moreover, the entropy loss is almost 0 for proper signals, confirming that there is no loss of entropy due to the improperness. On the contrary, the improper signals come at the cost of a high entropy drop.

6.2. Proper and Improper Signals for an $\mathbb{H}$-Proper QVAE

We consider a plain QVAE with a quaternion encoder network and a quaternion decoder network. The first model is comprised of a stack of five quaternion fully connected layers (QFC) with an increasing number of weights $[32,64,128,256,512]$, as in [26,64] and split ReLU activation function as the PixelHVAE in [65]. The statistics of the latent distribution are learnt through two additionally quaternion linear layers with number of weights equal to 4 times the latent dimension, which we fix to 25. The decoder has a mirrored structure, and thus quaternion fully connected layers are piled up as $[512,256,128,64,32]$, with an additional refiner layer at the end of the stack. We do not consider including the quaternion batch normalization [38,45] since it can introduce randomness which may affect the correct learning of the distribution statistics [24,66]. For every experiment, the prior distribution is a centered isotropic $\mathbb{H}$-proper quaternion Gaussian distribution, as described in Section 4. We employ a training signal with 1000 samples, a validation set of 500 and a test set of 1000 without mini batches and we perform 5k iterations with Adam having a learning rate equal to $0.0005$. As in the original QVAE [26], the cost function is a weighted sum of a quaternion mean squared error as defined in [37], and the KL divergence of Section 5, for which we fix the scale parameter $\lambda$ equal to $0.1$. Figure 1 shows the architecture of the QVAE we consider in our tests.

For each experiment, we evaluate the performance of the model in the reconstruction task and generation task as well as on the ability of preserving the properness or the improperness of the incoming signal. In order to assess the generation ability of the model, in Figure 2, we plot the generated components and the complete signal sampling from the model trained on a proper signal and on an improper signal. While the components ${\mathbf{x}}_0,{\mathbf{x}}_1,{\mathbf{x}}_2,{\mathbf{x}}_3$ of the improper signal show different variances, the ones from the proper signal has the same variance according to the properties in Table 1. Figure 3 reports instead the original and reconstructed distributions for the proper and improper case, in the first and in the second line, respectively, with an MSE equal to $0.1161$ for the proper signal and to $0.01293$ for the improper one. In the figure, the first four columns represent the four components ${\mathbf{x}}_0,{\mathbf{x}}_1,{\mathbf{x}}_2,{\mathbf{x}}_3$, while the last one displays the distribution of the complete signal $\mathbf{x}$. The $\mathbb{H}$-QVAE is able to perfectly reconstruct the input distribution even in the improper case, grasping the different variances in each component and learning the complete signal. Thus, the QVAE is effective even if approximated for improper distributions, for which it is capable of learning the correct statistics of each component.

Table 3. Average improperness coefficient I (16), differential entropy $\mathrm{H}$ (20), improperness measure $\mathcal{P}$ (19) and entropy loss ${\mathcal{P}}_{\mathrm{H}}$ (22) for different outputs from the QVAE (Recons stands for the reconstructed signal, Sample for the generated one) with different kinds of $\mathbb{H}$-proper and improper signals. The QVAE preserves the structure of the signal, learning the properness and improperness nature of the input.

Signal	Out	I	$\mathbf{H}$	$\mathcal{P}$	${\mathcal{P}}_{\mathbf{H}}$
P1. (proper)	Recons	0.0287	5.1519	0.0030	0.0029
	Sample	0.0537	2.3859	0.0096	0.0095
P2. (proper)	Recons	0.0640	3.4614	0.0124	0.0123
	Sample	0.1099	−0.2705	0.0348	0.0347
I1. (improper)	Recons	0.9989	0.3749	9.9700	9.9700
	Sample	0.9930	−3.6337	8.1208	8.1208
I2. (improper)	Recons	0.9998	−8.1100	13.0767	13.0767
	Sample	0.9964	−10.7147	12.1469	12.1469

presents the results of the improperness and entropy analysis on the reconstructed and generated signals on the test set. As for the original samples in Table 2, we compute the improperness coefficient (16), the differential entropy (20), the improperness measure (18) and the entropy loss (19). It is worth noting that the QVAE trained on $\mathbb{H}$-proper signals still generates and reconstruct $\mathbb{H}$-proper signals, as measured by the improperness measures and the entropy loss. Moreover, the entropy of this kind of signal is higher than the entropy of the improper samples. While we already knew that $\mathbb{H}$-properness is preserved by linear transformations, this result asserts that it is maintained also for non-linear transformations brought by activation functions in neural networks, adding an important milestone in the study of $\mathbb{H}$-proper signals.

Another interesting result comes when the QVAE is trained on improper signals. While the improperness coefficients of both reconstructed and generated signals are close to the coefficients of their respective original signals in Table 2, indicating the improperness of these random variable, the entropy values and losses values are intriguing. Indeed, the differential entropy is increased with respect to the input signal (originally, ≈−51 and ≈−45), approaching values from ≈−10 up to 0, meaning that the $\mathbb{H}$-proper structure of the QVAE works towards an increase in entropy and an improperness reduction in the input signal. Confirming this, the loss of entropy ${\mathcal{P}}_{\mathrm{H}}$ was drastically reduced with respect to the original signal, ranging from approximately 8 and 13 while the input entropy loss is around ≈51.

Hence, the $\mathbb{H}$-proper QVAE can be approximated to learn improper distributions, on which it preserves the original structure of the signal while increasing the entropy of the latent space, at the cost of a limited improperness reduction. Although the slight modification on the distribution, the QVAE is still able to catch the correct statistics and to perform a good generation as well as a precise reconstruction, as Figure 2 and Figure 3 show.

To further assess the performance of the QVAE and provide a comparison with its real-valued counterpart, we test a real-valued VAE on the same signal benchmarks. We expect that the real-valued VAE performs similar to the QVAE in the proper case, but that the quaternion model outperforms the real-valued one when dealing with improper signals. Indeed, thanks to the quaternion algebra properties, the QVAE is able to learn intra-component correlations while the VAE do not catch them. Thus, in cases with no correlation among components i.e., proper signals, the two models should have similar performances. On the contrary, in the improper case where the correlation among components is higher the QVAE gains advantages from the quaternion algebra and obtains better performances. Results and details are reported in the Appendix A. However, thanks to the quaternion algebra properties, the QVAE has just $25\%$ of free parameters with respect to its real-valued counterpart.

7. Conclusions

In this paper, we have conducted a thorough analysis on a $\mathbb{H}$-proper quaternion-valued variational autoencoder (QVAE) under an information-theoretic perspective, investigating the entropy loss of the model due to the measure of improperness in the input signal. We have proved that the QVAE can be approximated for improper signals, for which it increases the entropy of the latent space by slightly reducing the improperness. Moreover, it is worth noting that, in each of our experiments, the properness is preserved also for nonlinear transformations brought by activation functions and not only for simple quaternion linear operations. As a conclusion, the QVAE is able to learn the correct distribution of both $\mathbb{H}$-proper and improper random variables, while the measure of improperness, as well as the entropy of the latent space, can be calibrated through the KL divergence in the cost function.