Relative Consistency of Sample Entropy Is Not Preserved in MIX Processes

Abstract

Relative consistency is a notion related to entropic parameters, most notably to Approximate Entropy and Sample Entropy. It is a central characteristic assumed for e.g., biomedical and economic time series, since it allows the comparison between different time series at a single value of the threshold parameter r. There is no formal proof for this property, yet it is generally accepted that it is true. Relative consistency in both Approximate Entropy and Sample entropy was first tested with the $MIX$ process. In the seminal paper by Richman and Moorman, it was shown that Approximate Entropy lacked the property for cases in which Sample Entropy did not. In the present paper, we show that relative consistency is not preserved for $MIX$ processes if enough noise is added, yet it is preserved for another process for which we define a sum of a sinusoidal and a stochastic element, no matter how much noise is present. The analysis presented in this paper is only possible because of the existence of the very fast NCM algorithm for calculating correlation sums and thus also Sample Entropy.

1. Introduction

Relative consistency of Sample Entropy ($SampEn$) has been assumed in all clinical and economic applications [1,2,3,4]. Indeed, if we say that a one time series is more complex than another on the basis of their value at a certain threshold r, we assume this either explicitly or tacitly. It is quite surprising that there are no comprehensive studies on this property of Sample Entropy. The analytic proof of this property would be very hard to derive. In fact, very little theoretical work has been done on these parameters, most of which is limited to the Moorman and Richman paper [4]. We do not know the distribution of Sample Entropy (the t distribution assumed in [4] is based on data, not analytical properties), we do not know entropy profiles of most common processes or the influence of noise on these processes. The experimental verification of relative consistency requires calculation of $SampEn$ across many different thresholds for many time series e.g., many $RR$ (i.e., the distance between two consecutive R-vawes in the ECG) intervals time series, acquired with the same equipment (the same sampling rate as well as other external conditions). This is difficult because of the computational burden of this task. In this paper, we overcome this difficulty by using the NCM algorithm [5]. Unlike a formal proof, this procedure would not yield certainty about relative entropy, but it could either corroborate, or decisively refute it.

We believe that the methodology developed in this paper is systematic and applicable to all types of time series studied with the use of Sample Entropy. Furthermore, we provide the software necessary to perform such an analysis quickly and without large hardware investments.

This paper is not the first one to study relative consistency of Sample Entropy as a universal property. Indeed, even the creators of $SampEn$ allow for the possibility of Sample Entropy not holding universally for all time series [4].

In [6], the authors perform an analysis of short time series acquired by recording gait data. The most significant result in the context of the present paper is the finding that $SampEn$ is not relatively consistent at $r=0.2$ for the time series of step time of length 200—the averaged entropy profiles for short data of young subject cross with the profiles of older subjects. The authors do not provide specific results, i.e., how many curves cross, but nonetheless this is a significant finding for this time series. The cross between entropy profiles in [6] was observed for very short recordings, but this finding was corroborated in longer recordings [7]. The authors find that, for one hour recordings of time series of step time, the relative consistency is lost between overground and treadmill walking recordings at $r=0.2$ for $m=2$ and $m=3$. The authors attribute this result to $r·SD$ being close to the precision of the data. Still, this study demonstrates that relative consistency is at least subject to some technical conditions.

In [8], it is found that relative consistency is not preserved in very short ($N=50$) sinusoidal signals for sample entropy, while it is for the Fuzzy Entropy, a parameter that is introduced in that paper. While analyzing a similar set of measures, i.e., Approximate Entropy, Sample Entropy, and Fuzzy Measure Entropy, Zhao et al. [9] find that sample entropy does not behave consistently in distinguishing between normal sinus rhythm and congestive heart failure groups, which may be indicative of a lack of relative consistency. This paper is not entirely conclusive in this respect as it uses a segmented approach, and thus it is quite dissimilar to our study as well as the above-mentioned papers, but they do find that Fuzzy Measure Entropy is, as expected, totally consistent in this respect.

In this paper, we concentrate on the process which was used to demonstrate and study the properties of Approximate Entropy and $SampEn$, i.e., the $MIX$ process. We contrast the results obtained for this process with a closely related process and show that their properties with respect to relative consistency are widely different. The considerations in this paper are limited to Sample Entropy because the fact that Approximate Entropy is not relatively consistent with respect to the $MIX$ process has already been shown in [4].

1.1. Sample Entropy

Given a time series

$$U_i=\{u\left(1\right),u\left(2\right),\dots ,u\left(N\right)\},$$

(1)

where N is the number of data points, let us build an auxiliary object

$$V_i^{m,\tau }=\{\overrightarrow{v}\left(1\right),\overrightarrow{v}\left(2\right),\dots ,\overrightarrow{v}(N-(m-1)\tau )\},$$

(2)

which is a set of vectors in an m-dimensional embedding space [10,11], Vectors ${\overrightarrow{v}}_{m,\tau }\left(i\right)=[u\left(i\right),u(i+\tau ),u(i+2\tau ),\dots ,u(i+(m-1)\tau )]$, i.e., the ${\overrightarrow{v}}_{m,\tau }\left(i\right)$ consist of m ordered points, beginning at position i. The parameter $\tau$ is known as time lag. Therefore, we have $L_m=N-(m-1)\tau$ vectors $V_i^{m,\tau }$ for a fixed $\tau$—these are often called templates.

Let us define the so called correlation sum:

$$C^m\left(r\right)=L_m^{-1}\sum _{i=1}^{L_m}{C}_i^m\left(r\right).$$

(3)

$C_i^m$ are defined as

$$C_i^m\left(r\right)=\left\{\begin{array}{cc}{(L_m-1)}^{-1}\sum _{j=1,j\ne i}^{L_m}\mathsf{\Theta }(r-|{\overrightarrow{v}}_m\left(i\right)-{\overrightarrow{v}}_m\left(j\right)\left|\right)\hfill & \mathrm{if} i\le L_m\hfill \\ 0\hfill & \mathrm{if} i>L_m\hfill \end{array}\right.,$$

(4)

$\mathsf{\Theta }$ is called the Heaviside function

$$\mathsf{\Theta }\left(x\right)=\left\{\begin{array}{cc}1\hfill & \mathrm{if} x\ge 0\hfill \\ 0\hfill & \mathrm{if} x<0\hfill \end{array}\right..$$

(5)

where r is called the radius of comparison [12], which is used to check the similarity of two vectors by checking their distance with respect to a norm. The distance between two vectors in can be defined in many ways, but the following maximum coordinate distance definition seems to have the best mathematical properties [13]:

$$|{\overrightarrow{v}}_m\left(i\right)-{\overrightarrow{v}}_m\left(j\right)|=\underset{k=1,2,\dots ,m}{max}\left(\right|u(i+(k-1)\tau )-u(j+(k-1)\tau )\left|\right).$$

(6)

$SampEn$ (just like Approximate Entropy) is an attempt to build an estimator the Eckmann and Ruelle [14] entropy:

$$ER=\underset{r\to 0}{lim}\underset{m\to \infty }{lim}\underset{N\to \infty }{lim}[{\mathsf{\Phi }}^m\left(r\right)-{\mathsf{\Phi }}^{m+1}\left(r\right)],$$

(7)

where N, r, and m have the same definition as before. This expression involves limits, so it cannot be directly applied to a realistic, measured time series. In order to make this possible, this definition was rewritten by Richman and Moorman [4] to the following form for a finite time series:

$${\mathsf{\Phi }}^m\left(r\right)={(N-m+1)}^{-1}\sum _{i=1}^{N-m+1}log{B}_i^m\left(r\right),$$

(8)

$B_i^m$ has the same definition as $C_i^m$, with the only difference that self matches are included in $B_i^m$ and excluded from $C_i^m$. Richman and Moorman propose using the following, closely related quantity as complexity measure instead of the Eckmann–Ruelle entropy

$$SampEn(m,r)=\underset{N\to \infty }{lim}[-ln\frac{C^{m+1}\left(r\right)}{C^m\left(r\right)}],$$

(9)

which, for a finite time series, can be estimated by

$$SampEn(m,r,N)=-ln\frac{C^{m+1}\left(r\right)}{C^m\left(r\right)}.$$

(10)

A detailed look into the constitutive elements of these formulas lead to the conclusion that Sample Entropy is the negative logarithm of the conditional probability that two sequences, which are within the r radius of tolerance of one another for m points, remain at the same radius of tolerance for $m+1$st point. A more detailed treatment of $SampEn$ may be found in [4].

1.2. Relative Consistency

The notion of relative consistency was introduced by Pincus in [1,3], and this property follows from the properties of the Kolmogorov–Smirnov (KS) entropy. Rewritten in terms of $SampEn$, we have the following property: for deterministic dynamical processes A, B, we should have that, from $\mathrm{KS}\mathrm{entropy}\left(\mathrm{A}\right)<\mathrm{KS}\mathrm{entropy}\left(\mathrm{B}\right)$, it follows that $SampEn(m,r)\left(A\right)<SampEn(m,r)\left(B\right)$ and, conversely, for a wide range of m and r. This entails that, if $SampEn(m_1,r_1)\left(A\right)<SampEn(m_1,r_1)\left(B\right)$, then $SampEn(m_2,r_2)\left(A\right)<SampEn(m_2,r_2)\left(B\right)$ and vice versa. In other words, if $SampEn$ for one process is lower than that for another process for a set of parameters $(m_1,r_1)$, then this holds true for any other set $(m_2,r_2)$ [4]. It should be stated clearly that this is an expectation and a desirable property of $SampEn$, rather than a mathematically proven property. If this holds true, then we are able to compare two processes at a single point $(m_1,r_1)$ and draw conclusions for all points. This is what is actually happens in applications.

1.3. The $MIX$ and $MIXTURE$ Processes

Let us now define the two processes which will be used to test the relative consistency of $SampEn$ under different conditions.

1.3.1. $MIX\left(p\right)$ Process

Let $0\le p\le 1$ be discrete probability. Let us define three time series [4]:

$$X_j=\sqrt{2}sin\left(j\right),$$

(11)

In the above, we do not use the frequency modifying factor $\frac{2\pi }{12}$ since our sampling is quite dense, as will become apparent in the Data Analysis section.

$$Y_j=U(-\sqrt{3},\sqrt{3}),$$

(12)

i.e., uniform independent, identically distributed random variable, and

$$Z_j=B(1,p),$$

(13)

i.e., a Bernoulli random variable with probability of success equal to p. We can now define the $MIX\left(p\right)$ process as

$$MIX\left(p\right)=(1-Z_j)X_j+Z_j{Y}_j.$$

(14)

1.3.2. $MIXTURE\left(\lambda \right)$ Process

This is a very simple process which is composed of a sum of two processes: a deterministic and stochastic process, the second of which is controlled by a tuning parameter $\lambda$. Let us define

$$X_j=\sqrt{2}sin\left(j\right),$$

(15)

$$Z_j=U(0,1),$$

(16)

and the final process

$$MIXTURE\left(\lambda \right)=X_j+\lambda (Z_j-0.5).$$

(17)

It can be seen that the $\lambda$ parameter controls the amplitude of the added noise. Figure 1 presents a few examples of of the above processes.

It should also be noted that, in spite of their apparent similarity, these two processes are very different. In the first of them, we tune how much randomness is in the signal, i.e., we control how many samples, on average, come from the random process. In the other process, we control how large the random effect is, i.e., what contribution each point gets from the random process. It may be argued that, in the $MIX\left(p\right)$ process, the parameter (p) controls the amount of the random component in the deterministic signal, while the variance of a single random insertion remains the same, whereas in the $MIXTURE\left(\lambda \right)$ process the amount of the random component is constant (maximum), and the parameter ($\lambda$) controls the variance of the random addition. This is interesting since it has been argued that the amount of variance in the analyzed time series affects the results of entropy calculations [15,16].

1.4. The NCM Algorithm

The NCM algorithm is the fastest algorithm for calculating correlation sums and $SampEn$ which at the same time uses the whole time series, without any sub-sampling or simplifications.

This algorithm is of the look-up table type and uses many tricks limiting the number of operations as compared to the brute-force approach. The central objects of NCM are triangular matrices $\widehat{N}$ whose elements are defined in the following way:

$$n_{ij}=\left|\right|u_i-u_{i+(j+1)·\tau }\left|\right|,$$

(18)

where u are the elements of the U time series and $\tau$ is the time lag. For a time series with N points, the dimensionality of this matrix is $N-\tau -1\times (N-1)/\tau -1$. It is quite obvious that for any realistic time series this amounts to a very large matrix. The first operation-reducing technique is limiting calculation to sub-blocks. Using the symmetry of the matrix, elements with indices not meeting the condition $i+(j+1)·\tau \le N-\tau -1$ are set to zero, thus halving the number of summations in the correlation sum. Another result of this approach is removing redundant operations in calculating $C^m$ by reducing the number of operations for maximum norm from $m^2$ to m. The last operation is searching for the first occurrence of 0 from an arithmetic operation instead of a loop, with

$$(n_{i,j}^m-b)/a,a=-\frac{r_{max}-r_{min}}{n-1},b=r_{max}.$$

(19)

Other, more standard optimization and algorithmic techniques as well as hardware scaling can be used to further improve the performance of the procedure. The full description of the algorithm may be found in [5] and Python code on the GPL license may be downloaded from https://github.com/sebzur/NCM-algorithm.

2. Materials and Methods

All the signals were processed with the use of the Python programming language. The correlation sums were calculated with the use of the NCM algorithm using the software available at https://github.com/sebzur/NCM-algorithm. The results were analyzed with the R programming language and statistical system.

2.1. The $MIX$ Process

One hundred signals were generated with four thousand samples, and four periods were generated at the frequency of 5 $Hz$. The tuning parameter p for these signals changed from 0 to 1. For each of the signals, the correlation sums $C_m$ and $C_{m+1}$ were calculated for $m=2$ (compare Equation (9)) and $SampEn$ was calculated using these results. The $SampEn$ profiles were drawn for values of threshold r spanning the segment $(0·SD,4·SD)$. As a result, we obtained 1100 plots for the different levels of the tuning parameter. We searched for crossing profiles within the plots by subtracting the profiles and counting how many times the difference changed sign.

2.2. The $MIXTURE$ Process

One hundred signals were generated according to the definition from formula (17). The parameters assumed were the same as for the $MIX$ process, and the $\lambda$ tuning parameter also changed in the range $\lambda \in (0,1)$. We calculated correlation sums and $SampEn$ profiles for the same sets of parameters as in the $MIX$ process case. As before, we got 1100 plots and we checked for crossing profiles.

3. Results

3.1. The $MIX$ Process

Figure 2 presents the entropy profiles for $m=2$.

We have observed that there are crossings in the entropy profiles. All the crosses in the present study behave in the same way: the lines always cross at two points, which means that there is a finite region for which the order of the lines reverses. The crosses have been summarized in

Table 1. Crosses between entropy profiles for the $MIX\left(p\right)$ process.

Crossing Lines	${\mathit{r}}_1$	$\mathit{SampEn}$	${\mathit{r}}_2$	$\mathit{SampEn}$
p0.7, p1.0	1.82	0.2565933	2.14	0.1591130
p0.8, p0.9	1.50	0.3900531	2.30	0.1206111
p0.8, p1.0	1.40	0.4402257	2.26	0.1298239
p0.9, p1.0	1.30	0.4964603	2.24	0.1344648

below. For each line, it contains two values of r at which the lines cross, and the $SampEn$ value at the crossing point. It is interesting to notice that many entropy profiles cross with the maximum randomness $MIX\left(1\right)$ process, and there is also one more case which does not involve this process, namely $MIX\left(0.8\right)$ and $MIX\left(0.9\right)$.

3.2. The $MIXTURE$ Process

Figure 3 presents the entropy profiles for $m=2$. There are no crossings in this type of process, irrespective of the value of $\lambda$.

4. Discussion

In the present paper, we have studied the relative consistency property of the Sample Entropy parameter. We have experimentally studied two synthetic time series—the $MIX\left(p\right)$ and $MIXTURE\left(\lambda \right)$ processes. Both of these processes have one tuning parameter; however, in the $MIX\left(p\right)$ process, the amount of randomness is controlled, and, in the $MIXTURE\left(\lambda \right)$ process, the size of the random effect is controlled. It turns out that relative consistency is not preserved for the former, while it is preserved in the latter.

The nature of Sample Entropy profiles crossing is different from that found in [6], or the behavior observed in Approximate Entropy [2], which is a flip behavior. In our analysis, the entropy profiles for $MIX\left(p\right)$ always cross twice. Of course, the reason could be the lack of possibility to observe the other cross in the above cited studies.

It is also interesting to notice that, though the amount of variance in a time series has been reported to influence the results of entropy calculations [15,16], this does not seem to be the case for the relative consistency of $MIXTURE\left(\lambda \right)$, which is preserved for all values or $\lambda$.

A question may arise whether the values of r at which the crosses were found are important for practical applications. In our opinion, it is impossible to relate the value of r for the $MIX$ process with a value of r from, say, an $RR$ intervals time series. These are two completely different processes and the values cannot be compared. The $MIXTURE$ process is in fact a good example here—there are no crosses for this process, so no crossing value of r in $MIX$ corresponds to any r value in the $MIXTURE$ process.

We believe that finding a process which is clearly relatively consistent for all values of the parameters, as opposed to a process which is mostly relatively consistent, is one of the most important results of this paper. This finding may have deep consequences.

In applications to medicine and economy relative consistency is assumed anywhere where two signals are compared and conclusions are drawn on the basis of a single set of parameters $(m,r)$. It is absolutely necessary to study real signals, e.g., time series of $RR$ intervals, using the methodology we have presented in this paper or a similar one. If the studied real signals behave more like the $MIXTURE\left(\lambda \right)$ process, then we can continue assuming relative consistency, if they behave more like the $MIX\left(p\right)$ processes, the approach to comparing signals may need to be modified.

Looking at the results obtained in this manuscript as well as the cited papers, we can notice that various processes have various regions of relative consistency, i.e., a region in a multidimensional space within which the process is relatively consistent. For the $MIX\left(p\right)$ region, we can see that process is relatively consistent for a certain region in the $(r,p)$ space, and, for the $MIXTURE\left(\lambda \right)$ relative consistency, holds for the entire studied region in the same space. These regions have been demonstrated in Figure 4.

We suggest that such regions be found for real-life processes, and, from the data gleaned from other studies, we can suspect that they will be different for different processes and for different spaces. For example, the results presented in [6,7] suggest that for the gait signal there is at least one region where the signal is not relatively consistent in the $(r,f)$ space, where f is sampling frequency, and the point $(0.2,148Hz)$ belongs to this non-consistency region (for a study on the influence of sampling frequency on $SampEn$ see also [17]). The same two studies by Yentes et.al. suggest that other spaces of interest could be $(r,N)$, where N is the length of the time series, and $(N,f)$ or even some multidimensional spaces involving these variables can be studied in this way.

The $RR$ intervals time series seems to be a good candidate for such a study because of its widespread use, equipment availability, and the fact that many researchers are already very familiar with this time series. Such a study would require a group of recordings from similar subjects, taken with the same equipment, in stationary conditions. They would also need to be long enough to let the underlying process assume as many intermediate stages as possible. In our opinion, the main obstacle in studying longer time series is the computational burden of calculating entropy profiles. The method described in this paper as well as the open source software we make available makes this possible.

We believe that the answer to the question of the regions in which relative consistency holds in various types of signals is one of the most important problems for the applicability of $SampEn$ because, in order to be able to safely compare the values of $SampEn$ at selected points, we need to make sure that the comparisons take place within a region of relative consistency.