Infrared Small-Target Detection Using Multidirectional Local Difference Measure Weighted by Entropy

Abstract

Detecting small targets from infrared remote sensing images is still a challenging task. In this research, we propose a multidirectional local difference measure weighted by entropy (MDLDE) to detect small targets from infrared images with messy backgrounds. First, a new multidirectional local difference measure is proposed to suppress the clutter background. Then, the entropy, which captures the overall heterogeneity between the target and the background, is utilized to enhance the target. Lastly, an adaptive threshold was adopted to segment the target region from the background. The designed MDLDE could effectively enhance the target and simultaneously suppress the background clutter. Experimental results on six datasets indicate that the proposed method outperformed other state-of-the-art methods in terms of the signal-to-clutter ratio gain (SCRG), background suppression factor (BSF), and receiver operating characteristic (ROC) curves.

1. Introduction

In infrared search and tracking (IRST) systems, infrared (IR) small-target detection is an active research field that occupies a crucial position and is broadly applied to a variety of fields, including early-warning, precision-guidance, and missile-tracking systems [1,2,3]. However, due to the far imaging distance and atmospheric transmissions, IR targets are small and have no concrete shape or texture features. In most instances, targets are drowned in heavy clutter and noise [4,5]. Additionally, target detection is usually deteriorated by background edges and high-brightness backgrounds. All those unfavorable factors render IR target detection more challenging [6,7,8].

Over the past several decades, researchers have been committed to studying IR small-target detection and have designed all kinds of algorithms [9,10,11,12,13,14,15]. Existing IR small-target detection methods are mainly divided into the following two categories: the first is single-frame detection, which is suitable for high-speed motion detection platforms and has faster detection speed, so it has more practical applications. The other is sequential detection. Unlike the former, sequential detection has limitations in stationary targets and rapidly changing backgrounds. In addition, it is usually based on single-frame detection such as the top-hat filter [16], max-mean/max-median filter [17], and wavelet transformation [18]. These methods are classified as traditional single-frame detection methods. However, they deliver dramatically declining performance when the background is messy.

Among numerous single-frame detection methods, methods that integrate the human visual system (HVS) are paid much attention by many researchers. The HVS indicates that human vision is more focused on contrast than brightness. As the most important characteristic of HVS, the contrast mechanism has been broadly adopted in IR small-target detectors [19]. For instance, Chen et al. [20] considered the central cell’s maximal gray value and calculated the ratio to the mean gray value of background cells to enhance the local contrast. Unlike the above method, Wei et al. [21] took into account the gray mean of the central patch instead of the maximal value and calculated the difference between the central patch and the eight background patches in the corresponding directions. Furthermore, Xiong et al. [22] also fully used directional information and proposed to detect targets in the gradient vector field. HB-MLCM [23] and MLHM [24] further enhanced the target area by designing an improved high-boost filter and calculating intrapatch homogeneity, respectively. However, in these methods, the size of the supposed background region in the sliding window increases proportionally with the growth of the target cell, which causes problems such as expansion effects and background clutter enhancement, thus affecting the detection results with false alarms and misdetections.

Recently, some novel methods for improving the structure of sliding windows have been proposed, such as TLLCM [25], in which study a trilayer filter was developed to measure the local contrast. Aghaziyarati et al. [26] designed a novel multiscale sliding window with only a one-pixel-wide background and claimed that it could compensate for three weak points of the regular methods. Wu et al. [27] proposed a novel double-neighborhood window. In addition, many infrared small-target detection methods integrating weighting functions have emerged [28,29,30,31]. For instance, Deng et al. [28] employed local entropy to weigh multiscale gray differences. Liu et al. [30] compared the gray value of the pixel in the central area with its neighborhood pixels to calculate the probability that the pixel belongs to targets. The above operations increase the robustness of the algorithm to a certain extent. However, the detection performance of those methods worsens when targets are submerged in a complex background with heavy clutter. In this paper, to robustly detect small targets from IR images with cluttered backgrounds, a novel algorithm named multidirectional local difference measure weighted by entropy (MDLDE) is proposed.

Our contribution is twofold:

(1)

A multidirectional local difference measure is proposed that can adaptively enhance a target despite variance in its size.

(2)

The overall heterogeneity measured by entropy is used for weighting directional local difference measures to enhance the robustness of the proposed method.

This paper is organized as follows. In Section 2, we describe the proposed MDLDE method in detail. The effectiveness of MDLDE is verified on six datasets in Section 3. In Section 4, the paper is concluded.

2. Methodology

2.1. Multidirectional Local Difference Measure

The flowchart of MDLDE is given in Figure 1, where the detected targets are marked in a red box. First, a multidirectional local difference measure is proposed to suppress the background. Second, the Renyi entropy is calculated to enhance the target. Both take a multiscale approach, and the saliency map is obtained. Lastly, the target segmentation separates the target from the saliency map.

To characterize the contrast between the central and surrounding areas, we adopted the sliding window given by MPCM [21] that divides the local area into the central patch (denoted as T) and eight surrounding background patches (denoted as $Bi$, $i=1,2,\dots ,8$). Then, the dissimilarity between patches, along with different directions, is defined as follows:

$$d_i=d(T,Bi)\times (T,Bi+4),(i=1,2,\dots ,4),$$

(1)

where × stands for the multiplication operation, and $d(T,Bi)=m_T-m_{Bi},(i=1,2,\dots ,8)$. $m_T$ and $m_{Bi}$ represent the mean values of the patch T and patch $Bi$.

The local difference at a given scale is defined as follows:

$$D_1=\underset{i=1,2,\dots ,4}{min}{d}_i.$$

(2)

Subsequently, to deal with the problems caused by the proposed background increasing proportionally to the target cell in the complex background and inspired by the idea of background-pixel fixation in the cumulative directional derivatives of [26], a novel multiscale convolutional kernel with eight directions is designed. As shown in Figure 2, the size of each kernel on the scale n was set to $(n+2)\times (n+2)$, which was composed of an $n\times n$ target window (surrounded by a bold box in Figure 2) and a single-pixel-wide surrounding edge. Then, the pixel values along the given direction in the target window were set to 1 except for the central one. In contrast, the pixel at the directional far end of the kernel was set as the opposite cumulative number.

In the proposed multiscale convolutional kernel, the target window is supposed to change with the target size. Meanwhile, the size of its surrounding background is fixed to one pixel wide, which is less sensitive to background size. When the target window slides into the target area and is just the right size for the target, it is obvious that the pixel value of the target area is significantly greater than that of the surrounding background, so the target can be enhanced to some extent. Moreover, setting the central pixel of the convolutional kernel as 0 can suppress pixel-size noise with high brightness (PNHB). The multiscale convolutional kernel with eight directions takes into account the directional information of the local region; thus, sharp edges can be eliminated.

Generally, infrared targets are brighter or darker than the corresponding surrounding environment. Most methods only consider the positive contrast between the target and its surroundings to enhance a target. The negative contrast generated by a dark target is treated as the background. To tackle the negative contrast values caused by dark targets, it is necessary to analyze the types of targets in the scene, so a weighting coefficient $E\left(x\right)$ in three application scenarios is constructed as follows: When there are only bright targets in the scene,

$$E\left(x\right)=\left\{\begin{array}{cc}1& x\ge 0\\ 0& x<0\end{array}\right..$$

(3)

When there are only dark targets in the scene,

$$E\left(x\right)=\left\{\begin{array}{cc}-1& x<0\\ 0& x\ge 0\end{array}\right..$$

(4)

When there are both dark and bright targets in the scene,

$$E\left(x\right)=\left\{\begin{array}{cc}1& x\ge 0\\ -1& x<0\end{array}\right..$$

(5)

In practical application, the application of the $E\left(x\right)$ coefficient needs to be selected according to the type of scene. Then, to calculate the difference between the target and the fixed background in the i-th direction, we calculated the cumulative directional derivatives with

$$K_i=I\ast Fi\times E(I\ast Fi),(i=1,2,\dots ,8),$$

(6)

where $Fi$ is the cumulative directional filter of the i-th direction, I is the original image, and ∗ is the convolutional operation. The minimal value of $K_i$ is

$$D_2=\underset{i=1,2,\dots ,8}{min}{K}_i.$$

(7)

Lastly, the multidirectional local difference measure can be defined as follows:

$$D=D_1\times D_2.$$

(8)

With D, our method is more robust to the variance in size between the target and its local surrounding textural structures, thus notably reducing the false alarm rate (FAR).

2.2. Entropy-Based Target Enhancement

Directional local difference measure generally emphasizes the minimal differences between the central patch and its directional neighborhoods while overlooking the overall heterogeneity between the target and its surrounding background.

Regarding the consideration above, we introduced Rényi entropy to weight the local difference measure D. Rényi entropy is represented as follows:

$$H_{\alpha }\left(x\right)=\frac{1}{1-\alpha }{log}_2\sum _{i=1}^n{p}_i^{\alpha }.$$

(9)

When $\alpha =2$, entropy was more sensitive to local brightness variation, so entropy can be defined as follows:

$$H=-{log}_2{\sum }_{i=T,B}{P}_i^2,P_i=\frac{m_i}{{\sum }_{i=T,B}{m}_i},$$

(10)

where $m_i$ represents the mean value of the pixel in the i-th $(i=T,B;T=target,B=background)$ region. Figure 3 illustrates the background edge (EB), high-brightness background (HB), and target region (T) in a real IR image and their corresponding entropy values. When the central region belongs to the target region, its diversity with the background is remarkable. Thus, entropy is small. Otherwise, when the central region is part of the background, the variation in gray values is slight, thus leading to a larger entropy value.

With H, the heterogeneity of the central region and its whole surrounding background is described. By enhancing our multidirectional local difference measure with entropy, we can obtain the saliency map C as follows:    

$$C=\frac{D}{H}.$$

(11)

For each scale, the sliding window scans the whole image from the top left to the bottom right. Lastly, the maximal C of different scales is taken as the contrast value in the final saliency map (MDLDE map).

The computational process of MDLDE is given in Algorithm 1, where L is the largest scale, and $C^l$ represents the saliency map on the l-th scale. $p_1$ and $q_1$ stand for the number of rows and columns of the input image, respectively.

Algorithm 1 MDLDE

Input: Input Image Output: MDLDE map    1: for  $l=1:L$  do    2:    Compute multidirectional local difference measure map D according to Equation (8).    3:    Compute entropy-based target enhancement map H according to Equation (10).    4:    Obtain the saliency map $C^l$ according to Equation (11).    5: end for    6: for $p=1:p_1$do    7:    for $q=1:q_1$  do    8:     ${\hat{C}}_{p,q}={max}_{l=1,2,\dots ,L}{C}^l(p,q)$    9:     Replace the current central pixel value with the obtained ${\hat{C}}_{p,q}$.  10:    end for  11: end for  12: return MDLDE map

2.3. Target Segmentation

The greater the contrast value in the saliency map is, the greater the probability of it belonging to the target. Aiming at segmenting the target from the image, the adaptive threshold of the proposed algorithm is given as follows:

$$Th=\mu +k\times \delta ,$$

(12)

where k is an adjustable parameter, $\mu$ and $\delta$ stand for the mean value and standard deviation of the MDLDE map, respectively. If the pixel’s value on the saliency map is greater than $Th$, it can be extracted as if it belongs to the target region.

2.4. Algorithm Analysis

In addition to the target itself, other elements in the infrared image, such as background, pixel-size noise with high brightness, and background edge, all affect the effect of target detection to a certain extent. However, it is important to suppress these elements while enhancing the target. Next, we discuss the proposed algorithm from the following different scenarios.

(1)

If the pixel belongs to the target (T): the target is significantly heterogeneous from the background and is usually brighter (bright target) or darker (dark target) than the background. Then, we can find that

$$D_T>0,H_T<1.$$

(13)

Thus,

$$MDLD{E}_T>0.$$

(14)

In the multidirectional convolutional kernel, the central pixel of the target window is set to 0, which means that the current pixel does not participate in the calculation. However, since the neighboring pixel value of the current pixel is usually greater than the pixel value of the background area, the value of the multidirectional local difference is greater than 0. Moreover, entropy can further enhance the target area. Therefore, the target area is effectively enhanced.

(2)

If the pixel belongs to the background (B): the gray value of the background pixel is usually equal to that of the neighborhoods, so

$$D_B\approx 0,H_B>H_T.$$

(15)

Thus,

$$MDLD{E}_B\approx 0.$$

(16)

(3)

If the pixel belongs to pixel-size noise with high brightness (PNHB):

$$D_{2PNHB}\approx 0,D_{PNHB}\approx 0,H_{PNHB}>H_T,$$

(17)

thus

$$MDLD{E}_{PNHB}\approx 0.$$

(18)

When the pixel is the central pixel of the target window, the pixel is not calculated in the multidirectional convolutional kernel, so PNHB is equal to 0 in the multidirectional local difference measure and can be eliminated eventually.

(4)

If the pixel belongs to the background edge (EB):

$$D_{EB}\approx 0,H_{EB}>H_T,$$

(19)

thus

$$MDLD{E}_{EB}\approx 0.$$

(20)

The above analysis shows that MDLDE can effectively enhance the target while suppressing other elements in the infrared image.

3. Experimental Results

3.1. Experimental Data

To validate the detection performance of the MDLDE, infrared image sequences of various scenes, including sky, sea–sky, ground, and architectural scenes, were adopted in our experiments. These scenes constituted six image sequences denoted as Sequences 1–6. The six datasets’ details are shown in

Table 1. Details of the experimental data.

Seq	Frames	Size	Target Number	Target Type	Image Detial
1	70	256 × 20	1	bright target	Sky scene High-brightness background
2	60	320 × 240	1	bright target	Low SCR, dim target PNHB
3	20	576 × 768	3	bright/dark target	Sea surface, sea–sky lines One dark target
4	184	Multiple	1	bright target	Sky, sea–sky, ground, and architectural scenes
5	200	320 × 256	1	bright target	Sky scene Thick clouds
6	200	320 × 256	1	bright target	Sky scene Heavy clutters

. Meanwhile, Figure 4 shows examples of six real infrared image sequences.

Since the size of an infrared small target is usually in the range of 3 × 3–9 × 9, the number of scales L was set to 4, and the size of each scale was chosen as n = 3, 5, 7, 9 in our experiments. To validate the proposed method, seven advanced algorithms, namely, LCM [20], MPCM [21], RLCM [32], AADCDD [26], HWLCM [33], DNGM [27], and NTFRA [34], were used to compare with the proposed MDLDE. The parameters of all comparison methods were set with reference to their own articles.

All the experiments were conducted with MATLAB R2019b on a computer with a 3.8 GHz Intel Core i7 CPU and 8 GB RAM.

3.2. Evaluation Metrics

To evaluate the detection capability of algorithms in different scenes, BSF, SCRG, and ROC curves were adopted as the evaluation metrics.

BSF and SCRG were used to measure the degree of background suppression and the validity of target enhancement, which are defined as follows:

$$BSF=\frac{{\delta }_{in}}{{\delta }_{out}+w},$$

(21)

$$SCRG=\frac{SC{R}_{out}}{SC{R}_{in}},SCR=\frac{\left|P_t-{\mu }_b\right|}{{\delta }_b+w},$$

(22)

where ${\delta }_{in}$ and ${\delta }_{out}$ denote the standard deviation of the raw image and the saliency map, respectively. $P_t$ stands for the gray peak of the target. ${\mu }_b$ and ${\delta }_b$ are the average grayscales and standard deviation of the background. w is an adjustable coefficient to avoid dividing by 0, which was set to 0.001 in this paper. $SC{R}_{in}$ and $SC{R}_{out}$ represent the SCR values of the raw image and the saliency map, respectively.

By plotting the probability of detection (PD) and FAR under different threshold settings, the ROC curve can be created. PD and FAR can be represented as follows:

$$PD=\frac{numberofdetectedtruetargets}{totalnumberofrealtargets}\times 100\%,$$

(23)

$$FAR=\frac{numberofdetectedfalsetargets}{numberoftestedframes}\times 100\%.$$

(24)

3.3. Experimental Results and Comparisons

The 3D mesh view of example images and their saliency maps derived from different algorithms is shown in Figure 5. Since most infrared small-target detection methods can only be applied to bright targets, the $E\left(x\right)$ of the proposed method is set only to detect the bright targets in all the six image sequences. The saliency maps of all algorithms were normalized to [0 1] in the experiment. Figure 5a1–a6 show the 3D mesh view of the raw image. As shown in Figure 5b1–b6, LCM could not suppress the background well in all six image sequences. It enhanced the target region to some extent, but was still not significant enough. In Sequence 1, MPCM significantly enhanced the hole-shaped dark areas in thick clouds. Except Sequence 3, HWLCM and RLCM also obtained strong residual clutter in the other five sequences, thus leading to low contrast, which is an obstacle to separating the target from the background. For Sequence 2, in the scene with heavy background clutter and a low signal clutter ratio, MPCM generated much background clutter in the saliency map. As shown in Figure 5h2, NTFRA did not enhance the dim target since the saliency map was flat. AADCDD significantly enhanced the nontarget region in Sequence 5. Our algorithm had strong capabilities for background suppression while enhancing the target, so it could achieve better results in real infrared image sequences. In the scene with only bright targets, all algorithms could enhance the target region to different degrees, but when dark targets appeared in the scene (see Figure 5a3–i3), except MPCM, most of the algorithms could not detect the dark target. In this case, we also set E(x) of the scene with bright and dark targets to enhance the target area in the corresponding environment. Overall, the proposed method performed best, enhancing the target while effectively suppressing the background.

The ROC curves of different algorithms in various scenes are presented in Figure 6. For Sequence 1, there were heavy clouds in the background. When FAR was 0, NTFRA, AADCDD, and MDLDE algorithms could detect all targets, and LCM and RLCM had slightly higher FAR values. In Sequence 2, because the target in the scene was dim and had continuous high-brightness noise in some images, none of the other seven methods could detect all the targets, but our method had a 100% PD while obtaining a low FAR. In Sequence 3, MPCM achieved superior detection results because it can detect dark and bright targets. Sequence 4 mainly consisted of several single frames of sky-scene images, including sea–sky, ground, and architectural scenes. In Sequence 4, the proposed method performed best. RLCM, AADCDD, DNGM, and MPCM also achieved good results, while the HWLCM, LCM, and NTFRA algorithms had higher FAR when the PD was relatively low. The target brightness in Sequence 5 was low, and the proposed method could achieve the highest detection rate and the lowest false alarm rate, which has significant advantages compared with other methods. In Sequence 6, the ROC curves of the proposed method were in the upper left corner, indicating better detection performance. Figure 7 shows the area under the curve(AUC) of different algorithms in six sequences. The figure shows that the proposed method achieved the maximal AUC value in Sequences 1, and 4–6, and obtained a suboptimal value in Sequence 2. The experimental results of ROC curve indicate that the proposed algorithm is more robust in a variety of scenarios.

Table 2. SCRG values of different algorithms.

Seq	LCM	MPCM	RLCM	AADCDD	HWLCM	DNGM	NTFRE	Proposed
1	0.49	11.86	1.08	22.46	1.05	45.58	3.67	65.80
2	0.46	5.19	1.10	18.94	1.78	104.97	0.00	30.50
3	2.09	112.43	2.80	39.37	4.55	44.68	90.38	53.13
4	0.43	12.90	2.27	26.41	1.35	52.41	10.27	47.49
5	1.72	81.03	28.09	123.42	25.66	198.57	48.44	245.21
6	1.82	204.09	46.72	239.94	28.58	245.14	114.96	295.19

and

Table 3. BSF values of different algorithms.

Seq	LCM	MPCM	RLCM	AADCDD	HWLCM	DNGM	NTFRA	Proposed
1	125.11	1867.96	1080.78	2609.87	705.77	3419.44	1481.26	3679.54
2	58.23	895.20	393.01	1951.16	577.41	2057.85	16,225.85	2249.11
3	458.11	13,356.34	5373.62	17,332.80	3170.60	17,928.35	7205.82	20,156.38
4	191.29	2871.36	1504.94	3648.15	1815.58	4471.42	1330.14	4115.80
5	263.01	6028.28	1902.47	7556.93	1861.01	9600.54	3927.31	11,797.75
6	223.35	8803.59	2546.38	10,480.11	1570.30	11,037.57	6135.20	12,725.94

show the SCRG and BSF values of different algorithms, respectively. The maximal SCRG and BSF values are marked in bold and are underlined in the table. In Sequence 2, NTFRA achieved the maximal BSF value, while the SCRG value was 0 because NTFRA cannot effectively detect dim targets, and the saliency map of NTFRA is a plane with a value of 0. The proposed algorithm obtained the maximal SCRG values in Sequences 1 and 4–6, and suboptimal values in Sequence 2. Meanwhile, it obtained the maximal BSF values in Sequences 1, 3, 5, and 6, and suboptimal values in Sequences 2 and 4. The performance of the proposed method in BSF and SCRG showed that it had superior background suppression and background enhancement.

Table 4. Time consumption of different algorithms (in seconds).

Seq	LCM	MPCM	RLCM	AADCDD	HWLCM	DNGM	NTFRE	Proposed
1	0.3407	0.3364	3.1052	0.3168	3.1561	0.3304	2.2512	0.3526
2	0.3602	0.3603	4.8237	0.3422	4.8577	0.3572	3.3301	0.3526
3	0.5575	0.6147	29.8228	0.5033	28.0230	0.6475	10.3316	0.8310
4	0.4092	0.4137	3.5212	0.3970	3.5906	0.405	3.3693	0.4509
5	0.4208	0.3761	4.8008	0.3610	4.8589	0.3939	2.9728	0.4353
6	0.3797	0.3829	4.7386	0.3646	4.7686	0.3896	3.0696	0.4353

presents the time consumption of different algorithms. Compared with LCM, MPCM, AADCDD, and DNGM, the proposed method requires more running time, but the difference is relatively small, which means that the time consumption of the proposed method is at the same level as those of LCM, MPCM, and AADCDD. The detection speed of RLCM, HWLCM, and NTFRA is slower, requiring more time. The running time of the proposed algorithms is primarily constituted by the following two parts: multidirectional local difference measure and entropy computation. Theoretically, the computational complexity of the MDLDE algorithm is approximately $O\left(LMN{C}^2\right)$, where M and N represent the number of rows and columns in the raw image, respectively. L is the scale number. C is the size of each scale. LCM, MPCM, and AADCDD had the same time complexity as the proposed method. The time complexity of RLCM and HWLCM is $O(L{C}^{2}log{C}^{2}MN)$. The time complexity of NTFRA and DNGM is $O(n_1{n}_2{n}_{3}log\left(n_1{n}_2{n}_3\right)+{\sum }_{i=1}^{3}min(n_i,n_{i+1}))$ and $O\left(MN{C}^2\right)$, respectively. In conclusion, the proposed algorithm has medium computational complexity.

4. Conclusions

In this paper, an IR small-target detection method based on the local difference measure combined with entropy was proposed. First, the minimal difference between the central patch and its directional neighborhoods was calculated with a novel multiscale convolutional kernel. Second, the overall heterogeneity between the target and its surrounding background was computed via entropy, and the saliency map was then obtained. Finally, an adaptive threshold was adopted to extract the target region from the saliency map. By utilizing this new detection model, we could suppress clutters and noise and effectively enhance the targets. Extensive experimental results illustrate that our algorithm could robustly detect IR small targets under complex backgrounds.

However, the proposed algorithm has limitations in time efficiency. Potential further improvement may include applying a GPU to accelerate the procedures.