UCDnet: Double U-Shaped Segmentation Network Cascade Centroid Map Prediction for Infrared Weak Small Target Detection

Abstract

In recent years, the development of deep learning has brought great convenience to the work of target detection, semantic segmentation, and object recognition. In the field of infrared weak small target detection (e.g., surveillance and reconnaissance), it is not only necessary to accurately detect targets but also to perform precise segmentation and sub-pixel-level centroid localization for infrared small targets with low signal-to-noise ratio and weak texture information. To address these issues, we propose UCDnet (Double U-shaped Segmentation Network Cascade Centroid Map Prediction for Infrared Weak Small Target Detection) in this paper, which completes “end-to-end” training and prediction by cascading the centroid localization subnet with the semantic segmentation subnet. We propose the novel double U-shaped feature extraction network for point target fine segmentation. We propose the concept and method of centroid map prediction for point target localization and design the corresponding Com loss function, together with a new centroid localization evaluation metrics. The experiments show that ours achieves target detection, semantic segmentation, and sub-pixel-level centroid localization. When the target signal-to-noise ratio is greater than 0.4, the IoU of our semantic segmentation results can reach 0.9186, and the average centroid localization precision can reach 0.3371 pixels. On our simulated dataset of infrared weak small targets, the algorithm we proposed performs better than existing state-of-the-art networks in terms of semantic segmentation and centroid localization.

1. Introduction

Infrared small target detection technology, as the main technical support for surveillance and reconnaissance [1,2], precise localization [3,4,5], and attitude estimation [6], has been widely applied in various fields. In the field of surveillance and reconnaissance, such as drone tracking and search and rescue operations, not only the rough detection of infrared targets is required but also precise segmentation and centroid localization of the targets. This is essential for effective prediction of target motion trajectories, thereby enabling early warning and appropriate measures. In the field of the Internet of Things (IoT), such as smart transportation and smart agriculture, precise detection is a prerequisite for achieving perception and decision-making. In the aforementioned application scenarios, targets often occupy a small number of pixels in the image and suffer from issues such as texture loss and low signal-to-noise ratio.

Challenges:

On one hand, as the distance between the imaging device and the target increases, the size of the target in the image becomes smaller, and the target appears in a faint and weak state. Existing semantic segmentation algorithms perform well in segmenting large-sized or clearly bounded targets, but they show limited effectiveness in segmenting infrared small targets with low signal-to-noise ratio and weak texture information.

On the other hand, existing infrared weak small target detection methods focus on bounding box detection and pay less attention to the problem of accurate centroid localization of these targets. When the infrared small targets have irregular shapes and undergo continuous changes in posture, the center of the bounding box cannot effectively represent the centroid of the target. Existing deep learning-based methods for predicting target centroids are still limited to pixel-level predictions, only able to predict which pixel the centroid belongs to in the image. In certain specific scenarios, a deviation of one pixel in predicting the target centroid can result in several meters of actual distance, and this deviation may be further amplified as the process continues.

Existing Methods:

To date, there are various traditional methods for infrared small target detection. Common approaches include: (1) filter-based methods [7,8,9], (2) methods based on the human visual system [10,11,12,13,14], and (3) methods based on low-rank sparse recovery [15,16]. However, filter-based detection algorithms can only suppress uniform background to a certain extent and fail to address complex background problems, resulting in low detection performance and poor robustness. Methods based on the human visual system rely on contrasting the differences between small target regions and their surrounding areas, making them unsuitable for detecting weak small targets. Low-rank sparse recovery methods can improve detection precision by applying certain constraints. However, in infrared images with weak small targets and complex backgrounds, some strong clutter signals may be as sparse as the target signal, leading to increased false alarm. Overall, traditional algorithms heavily rely on manual craftsmanship, lack adaptability, and are difficult to be widely applied in infrared target detection in various application scenarios.

Recent approaches have turned to deep learning methods to address these issues. For example, Liu et al. [17] developed a CNN-based method for infrared small target detection. Y. Dai et al. [18] encoded spatial information and proposed an Asymmetric Context Module (ACM). They also introduced the SIRST dataset, which has been widely used in the field of infrared small target detection. Other researchers, such as Q. Hou et al. [19], proposed a robust infrared small target detection network (RISTDnet) based on deep learning. They constructed a feature extraction framework that combines handcrafted features and convolutional neural networks, established a mapping network between the feature map and the likelihood of small targets in the image, and applied threshold segmentation to identify real targets. X. Zhou et al. [20] unfolded the sparse low-rank regularization model into a deep neural network. They used the infrared patch-image (IPI) model to convert the original infrared image into patch images through local patch construction. Zhao et al. [21] proposed a detection pattern based on generative adversarial networks (GAN) to automatically learn the features of the target and directly predict the intensity of the target.

Our Contribution:

In this paper, to address the sub-pixel-level accurate centroid localization of infrared weak small targets, we propose UCDnet (Double U-shaped Segmentation Network Cascade Centroid Map Prediction for Infrared Weak Small Target Detection), and its prediction results are shown in Figure 1. The algorithm takes the input infrared image and passes it through a semantic segmentation subnet for feature extraction, feature fusion, and dense prediction, outputting pixel-level predictions (i.e., semantic segmentation results) of the targets (Figure 1a). Meanwhile, the input infrared image is also fed into the centroid localization subnet, which generates a centroid map with target centroid position information. The pixel-level predictions from the semantic segmentation subnet are mapped onto the centroid map, allowing the centroid map to retain only the results corresponding to the target’s pixel locations. Finally, through decoding operations, accurate centroid results of the targets can be obtained (Figure 1b). Additionally, we have created an infrared simulation dataset specifically tailored for weak small targets.

Our contributions can be summarized as follows:

We have designed a novel sub-pixel-level centroid localization method for infrared weak small targets. We innovatively propose the concept and method of centroid map prediction, addressing the challenge of accurately locating the centroid of point targets. Our method utilizes a lightweight fully convolutional structure and incorporates a novel Com loss function. We also introduce a new evaluation metric for centroid localization. Experimental results demonstrate that our average centroid localization precision achieves 0.3371 pixels when the signal-to-noise ratio of the targets is above 0.4.

We have proposed a feature extraction network for semantic segmentation of infrared weak small targets. Our method employs a novel double U-shaped structure, which effectively combines the feature extraction module and attention module to finer predict the easily overlooked edge pixels, which is crucial for precise centroid localization. Experimental results show that our semantic segmentation IoU achieves 0.9186 when the signal-to-noise ratio of the targets is above 0.4.

We have simulated a series of infrared weak small target datasets for training, validation, and testing purposes. These datasets include various forms of infrared small targets. To replicate real-world scenarios, we have introduced multiple types of vertical line noise and Gaussian noise in the images. We have also specifically simulated the dim and weak states of targets when they pass through cloud, which pose challenges for detection algorithms.

2. Materials and Methods

The main idea of UCDnet is to construct an infrared small target detection network that achieves dense prediction and sub-pixel-level centroid localization. Its network structure is shown in Figure 2 and consists of two parts: the semantic segmentation subnet and the centroid localization subnet. Section 2.1 introduces the simulation content of the dataset. Section 2.2 presents the proposed semantic segmentation method, while Section 2.3 describes the proposed sub-pixel-level centroid localization method. Finally, Section 2.4 discusses the loss function and training method used during network training.

2.1. Dataset Simulation

The dataset used in this paper is simulated based on satellite imagery (with a size of around 10,000 × 10,000 pixels) collected by meteorological satellites. The simulated targets are generated based on the grayscale distribution of real targets, and a total of 125 targets were simulated. The extraction of target features from infrared images is closely related to background information and noise [22]. The image data from different backgrounds exhibit significant differences. For meteorological satellite images, cloud cover is the main source of interference, and the presence of high-intensity noise further complicates target detection. We simulated 8 to 14 frames of motion trajectories for each target individually. In order to make the dataset more representative of real-world scenarios, slight variations in the shape and pixel values of the targets were introduced within each trajectory, resulting in a total of 1306 images. Finally, random cropping was performed on these images to obtain sub-images of size 256 × 256 pixels containing the targets (the simulation process of the infrared weak small target dataset is illustrated in Figure A1 in Appendix A). The size of the targets ranged from 6 × 4 pixels to 10 × 13 pixels, and the signal-to-noise ratio (SNR) of the images varied from 0.4 to 6.4. It can also be observed from Figure 3 that the SNR of the image complies with the central limit theorem. Figure 4 illustrates the range of target sizes, with blue dots representing the number of targets for each size. Figure 5 showcases the approximate shapes of the targets.

2.2. Semantic Segmentation Subnet

The semantic segmentation subnet consists of a feature extraction backbone network and a feature fusion prediction branch. The feature extraction backbone network adopts a double U-shaped structure, focusing on extracting edge features of the targets to handle low signal-to-noise ratio situations. It incorporates embedded attention modules to enhance segmentation precision. The feature fusion prediction branch performs feature fusion at different hierarchical levels and conducts per-pixel binary classification based on the fused results to generate semantic segmentation masks.

The double U-shaped network structure is shown in Figure 6. Inspired by the U-Net [23] and DNAnet [24] network structures, we have made improvements to the main feature extraction network. It follows the overall Encoder–Decoder structure, which is similar to the combination of two large U-shaped networks and one small U-shaped network. The two large U-shaped networks perform four downsampling operations, where the shallow network extracts more target texture information, and the deep network allows for a larger field of view and captures deeper semantic information [25]. The small U-shaped network just performs two downsampling operations, reserving more target positional information than bigger ones. Compared to the U-Net network, our approach uses more feature concatenation to achieve precise extraction of edge features.

In contrast to most existing infrared small target semantic segmentation methods, inspired by the DNAnet network structure, each layer of our main feature extraction network incorporates a specific attention module [26]. Each attention module in a layer includes channel attention and spatial attention components, enabling adaptive learning of the importance of different channels and spatial positions in the image. The channel attention dynamically adjusts the impact of each channel in the output feature map, while the spatial attention dynamically searches for the precise location of the targets on the feature map. The combination of two attentions helps the network pay more attention to the positional information of small targets and some detailed edge information in each feature map. The feature extraction network determines the overall position of the infrared small targets based on their grayscale and contrast information, while the attention modules focus more on the edge information of the targets.

The basic and core component of the entire network is the Res_Attention Block. In each layer, the input feature map undergoes a series of operations, including a 3 × 3 convolution, normalization, activation function, another 3 × 3 convolution, and normalization, followed by a hybrid attention mechanism. The output of this series of operations is then added to the original input to obtain the output of the Res_Attention Block. The hybrid attention mechanism consists of a channel attention mechanism followed by a spatial attention mechanism. For an input F, the equations for the channel attention mechanism (Mc(F)) and the spatial attention mechanism (Ms(F)) can be expressed as:

$$M_c\left(F\right)=\sigma \left(MLP\left(AvgPool\left(F\right)\right)+MLP\left(MaxPool\left(F\right)\right)\right)$$

(1)

$$M_s\left(F\right)=\sigma \left(f^{7\times 7}\left(\left[AvgPool\left(F\right);MaxPool\left(F\right)\right]\right)\right)$$

(2)

where σ represents the sigmoid operation, MLP represents a convolution operation with a kernel size of 1 × 1, AvgPool represents average pooling, MaxPool represents maximum pooling, [;] represents channel concatenation, and $f^{7\times 7}$ represents a convolution with a kernel size of 7 × 7.

2.3. Centroid Localization Subnet

The centroid localization subnet (shown in Figure 7) consists of the centroid feature extraction network and the mapping module. The feature extraction network includes a hybrid attention mechanism, downsampling, and upsampling structures. The hybrid attention mechanism focuses on learning the relationship between the image’s grayscale, contrast, and target centroids. The mapping module performs pixel-wise multiplication, multiplying the binary segmentation result with the centroid map. This operation filters out the parts that do not belong to the target, resulting in more accurate centroid localization.

The centroid localization subnet predicts the “floating-point” centroid map (as shown in Figure 8) based on the semantic segmentation results to calculate the target centroid. The centroid map, which is different from the heatmap proposed in CenterNet [27], is first introduced and applied by us. In the centroid map, the value of each pixel represents the proximity to the target centroid. To achieve sub-pixel-level centroid localization, we do not designate a specific pixel as the centroid in both the ground truth label map and the predicted map. Instead, we employ an indirect constraint method to obtain the floating-point centroid position (sub-pixel-level). Specifically, during the training phase, the labels for the centroid localization subnet are calculated using a two-dimensional Gaussian function [28]. When the target centroid position is (x₀, y₀), the label value at any position in the centroid map can be obtained using the two-dimensional Gaussian function as follows:

$$V_{ij}={\mathrm{e}}^{-\frac{{\left(x_0-j\right)}^2+{\left(y_0-i\right)}^2}{2}}$$

(3)

where j represents the current pixel’s column, i represents the current pixel’s row, and V_ij represents the corresponding label value at that position. As shown in Figure 8, when the centroid position is x₀ = 3.3, y₀ = 4.6, we can calculate the values of V₄₃, V₅₃, and V₅₄ as follows:

$$V_{43}={\mathrm{e}}^{-\frac{{\left(3.3-3\right)}^2+{\left(4.6-4\right)}^2}{2}}\approx 0.79852 (\mathrm{yellow} \mathrm{pixel})$$

$$V_{53}={\mathrm{e}}^{-\frac{{\left(3.3-3\right)}^2+{\left(4.6-5\right)}^2}{2}}\approx 0.88250 (\mathrm{red} \mathrm{pixel})$$

$$V_{54}={\mathrm{e}}^{-\frac{{\left(3.3-4\right)}^2+{\left(4.6-5\right)}^2}{2}}\approx 0.72253 (\mathrm{blue} \mathrm{pixel})$$

During the prediction phase, the actual centroid positions are calculated based on the predicted centroid map, following these steps:

• Firstly, determine the number of targets N in our predicted centroid map, which corresponds to N clusters of pixels.
• For each cluster of pixels mentioned above, select the maximum value $V_{i_1{j}_1}$ from our predicted results as the centroid reference point $A_{i,i\in \left[1, N\right]}$ (represented by the red position in Figure 8).
• Take the centroid reference point $A_{\mathrm{i}}$ as the center and find the maximum value $V_{i_2{j}_2}$ within the vertical neighborhood ${\mathrm{N}}_4(A_{\mathrm{i}})$ to serve as the centroid vertical correction point $B_{i,i\in \left[1, N\right]}$ (represented by the yellow position in Figure 8).
• Similarly, take the centroid reference point $A_{\mathrm{i}}$ as the center and find the maximum value $V_{i_3{j}_3}$ within the horizontal neighborhood ${\mathrm{N}}_4(A_{\mathrm{i}})$ to serve as the centroid horizontal correction point $C_{i,i\in \left[1, N\right]}$ (represented by the blue position in Figure 8).
• Combining the centroid reference point $A_{\mathrm{i}}$, the vertical correction point $B_{\mathrm{i}}$, and the horizontal correction point $C_{\mathrm{i}}$, calculate the sub-pixel-level precise positions x₀ and y₀ of each centroid using the following equations:

  $$V_{i_1{j}_1}={\mathrm{e}}^{-\frac{{\left(x_0-j_1\right)}^2+{\left(y_0-i_1\right)}^2}{2}}$$

  (4)

  $$V_{i_2{j}_2}={\mathrm{e}}^{-\frac{{\left(x_0-j_2\right)}^2+{\left(y_0-i_2\right)}^2}{2}}$$

  (5)

  $$V_{i_3{j}_3}={\mathrm{e}}^{-\frac{{\left(x_0-j_3\right)}^2+{\left(y_0-i_3\right)}^2}{2}}$$

  (6)

  where i₁ = i₃ and j₁ = j₂, the above three equations can be used to obtain the following:

  $$x_0=\frac{2\ln \frac{V_{i_3{j}_3}}{V_{i_1{j}_1}}-\left(j_1^2-j_3^2\right)}{2\left(j_3-j_1\right)}$$

  (7)

  $$y_0=\frac{2\ln \frac{V_{i_2{j}_2}}{V_{i_1{j}_1}}-\left(i_1^2-i_2^2\right)}{2\left(i_2-i_1\right)}$$

  (8)
• Repeat the above steps 2 to 5 until all N predicted targets have been traversed.

In summary, compared to directly predicting the centroid positions (x₀, y₀) using the network, predicting the centroid map makes the network training more easily convergent (due to the presence of more constraints). Furthermore, compared to predicting bounding boxes and then post-processing to obtain the centroids, the proposed method of predicting the centroid map and then extracting the centroids is more efficient and accurate.

2.4. Loss Function

The loss function used in our method consists of two parts. For the semantic segmentation part, as shown in Equation (9), we employ Dice Loss as the loss function. For the centroid localization part, as shown in Equation (10), we have developed our own loss function based on the MSE Loss, which we refer to as the Center of Mass Loss (Com Loss).

$$\mathrm{Dice} \mathrm{Loss}=1-2\frac{\left|A\cap B\right|}{\left|A\right|+\left|B\right|}$$

(9)

$$\mathrm{Com} \mathrm{Loss}=\left\{\begin{array}{c}{\left(y-p\right)}^2 if y>0.1\\ {\left(y-p\right)}^2{\left(1-y\right)}^4 other\end{array}\right.$$

(10)

In Equation (9), A and B represent the semantic segmentation label (per-pixel) and the predicted semantic segmentation results (per-pixel), respectively. In Equation (10), y represents the centroid map label (per-position), and p represents the predicted centroid map (per-position).

To improve the algorithm’s performance, we first optimize the semantic segmentation results. During the first 10 epochs, we only calculate the loss value for semantic segmentation. After the 10th epoch, we add to calculate the loss value for the centroid localization module. The weights assigned to the loss values of the semantic segmentation and centroid localization parts follow specific function distributions: $\frac{1}{1+e^{-0.1\left(60-epoch\right)+0.5}}+1$ for the semantic segmentation part and $\left(1-\frac{1}{1+e^{-0.1\left(60-epoch\right)+0.5}}\right)\times 0.8+0.2$ for the centroid localization part (shown in Figure 9). This approach improves the overall efficiency of the network by filtering out unnecessary computations and further enhances the performance of the centroid localization module.

3. Results

This section presents the experimental results of the proposed method in this paper, along with some comparative results, which demonstrate that our method outperforms other approaches. Section 3.1 provides a detailed introduction to the evaluation metrics used in this study. Section 3.2 presents all the experimental procedures and analyses.

3.1. Evaluation Metrics

3.1.1. Semantic Segmentation Evaluation Metrics

Since there is only one semantic segmentation class in this paper, the main evaluation metrics used are IoU, Precision, and Recall. In semantic segmentation, the definition of IoU is the ratio of the intersection to the union of the ground truth and predicted sets. This ratio can be expressed as the ratio of True Positive (TP) to the sum of TP, False Positive (FP), and False Negative (FN), which represents the overall segmentation precision. Precision is defined as the ratio of the intersection to the predicted segmentation set. This ratio can be expressed as the ratio of TP to the sum of TP and FP, reflecting the precision of the predicted results. Recall is defined as the ratio of the intersection to the ground truth segmentation set. This ratio can be expressed as the ratio of TP to the sum of TP and FN, reflecting the recall or sensitivity of the predicted results. TP, FP, FN, and TN are shown in Figure 10.

$$\mathrm{IoU}=\frac{\mathrm{TP}}{\mathrm{FP}+\mathrm{FN}+\mathrm{TP}}$$

(11)

$$\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{FP}+\mathrm{TP}}$$

(12)

$$\mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{FN}+\mathrm{TP}}$$

(13)

3.1.2. Centroid Localization Evaluation Metrics

If there is only one target in each image, we can directly use the Euclidean distance between the predicted centroid and the actual centroid of the target as the evaluation metric for centroid localization. However, considering the cases of multiple targets or false alarms, this paper introduces a novel centroid localization evaluation metric. Specifically, clustering is performed with the real centroids as centers, and all detected centroids are assigned to the nearest real centroid. Then, the sum of distances between all predicted centroids and their closest real centroids is calculated. This distance sum is divided by the smaller value between the number of real centroids and the number of detected centroids. The calculation equation is as follows:

$$\mathrm{mDis}=\frac{{\sum }_{i=0}^m\underset{0\ll j\ll n}{\min }\left(\sqrt{{\left(x_i-x_j\right)}^2+{\left(y_i-y_j\right)}^2}\right)}{\min \left(m,n\right)}$$

(14)

where n represents the number of actual target centroids, m represents the number of detected target centroids, ${\mathrm{x}}_{\mathrm{i}}$ represents the horizontal coordinate of the detected target centroid, ${\mathrm{x}}_{\mathrm{j}}$ represents the horizontal coordinate of the real target centroid, ${\mathrm{y}}_{\mathrm{i}}$ represents the vertical coordinate of the detected target centroid, and ${\mathrm{y}}_{\mathrm{j}}$ represents the vertical coordinate of the real target centroid.

3.2. Experimental Results and Analysis

3.2.1. Experimental Setup

In this experiment, the input image size of the network is 256 × 256 pixels, and the batch size is set to 16. The training and testing dataset are divided approximately in a ratio of 5:1. The semantic segmentation subnet in the experiment utilizes Dice Loss as the loss function, while the centroid localization subnet employs Com Loss as the loss function. The optimization method for the loss function is Adaptive Gradient (AdaGrad), which is a first-order optimization method [29]. The advantage of using AdaGrad is that it allows the manual adjustment of learning rate to be replaced by setting hyperparameters. The hardware device mainly used in the experiment is an NVIDIA RTX 3090 graphics card with 24 GB of VRAM. The software programs primarily used include the deep learning framework PyTorch, the plotting tool Matplotlib, the image processing library OpenCV, and the scientific computing library NumPy.

3.2.2. Comparison with Other State-of-the-Art Methods

In this section, we evaluate our implementation by comparing it with traditional image processing method MLCM [30], classic semantic segmentation network U-Net [23], and the latest semantic segmentation networks DNAnet [24], HRNet [31], and MTU-Net [32]. We perform a quantitative and qualitative analysis of the experimental results.

(1)

Quantitative Analysis

Table 1. Comparison of our semantic segmentation algorithm with five other algorithms.

Semantic Segmentation Algorithm	IoU	Precision	Recall	F1-Score	GFLOPs
MLCM [30]	0.1810	0.1810	1.0000	0.3065	-
U-Net [23]	0.4905	0.6435	0.5774	0.6087	46.02
HRNetv2-w18 [31]	0.7216	0.8307	0.8443	0.8374	4.64
MTU-Net [32]	0.8712	0.9403	0.9229	0.9315	5.48
DNAnet [24]	0.8862	0.9566	0.9236	0.9398	14.05
UCDnet (ours)	0.9186	0.9673	0.9480	0.9576	15.56

compares the performance of our algorithm and the other five state-of-the-art algorithms in terms of evaluation metrics such as IoU, Precision, Recall, F1-score, and GFLOPs. The experimental results demonstrate that our method ranks first in all evaluation metrics.

(2)

Qualitative Analysis

We visualized the semantic segmentation results of three classical algorithms and our UCDnet. As shown in Figure 11, yellow annotations represent false alarms, red annotations represent missed detections, green annotations represent correctly predicted targets, and blue annotations indicate zoomed-in views of the targets. Due to the influence of factors such as complex background in infrared images and noise resembling target shapes, the existing algorithms and network detection performance are unsatisfactory. From the prediction results, it can be observed that the traditional MLCM algorithm has significant limitations, with excessive false alarms and almost complete loss of target edge information. The U-Net misses some targets in particular situations but retains some edge information compared to traditional algorithms. The DNAnet’s prediction results show a good segmentation of the edges, but still have some errors compared to the ground truth. However, it is evident that our algorithm’s predicted target edges closely match the ground truth, demonstrating high robustness without false alarms or missed targets. Our algorithm outperforms other comparative approaches in terms of detection and segmentation performance.

3.2.3. Analysis of UCDnet Feature Extraction Network

(1)

Quantitative Analysis

To validate the superior performance of the proposed network structures in feature extraction compared to other similar network structures, we conducted the following ablation experiments on the main feature extraction network of the semantic segmentation subnet. Throughout the ablation experiments, we kept the structure of the centroid localization subnet unchanged and investigated the impact of variations in the structure of the semantic segmentation subnet on the network performance. As illustrated in Figure 12, “ablation study1” represents the network structure of a single U-shaped structure shown in Figure 12a, “ablation study2” represents a network structure similar to double U-shaped structure shown in Figure 12b, “ablation study3” represents a network structure based on Figure 12b with an added attention module shown in Figure 12c, and “ablation study4” represents a network structure with the addition of a feature fusion strategy based on Figure 12c. Training each of the four networks for 100 epochs yielded the results shown in

Table 2. Comparison of results from four ablation studies.

Semantic Segmentation Algorithm	IoU	Precision	Recall
single U-shaped structure (ablation study1)	0.8665	0.9419	0.9155
double U-shaped structure (ablation study2)	0.8728	0.9600	0.9057
double U-shaped structure + attention module (ablation study3)	0.9070	0.9599	0.9428
double U-shaped structure + attention module + feature fusion (ours)	0.9186	0.9673	0.9480

and Figure 13.

From Table 2, it can be observed that the double U-shaped network structure achieves higher IoU, Precision, and Recall compared to the single U-shaped network structure. By incorporating attention modules into the network, the IoU of the network in the test results improves by two percentage points, while Recall increases by three percentage points. Additionally, the introduction of a feature fusion strategy leads to an IoU improvement of over one percentage point. Considering the high IoU requirement for the output results in this study, the network structure depicted in Figure 12d is adopted.

According to the curve shown in Figure 13, it can be observed that the network structure proposed in this paper consistently outperforms the other three ablation study network structures in terms of the evaluation metric IoU. Furthermore, the IoU values exhibit minimal fluctuations with increasing epochs. Considering the comprehensive set of ablation experiments, the proposed network structure in this paper demonstrates stronger feature extraction capabilities and better stability, while exhibiting superior learning ability for capturing detailed target features.

(2)

Qualitative Analysis

From Figure 14, it can be observed that in the encoding process of the first U-shaped structure, as the downsampling iterations increase, the network pays more attention to the brighter regions in the image. In the decoding process, with the increase in upsampling iterations, the network can further distinguish between the target and the background. In the encoding process of the second U-shaped structure, FM (2,1) serves both as an encoding and decoding component, enabling more accurate discrimination between the background and the target. With an increase in downsampling iterations during the encoding process, FM (4,1) is able to fully differentiate the background regions surrounding the target. Comparing FM (0,2) and FM (0,3), although the rough shape of the target can already be observed at the output of the first U-shaped structure, the output of the second U-shaped structure is closer to the actual target.

3.2.4. Analysis of Centroid Localization Network

(1)

Quantitative Analysis

As shown in

Table 3. Comparison of four centroid localization methods.

Centroid Localization Methods	Localization Precision (Pixels)
center of bounding box	0.6096
gray value centroid method	0.5775
squared weighted centroid method	0.5380
UCDnet (Ours)	0.3371

, we compared the performance between three classical centroid localization methods and our method. Our approach demonstrated the best localization precision, outperforming even the suboptimal squared weighted centroid method by 0.2009 pixels.

The solution method for the gray value centroid method is as follows: For image block B, determine the center of the minimum bounding rectangle enclosing the target. Establish a Cartesian coordinate system with this center as the origin. Then, calculate the centroid using the following equations:

$$\mathrm{m}00={{\displaystyle \sum }}_{x,y\in B}I\left(x,y\right)$$

(15)

$$\mathrm{m}10={{\displaystyle \sum }}_{x,y\in B}x\times I\left(x,y\right)$$

(16)

$$\mathrm{m}01={{\displaystyle \sum }}_{x,y\in B}y\times I\left(x,y\right)$$

(17)

$$\mathrm{C}=\left(\frac{m10}{m00},\frac{m01}{m00}\right)$$

(18)

where $I\left(x,y\right)$ represents the pixel value at position (x, y), and C represents the coordinates of the centroid obtained on the Cartesian coordinate system established with the center of the image block.

The solution method for the squared weighted centroid method is as follows: For image block B, determine the center of the minimum bounding rectangle enclosing the target. Establish a Cartesian coordinate system with this center as the origin. Then, calculate the centroid using the following equations:

$$\mathrm{m}00={{\displaystyle \sum }}_{x,y\in B}{I}_{\left(x,y\right)}^2$$

(19)

$$\mathrm{m}10={{\displaystyle \sum }}_{x,y\in B}x\times I_{\left(x,y\right)}^2$$

(20)

$$\mathrm{m}01={{\displaystyle \sum }}_{x,y\in B}y\times I_{\left(x,y\right)}^2$$

(21)

$$\mathrm{C}=\left(\frac{m10}{m00},\frac{m01}{m00}\right)$$

(22)

where $I\left(x,y\right)$ represents the pixel value at position (x, y), and C represents the coordinates of the centroid obtained in the Cartesian coordinate system established with the center of the image block.

(2)

Qualitative Analysis

We selected two images from the dataset and fed them into our UCDnet. We saved the final centroid localization results. Then, we extracted the target pixel blocks from the images, enlarged them eight times, and mapped our centroid localization results onto the magnified images (as shown in Figure 15). The centroid localization precision achieved sub-pixel-level precision.

3.2.5. Discussion

Currently, numerous classical target detection algorithms have been applied in the field of infrared weak small target detection. We selected five representative algorithms, namely, MLCM, U-Net, HRNet, MTU-Net, and DNANet, for comparative experiments with our proposed method and obtained outstanding results. The IoU of our semantic segmentation results reached a leading level because of the proposed double U-shaped structure with attention mechanism. Furthermore, we innovatively introduced a deep learning-based approach for sub-pixel-level centroid localization, significantly improving the centroid localization precision compared to existing methods. Extensive experiments validated that our proposed algorithm exhibits favorable performance and robustness.

Additionally, to verify the applicability and generalization ability of our proposed algorithm beyond the field of surveillance and reconnaissance, we evaluated our algorithm on three different publicly available datasets from diverse fields. On an industrial field dataset for motor magnetic tile defect detection, our algorithm demonstrated excellent segmentation results even when facing weak texture information of motor magnetic defects (see experimental results in the first row of Figure A2 in Appendix A). On a medical field dataset for cell semantic segmentation [33,34], our algorithm accurately extracted edges of irregularly shaped cells (see experimental results in the second row of Figure A2 in Appendix A). On an agricultural field dataset for olive fruit semantic segmentation, our algorithm achieved precise segmentation for small-sized olive fruits, remaining unaffected by factors like crowding and occlusion (see experimental results in the third row of Figure A2 in Appendix A). These supplementary experiments further demonstrate the high robustness of our proposed algorithm. Our algorithm proves to be effective not only on our simulated dataset but also on various datasets from other fields, showcasing its excellent performance.

The proposed algorithm in this paper also has some limitations: (1) Point targets in infrared images are prone to confusion with high-frequency noise and similar objects. The method proposed in this paper focuses on fine segmentation and localization but requires further improvement in false alarm removal. (2) Infrared weak small targets have low signal-to-noise ratio and can easily be overwhelmed by background clutter. The algorithm in this paper may experience missed detections when the target signal-to-noise ratio is below 1. This is unacceptable for applications such as security monitoring and autonomous driving, which demand high safety requirements. These limitations will be the focus of our future work.

4. Conclusions

This paper presents a novel method called UCDnet for infrared weak small target detection and centroid localization. The innovation of our proposed semantic segmentation subnet lies in the improved U-Net and DNAnet structures, with the integration of attention modules. The double U-shaped backbone feature extraction network in our approach enables more accurate segmentation of target edges, while the addition of attention modules improves the capturing of target positional information. The innovation of our centroid detection subnet lies in the ability to overcome the constraint of unit pixel size in the original image, achieving sub-pixel-level centroid localization and minimizing the difference between predicted and ground truth centroid positions. Extensive comparative experiments demonstrate the superiority of our proposed semantic segmentation and centroid localization methods in terms of detection precision and robustness compared to existing mainstream methods.

In the future, we plan to expand our research in three directions: (1) How to use the motion information of targets in sequential images to reduce false alarms. The algorithm proposed in this paper is based on single-frame image for target detection, which may encounter challenges in removing false alarms in certain low-quality images or complex scenes. We will explore the temporal characteristics of targets in sequential images, combining spatial and temporal features to improve target discriminability. (2) How to construct an integrated network for enhancement and detection to address the issue of missed detections when the target is extremely weak. In our future work, we will embed more efficient enhancement modules or super-resolution reconstruction modules into the detection network to enhance target features and capture target details, aiming to achieve or even surpass the human eye’s detection limit. (3) How to utilize multi-source image information fusion for handling weak small targets [35]. The infrared images used in this paper are obtained from thermal radiation imaging. Due to the technological limitations of sensors, issues such as loss of target texture information, high noise, and low signal-to-noise ratio may exist. In the future, we will develop a weak small target detection network based on multi-source data, incorporating data from multiple platforms and payloads to achieve stable and continuous detection and localization capabilities.