Efficient Water Segmentation with Transformer and Knowledge Distillation for USVs

Abstract

Water segmentation is a critical task for ensuring the safety of unmanned surface vehicles (USVs). Most existing image-based water segmentation methods may be inaccurate due to light reflection on the water. The fusion-based method combines the paired 2D camera images and 3D LiDAR point clouds as inputs, resulting in a high computational load and considerable time consumption, with limits in terms of practical applications. Thus, in this study, we propose a multimodal fusion water segmentation method that uses a transformer and knowledge distillation to leverage 3D LiDAR point clouds in order to assist in the generation of 2D images. A local and non-local cross-modality fusion module based on a transformer is first used to fuse 2D images and 3D point cloud information during the training phase. A multi-to-single-modality knowledge distillation module is then applied to distill the fused information into a pure 2D network for water segmentation. Extensive experiments were conducted with a dataset containing various scenes collected by USVs in the water. The results demonstrate that the proposed method achieves approximately 1.5% improvement both in accuracy and MaxF over classical image-based methods, and it is much faster than the fusion-based method, achieving speeds ranging from 15 fps to 110 fps.

1. Introduction

Unmanned surface vehicles (USVs) are intelligent unmanned maritime devices used for several dangerous maritime tasks [1], including search and rescue missions and coastal facility inspections. They have been widely studied, with recent advancements in robotics and artificial intelligence. As a key perception technology for ensuring the safe navigation of USVs, stable and accurate water segmentation is crucial.

Existing water segmentation studies [2,3,4,5,6,7,8,9,10] are mainly based on the use of a monocular camera due to its low cost. Although the images contain rich representational information, they are susceptible to light reflections and lack 3D spatial perception. Considering the complementarity of data, LiDAR sensors are often used with monocular cameras. Therefore, the water segmentation method [11] consists of fusing the dense RGB semantic information from images with the sparse 3D spatial information from LiDAR point clouds to improve perception ability. However, this fusion-based method requires both the a LiDAR point cloud and a camera image as the input in the training and inference phases. Therefore, USVs should be equipped with two paired sensors to increase the computational load during real-time perception.

To solve this problem, in this study, we propose a novel efficient water segmentation process for USVs in which the 3D LiDAR point cloud is combined with the 2D camera image using a transformer and knowledge distillation. 2DPASS [12] demonstrated the effectiveness of using 2D images in combination with 3D point clouds for semantic segmentation. However, in water environments, the pulsed radar signal is not reflected on most of the water surface. Therefore, a 3D LiDAR point cloud is used as an auxiliary of the 2D images for cross-modality fusion. The fused features are then used to guide the 2D modality for water segmentation.

A local and non-local cross-modality fusion module based on a transformer is first explored to obtain better fusion information. The common fusion architecture involves late fusion, whereby features from different modalities are concatenated (Figure 1a), only considering local features lacking non-local information. On the contrary, the transformer block is more non-local with self-attention, and it more effectively performs feature matching between the 2D images and 3D point clouds. The fusion module with a transformer is shown in Figure 1b. In addition, the experimental results presented in Section 4.3.1 show that this local and non-local cross-modality fusion module is effective. Based on the obtained robust fusion features, a multi-to-single-modality knowledge distillation (KD) module is implemented in the model training phase to transform multimodal fusion information into single-modal features. Consequently, the single-modal features extracted from the camera image can achieve similar performance to that of the multimodal fusion features. Finally, in practical perception, the proposed method only takes the camera image as input, and it is faster than the multimodal fusion method while achieving a similar performance.

The main contributions of this paper are summarized as follows:

• A novel local and non-local cross-modality fusion module based on a transformer is proposed to obtain better 2D image and 3D point cloud multimodal fusion information for water segmentation. The transformer block can obtain more non-local information by self-attention;
• A multi-to-single-modality knowledge distillation (KD) module for the training phase of the water segmentation model is proposed to transform the multimodal fusion information into single-modal features. Through this module, the 3D-related network can be discarded during the inference phase, and the single-modal water segmentation performance can be improved without real-time loss;
• Extensive experiments are conducted on the CMWS [11] datasets collected by USVs. The results demonstrate the effectiveness and efficiency of the proposed method, which achieves an approximately 1.5% improvement in accuracy and MaxF compared with other image-only segmentation methods. It is also much faster than the multimodal fusion methods.

The remainder of this paper is organized as follows. Section 2 introduces the related works on water segmentation. Section 3 details the proposed novel water segmentation process with a transformer and knowledge distillation for USVs. Section 4 shows the experimental results obtained on the CMWS [11] datasets. We discuss the limitations of the proposed method and its influencing factors, as well as some potential future works in Section 5. Finally, in Section 6, we present our main conclusions and future prospects.

2. Related Work

Water segmentation methods for USVs. Most of existing methods are image-based, using 2D images from a monocular camera as the network input [2,3,4,5,6,7,8,9,10]. Some of these methods perform the visual perception of USV water segmentation according to traditional aspects, such as image resolution [6], image color spaces [7], and a superpixel map [8,10]. The authors of other studies [2,3,4,5] attempted to apply well-known semantic segmentation networks, such as FCN [13], SegNet [14], and U-Net [15], to perform water segmentation, leading to satisfactory results. For instance, the authors of [2,3] proposed U-Net a modified version of U-Net as the backbone to extract features from the image, making it suitable for water segmentation via other refined operations. Akiyama, T. S. et al. [4] applied SegNet to automatically segment river water in images acquired by RGB sensors. The DeepLab family [16,17,18,19] proposed by Google is a series of networks applied to semantic segmentation. Among them, the DeepLabV3+ [19] is most representative, which the authors of [8] used as a comparison benchmark for proposed methods. However, these image-only methods heavily rely on image quality, which may be hindered by complex water environments. On the contrary, the proposed method leverages the 3D point cloud to assist the 2D image, which allows the model to learn more information in different dimensions and thus avoid the limitations of a single data type.

Fusion-based methods have also attracted significant attention due to the complementary of the camera and LiDAR vision sensors. Many researchers have attempted to fuse these two sensors in order to propose multisensor semantic segmentation methods and apply them in different domains. The development of fusion-based methods [20,21,22,23] is extremely rapid, especially in the field of unmanned ground vehicles (UGVs). However, in the specific water environment of USVs, few fusion-based water segmentation methods are available. Only the study presented in [11] proposed a camera–LiDAR fusion deep learning method for USV water segmentation. However, this fusion-based method only performs local feature aggregation by concatenation and lacks non-local information. In addition, it requires paired point cloud and image data as the input for the training and inference stages, which increases the computational load and real-time loss. Therefore, in this study, we explore a local and non-local cross-modality fusion module with a transformer block to improve the fusion of the features. This effective fused information is then distilled into the 2D network for water segmentation through a multi-to-single-modality knowledge distillation module.

Cross-modality knowledge distillation. The theory of knowledge distillation (KD) was first proposed by Hinton et al. [24]. KD allows a small student model to effectively learn from a large teacher model. In recent years, KD has been widely used in cross-modality applications [25,26,27,28]. KD networks have also been continuously enhanced [29,30]. They transfer knowledge from the viewpoints of prediction results and feature layers. In addition, the knowledge transfer between different modalities can prevent the loss of modal information. Inspired by the studies presented in [29,30], in this paper, we propose a multi-to-single-modality knowledge distillation module that focuses on transferring multimodel fusion information to a single-model 2D network for water segmentation perception. During the inference phase, the 3D-related network can be discarded, which effectively reduces the additional computational load.

3. Methods

This section details the proposed transformer and knowledge distillation method for USV water segmentation, which focuses on the combination of a 3D LiDAR point cloud with a 2D camera image to improve the segmentation performance of the 2D network. Figure 2 shows that the proposed method comprises three main components: (1) a model architecture with a two-branch network, (2) a novel local and non-local cross-modality fusion module, and (3) a multi-to-single-modality KD module. It first passes images and point clouds independently through a two-branch network to generate multiscale features in parallel. This two-branch network contains the U-Net 2D segmentation network [15] and the PointNet++ 3D segmentation network [31]. The local and non-local cross-modality fusion module is then applied to fuse the 2D and 3D multiscale features. Afterwards, through the multi-to-single-modality KD module, the fused features can guide the 2D network to obtain better 2D features for water segmentation perception. In the inference process, the proposed method discards the 3D branch to reduce the computational load. Moreover, the 2D single-mode method is much faster than the multimodal fusion methods, with similar performance. Therefore, in practical applications, USVs do not require expensive LiDAR devices but only a low-cost camera as the input of the proposed water segmentation network.

3.1. Local and Non-Local Cross-Modality Fusion Module

The images and point clouds are generally represented by separated pixels and points, making it difficult to directly transfer these two modalities of information. Therefore, the point–pixel correspondence is first established through the projection operation. Point-wise 2D features (${\widehat{F}}^{2D}$) are then generated using both point–pixel mapping ($O^{p2p}$) and the original 2D feature map ($F^{2D}$). Afterwards, the point-wise 3D features (${\widehat{F}}^{3D}$) from the 3D network and the point-wise 2D features (${\widehat{F}}^{2D}$) are fed into the transformer block to obtain fused features. The details are shown in Figure 3.

3.1.1. Point–Pixel Correspondence

This module aims to build the correspondence between the 3D points and 2D pixels to obtain paired features of two modalities, which are further used for feature fusion. The process of point–pixel mapping is illustrated on left side of Figure 3. Given the original image ($I\in {\mathbb{R}}^{H\times W\times 3}$) as the input of the 2D network to extract features, the 2D feature map ($F^{2D}\in {\mathbb{R}}^{H\times W\times C_I}$) containing color and texture information is obtained from the 2D encoder. Given the 3D LiDAR point cloud ($P={\left\{p_i\right\}}_{i=1}^N$), where N is the number of points, similar to the process presented in [11,12], each 3D point ($p_i=(x_i,y_i,z_i)\in {\mathbb{R}}^3$) is projected to a 2D image pixel (${\tilde{p}}_i(u_i,v_i)\in {\mathbb{R}}^2$) based on the camera’s intrinsic matrices ($K\in {\mathbb{R}}^{3\times 4}$) and extrinsic matrices ($T\in {\mathbb{R}}^{4\times 4}$). Point–pixel mapping is then obtained through:

$$O^{p2p}={\left\{\left(⌊u_i⌋,⌊v_i⌋\right)\right\}}_{i=1}^N\in {\mathbb{R}}^{N\times 2},$$

(1)

where $\lfloor ·\rfloor$ is the floor operation.

If the pixel in the feature map is included in $O^{p2p}$, a point-wise 2D feature (${\widehat{F}}^{2D}\in {\mathbb{R}}^{N\times C_I}$) is extracted from the 2D feature map ($F^{2D}$).

3.1.2. Transformer Fusion Block

The transformer fusion block mainly contains a multimodal attention mechanism (MA) and a feed-forward network (FFN), as shown in Figure 4. MA allows the multimodal transformer to reinforce 2D modality with the 3D features from the 3D modality by learning attention across the features of the two modalities, which results in a robust and effective fusion strategy. Afterwards, the fusion results are fed into the FFN, which is composed of two fully connected layers. The FFN makes the fusion features more expressive and allows for better representation of the relationship between 2D pixels and other 3D points.

Inspired by the transformer block presented in [32,33], in this study, we propose a novel approach to fuse the information of two modalities (i.e., 2D and 3D). The point-wise 2D features (${\widehat{F}}^{2D}\in {\mathbb{R}}^{N\times C_I}$) and point-wise 3D features (${\widehat{F}}^{3D}\in {\mathbb{R}}^{N\times C_P}$, where $C_{(·)}$) represents the feature dimension of pixels or points, are obtained. The query, keys, and values are then defined as $Q_{2D}={\widehat{F}}^{2D}{W}_Q$, $K_{3D}={\widehat{F}}^{3D}{W}_K$, and $V_{3D}={\widehat{F}}^{3D}{W}_V$, respectively, where $W_{(·)}$ denotes the weights: $W_Q\in {\mathbb{R}}^{C_I\times C_K}$, $W_K\in {\mathbb{R}}^{C_P\times C_K}$, and $W_V\in {\mathbb{R}}^{C_P\times C_V}$. Feature adaptation from 3D points to 2D pixels is performed using multimodal attention ($M{A}_{2D}$:= $M_{3D\to 2D}\left({\widehat{F}}^{2D},{\widehat{F}}^{3D}\right)$):

$$M{A}_{2D}=softmax\left(\frac{Q_{2D}·K_{3D}^{\top }}{\sqrt{D_k}}\right)V_{3D},$$

(2)

$${\widehat{F}}^{fuse}=Linear\left(M{A}_{2D}\right)\in {\mathbb{R}}^{N\times C_f}.$$

(3)

where $\left(·\right)$ denotes the dot product, softmax denotes the softmax operation, and $\sqrt{D_k}$ is used for scaling. As shown in Equation (2), $M{A}_{2D}$ is passed through a linear layer to output the fusion features. Finally, an FFN is injected to complete the multimodal attention, as shown in Figure 4.

3.2. Multi-to-Single-Modality KD Module

In this study, the multi-to-single-modality KD module aims to transfer useful information from a trained 2D and 3D multimodal fusion teacher network to a single 2D modality student network. Therefore, the distillation loss is designed at the feature level and prediction level, as illustrated in Figure 5. More precisely, two independent classifiers are first applied to point-wise 2D features (${\widehat{F}}^{2D}$) and fusion features (${\widehat{F}}^{fuse}$) in order to obtain segmentation prediction scores ($S_1^{2D}$ and $S^{fuse}$). The loss of the feature level between the point-wise 2D features (${\widehat{F}}^{2D}$) and fusion features (${\widehat{F}}^{fuse}$) before the softmax operation is computed using the $L_2$ norm:

$$L_{feat}={\parallel {\widehat{F}}^{fuse}-{\widehat{F}}^{2D}\parallel }^2.$$

(4)

Furthermore, the channel dependency of the feature representations (${\widehat{F}}^{2D}$ and ${\widehat{F}}^{fuse}$) is considered in the final classification [34]. ${\widehat{F}}^{2D}$ is fed into the classifier trained by ${\widehat{F}}^{fuse}$ to obtain the output ($S_2^{2D}$). Similar to the study presented in [30], the $L_{SR}$ loss between the two outputs ($S_2^{2D}$ and $S^{fuse}$) is computed as:

$$L_{SR}=-S^{fuse}log{S}_2^{2D}.$$

(5)

Afterwards, the KL divergence is used as the distillation loss ($L_{KL}$) between the 2D-predicted ($S_1^{2D}$) and fusion-predicted ($S^{fuse}$) losses. In addition, to avoid excessively difficult fusion prediction labels, which could result in poor 2D prediction learning, the $L_{KL}$ loss is calculated with a distillation temperate of T = 7:

$$L_{KL}=D_{KL}^T\left(S_1^{2D}\parallel S^{fuse}\right).$$

(6)

Thus, the total loss is expressed as:

$$L_{total}=\alpha L_{feat}+\beta L_{SR}+\gamma L_{KL}+\lambda L_{ce},$$

(7)

where $\alpha =0.3$, $\beta =0.2$, $\gamma =0.3$, $\lambda =0.2$, and $L_{ce}$ is the cross-entropy loss used to measure the gap between the 2D prediction and the 2D ground truth.

Figure 5. Illustration of the multi-to-single-modality KD module.

Figure 5. Illustration of the multi-to-single-modality KD module.

The proposed KD module has several advantages. It reduces the gap in the prediction level and enhances the learning of 2D features by fusing features at the feature level. In addition, the fusion features provide intensive spatial information to facilitate the learning of the 2D features without losing 2D-specific information, such as color and texture. Moreover, in practical applications, the proposed model directly applies water segmentation using a 2D network without the 3D network branch, without requiring any additional computational load.

4. Experiments

Section 4.1 briefly introduces the implementation details, including the parameter settings and evaluation metrics. In Section 4.2, the proposed method is evaluated by comparing it with three well-known 2D segmentation networks, and some qualitative results are visualized. In addition, the speed of the proposed method and that of the method presented in [11] are compared. Finally, the comprehensive analysis of the proposed module is presented in Section 4.3.

4.1. Implementation Details

Parameter settings. All the experiments were implemented using the PyTorch deep learning framework and a Ubuntu 18.04 operating system. The Stochastic Gradient Descent (SGD) optimizer was used to update the parameters of backpropagation in the network. During the training of the samples, the learning rate of the model was initialized to 0.1 with a decay of 0.9 for every 10 epochs. The number of samples for one training iteration (i.e., batch size) was 8, and 64 training epochs were applied. The same data partitions and the same samples were applied in all the experiments. The experiments were conducted on a computer equipped with NVIDIA GEFORCE RTX 3090.

Evaluation metrics. Similar to the study presented in [11], we used the accuracy (AC), precision (P), recall (R), max F-measure (MaxF) of P and R, false-positive ratio (FPR), and mean intersection over union (mIoU) as evaluation metrics to evaluate the performance of the proposed model. AC is calculated by dividing the number of correct predictions by the total number of samples. P denotes how many genuine positive samples exist among the total positive samples. R, which is also known as the sensitivity, is calculated by dividing the number of correct predictions by the number of all positive samples in the dataset. FPR denotes the number of negative samples that are incorrectly predicted as positive samples. MaxF is the harmonic index of precision and recall. For the segmentation tasks, the mIoU value visually represents how closely the predicted results match the ground truth. The specific parameters are shown in

Table 1. Results of the comparison between the proposed method and some representative image-based segmentation methods.

Method	AC $\left(\%\right)$	MaxF $\left(\%\right)$	P $\left(\%\right)$	R $\left(\%\right)$	FPR $\left(\%\right)$ ↓	mIOU $\left(\%\right)$
U-Net [15] ${}^1$	94.20	94.47	93.31	95.65	7.64	89.0
Ours-U-Net ${}^2$	95.91	95.71	93.85	97.65	7.13	92.1
FCN [13] ${}^1$	95.38	96.38	97.13	95.64	4.36	95.3
Ours-FCN ${}^2$	97.76	97.38	97.14	97.62	3.2	95.61
DeepLabV3+ [19] ${}^1$	96.33	96.69	96.54	96.83	3.17	95.12
Ours-DeeplabV3+ ${}^2$	97.84	97.67	97.2	98.14	3.14	95.76

¹ Only the basic 2D segmentation networks; ² our proposed the plug-and-play module is added to the basic 2D segmentation networks to further improve the segmentation performance.

. Note that the higher the values of AC, P, R, MaxF, and mIoU, the higher the performance of the model.

4.2. Comparative Experiment

The proposed model is a plug-and-play framework that can be applied to arbitrary segmentation networks. To demonstrate the generalization ability of this framework, it is plugged into three representative image-based segmentation networks: U-Net [15], FCN [13], and DeeplabV3+ [19]. Table 1 shows that the proposed method outperforms different basic segmentation networks. It adequately fuses the 2D image and 3D point cloud information and distills them into a 2D segmentation network. The experimental results show that fusing 3D point cloud information plays an important role in improving the image-based water segmentation methods. Moreover, the FCN [13] and DeeplabV3+ [19] networks outperform the lightweight U-Net [15] network. It can be clearly seen that the AC, MaxF, P, R, and mIoU metrics are all improved, and the FPR value is reduced, which indicates a lower false positive rate. Note that the specific evaluation metrics presented in Table 1,

Table 2. Comparison between different fusion strategies for 2D and 3D features. (Trans. denotes the transformer).

Fusion Strategy	AC $\left(\%\right)$	MaxF $\left(\%\right)$	P $\left(\%\right)$	mIoU $\left(\%\right)$
Concatenation	95.11	94.79	92.93	90.63
CBAM [35]	95.68	94.85	93.75	91.67
Ours-Trans.	95.91	95.71	93.85	92.1

,

Table 3. Comparison with different KD methods.

Method	AC $\left(\%\right)$	MaxF $\left(\%\right)$	P $\left(\%\right)$	mIoU $\left(\%\right)$
U-Net (no KD)	94.20	94.47	93.31	89.00
KL Divergence	94.80	94.96	93.96	90.09
Hinton et al. [24]	95.17	94.85	94.20	90.73
Yang et al. [30]	95.38	94.90	93.58	91.14
Ours KD	95.36	95.50	95.25	91.08

and

Table 4. Results of different ablation experiences.

Model	Local and Non-Local Cross-Modality Fusion Module	Multi-to-Single-Modality KD Module	AC $\left(\%\right)$	mIoU $\left(\%\right)$
A	–	–	94.20	89.00
B	✓	–	95.04	90.52
C	–	✓	95.36	91.08
D	✓	✓	95.91	92.10

are detailed in Section 4.1.

The qualitative evaluation results of the proposed method and the U-Net [15] basic segmentation method are shown in Figure 6. Most of the water surface can be segmented using the basic 2D segmentation network. However, the basic method does not perform well for near-shore segmentation due to water reflection and bad weather. On the contrary, the proposed method accurately segments the water surface in different scenes, which is closer to the ground truth, especially in the near-shore areas.

Furthermore, to validate the efficiency of the proposed method, an experimental speed comparison between the fusion method presented in [11] and the proposed method was conducted. Considering that the U-Net [15] is relatively simple, while DeeplabV3+ [19] is more complex, FCN [13] was chosen as the baseline method, and the same hardware environment and dataset were used. Figure 7 shows that the proposed method achieved significant speed improvement from 15 fps to 110 fps while maintaining a comparable accuracy to that of the fusion method presented in [11]. This proves that the proposed method achieves similar performance to that of the multimodal fusion method while being faster.

4.3. Comprehensive Analysis

Some experiments and analyses were conducted to demonstrate the effectiveness of the local and non-local cross-modality fusion module and the multi-to-single-modality KD module in the proposed water segmentation method. All the experiments were based on the same baselines: U-Net [15] for 2D images and PointNet++ [31] for 3D point clouds.

4.3.1. Effectiveness of the Cross-Modality Fusion Module

The component of the cross-modality fusion module (Section 3.1) covers the point–pixel correspondence and the transformer block. The fusion effects of 2D and 3D features with different fusion strategies were compared, as shown in Table 2. Two well-known fusion strategies were used for this comparison experiment: concatenation fusion and the CBAM mechanism [35]. Concatenation fusion was performed, whereby the features from different sources were concatenated. The CBAM mechanism [35] is an integrator of channel attention and spatial attention used to enhance the expressiveness of the network features. Table 2 clearly shows that the proposed fusion module with a transformer block outperforms the concatenation and CBAM methods, which can be verified by all the metrics. This shows that the transformer block plays an effective role in the fusion of the 2D and 3D features.

4.3.2. Comparison with Other KD Methods

To further validate the usefulness of the KD module, it was compared with some classical teacher–student KD architectures and KL divergence (Table 3). KL divergence is usually used to measure the similarity between two distributions. However, in the considered scenario, using only KL divergence as distillation loss is not accurate. In addition, based on the studies of Hinton et al. [24] and Yang et al. [30], the proposed KD module (Section 3.2) not only performs distillation loss on the prediction layer (Equation (6)) but also adds a distillation loss to the feature layer (Equation (4)). Table 3 shows that the proposed KD achieves the highest MaxF of 95.50% and a precision of 95.25%.

4.3.3. Ablation Study

Ablation experiments were performed using different components (Table 4) to study the contribution of each component of the proposed method. Model A is the baseline with only U-Net and PointNet++. Model B adds a cross-modality fusion module, and the AC and mIoU are improved by 0.84% and 1.52%, respectively. In addition, Model C simply uses the KD module to increase the AC by 1.16% and the mIoU by 2.08% relative to Model A. Finally, Model D is equipped with a local and non-local cross-modality fusion module and multi-to-single-modality KD module, achieving 95.91% AC and 92.10% mIoU, demonstrating the effectiveness of the proposed cross-modality fusion module with a transformer and multi-to-single-modality KD module water segmentation method.

5. Discussion

In this article, we proposed a water segmentation method for USVs that uses a local and non-local cross-modality fusion module based on a transformer block and a multi-to-single-modality knowledge distillation module to leverage 3D LiDAR point clouds in combination with 2D images. It achieves improved single-modal water segmentation performance without real-time loss. The effectiveness of the proposed method is demonstrated through the detailed proof of the experiment reported in Section 4. According to the experimental results, the proposed method achieves approximately 1.5% improvement both in accuracy and MaxF over three classical image-based methods, and it is much faster than the fusion-based method [11], achieving an increase in speed from 15 fps to 110 fps, as shown in Figure 6.

Considering the limitations of traditional image processing algorithms in terms of robustness and scalability, we usually choose a deep network with a more powerful learning ability for water segmentation. However, although the proposed method achieved good results in the experiments, we also identified some limitations and influencing factors. For example, the dataset in the field of USVs is not as large as that in the UGV field, which may result in overfitting problems with the limited dataset. Moreover, the CMWS [11] dataset used in our experiments was collected by USVs in inland waterways, which may potentially affect the generalization ability of our model to the sea surface environment. Therefore, we will explore the potential of semisupervised methods to address dataset scarcity to further optimize the USV water segmentation task in the future.

There are some other potential future works proposed. First, we will consider using other data sources, such as radar or sonar, to further improve the accuracy and efficiency of water segmentation methods for USVs. Secondly, the potential of transfer learning techniques will be explored to improve the performance of the proposed multimodal fusion water segmentation method when applied to different water scenes or environments. Thirdly, we will investigate the possibility of using reinforcement learning techniques to optimize the performance of the proposed water segmentation method in dynamic and changing water environments. Finally, we hope these techniques can be integrated into a real-time water segmentation system for practical USV applications in different scenarios, such as search and rescue missions or for environmental monitoring.

6. Conclusions

In this paper, we proposed a method for USV water segmentation that consists of applying a transformer and knowledge distillation to allow the 3D point cloud to assist the 2D image. The proposed method optimizes the performance of image-based methods while being faster than fusion-based methods. More precisely, a local and non-local cross-modality fusion module based on a transformer was used to effectively fuse the 3D point cloud and 2D image during the training phase. The fused information was then used to guide the 2D network in order to improve the water segmentation results through a multi-to-single-modality knowledge distillation module. Moreover, the 3D-related network can be discarded to reduce the computational load during the inference phase. The results of extensive experiments on various datasets of water scenes collected by USVs demonstrate that the proposed method outperforms image-based baselines and that it is faster than the fusion method. In addition, considering the limitations of data collection, in future work, we aim to further optimize the USV water segmentation task from the perspective of semisupervised methods.