TE-SAGAN: An Improved Generative Adversarial Network for Remote Sensing Super-Resolution Images

Abstract

Resolution is a comprehensive reflection and evaluation index for the visual quality of remote sensing images. Super-resolution processing has been widely applied for extracting information from remote sensing images. Recently, deep learning methods have found increasing application in the super-resolution processing of remote sensing images. However, issues such as blurry object edges and existing artifacts persist. To overcome these issues, this study proposes an improved generative adversarial network with self-attention and texture enhancement (TE-SAGAN) for remote sensing super-resolution images. We first designed an improved generator based on the residual dense block with a self-attention mechanism and weight normalization. The generator gains the feature extraction capability and enhances the training model stability to improve edge contour and texture. Subsequently, a joint loss, which is a combination of L1-norm, perceptual, and texture losses, is designed to optimize the training process and remove artifacts. The L1-norm loss is designed to ensure the consistency of low-frequency pixels; perceptual loss is used to entrench medium- and high-frequency details; and texture loss provides the local features for the super-resolution process. The results of experiments using a publicly available dataset (UC Merced Land Use dataset) and our dataset show that the proposed TE-SAGAN yields clear edges and textures in the super-resolution reconstruction of remote sensing images.

1. Introduction

The development of optical remote sensing technology has enabled researchers to obtain large-area, multi-temporal Earth observation images for social construction with ease. Nevertheless, optical satellite image quality suffers from the sensor imaging technology, limiting the quality and application of remote sensing images. Low-resolution remote sensing images bring trouble to ground object interpretations. Therefore, super-resolution reconstruction of remote sensing images has become an indispensable preprocessing step. Improving the spatial resolution of remote sensing images promotes clearer object boundaries and contours in remote sensing images, which provides clear data for subsequent applications of extracting features, such as the recognition roads and buildings by semantic segmentation methods [1,2,3] or change detection by extracting the object boundary [4,5]. Therefore, the super-resolution processing of remote sensing images has become an indispensable preprocessing process. Many researchers have utilized super-resolution techniques to effectively and accurately obtain images [6,7,8,9,10]. Super-resolution (SR) imaging is a technology that uses single or multiple low-resolution (LR) images to obtain high-resolution (HR) images using existing image acquisition devices. It has been widely applied since the 1980s, and Huang et al. [11] creatively proposed a method for reconstructing HR images based on the frequency field of multi-frame sequential LR images. Since then, SR processing of remote sensing images has become an active research topic, and numerous pertinent algorithms have been designed. At present, researchers have divided single-image super-resolution (SISR) processing methods into interpolation-, reconstruction-, and learning-based methods.

The interpolation-based method estimates the unknown pixel value via interpolation in the designed linear function according to the known pixels, such as nearest-neighbor interpolation [12], Lanczos filtering [13], or bicubic filtering [14]. Although these methods can achieve rapid SR image processing, filling the pixels with low-frequency information leads to shortcomings in the overall semantic features. Consequently, the super-resolution images may lose information of some details or boundaries may not be clear. Reconstruction-based methods aim to establish an appropriate prior constraint or registration model between LR and HR images, such as projection onto convex sets [15,16,17], iterative back projection [18], or maximum a posteriori probability method [19,20,21]. The core objective of reconstruction-based methods is to build a reasonable prior observation model. However, SR imaging is a highly underdetermined infinite-solution problem in mathematical theory. Even using the optimal parameters, the transformation performance of the model is weak in diverse tasks. Owing to this problem, the model requires considerable resources to accomplish repeatable and complicated calculations.

In recent years, with the development of machine learning frameworks, academic interest in learning-based methods has increased, particularly deep learning driven by big data [22]. Convolutional neural networks are a representative adaptive learning method widely used for the reconstruction of low- and high-frequency images and are less complex than traditional manual interventions. Researchers have thus expanded and applied convolutional neural networks to SR image tasks [23,24,25,26,27,28]. However, both early shallow networks with 3–5 layers [29,30] and deep super-resolution networks [31,32,33] tend to produce low-frequency results.

To address these issues, researchers have explored algorithms that yield results consistent with human observations and sought alternatives capable of obtaining high-frequency images. Ledig et al. [34] proposed an improved generative adversarial network (GAN) model, named SRGAN, which benefits from content and perceptual losses and exerts good performance. Based on SRGAN, the Residual-in-Residual Dense Block (RRDB) was introduced, which further improved the recovered image texture [35]. In the last years, many ESRGAN-based algorithms [36,37,38] have been improved. ESRGAN+ [36] replaces ESRGAN’s RRDB with RRDRB; Real-ESRGAN replaces the VGG-Net type discriminator in the original ESRGAN with a U-Net type discriminator [37], which has a good improvement in anime images and videos. However, these network parameters are too large to fit training conditions. When these methods are used to reconstruct remote sensing images, the texture details are still unclear, and some artifacts remain in the produced images.

The primary reasons for the abovementioned problems can be summarized as follows: (1) Owing to their peculiar characteristics compared to conventional images of the same image dimensions, remote sensing images contain abundant information, such as external contours and internal textures, that characterize the spatial relationships of ground objects, thereby making the content appear crowded. Therefore, algorithms that can accurately recover and characterize contours and internal textures are necessary. (2) Most existing methods do not consider global spatial connectivity and refining texture in the algorithm design. Owing to the local perception characteristic of convolutional neural networks, global feature information in large-scale remote scenes is ignored. Thus, the global remote sensing information is neglected and underutilized.

To resolve these concerns, we present TE-SAGAN, a GAN-based texture-enhanced SR processing network for remote sensing images, which sharpens SR image details while maintaining clear visual edges. The self-attention mechanism (SAM) is designed to learn global information from remote sensing images. Furthermore, the RRDB was introduced for image depth extraction. Simultaneously, a SAM with three submodules was used to obtain different levels of attention, where relevant information (such as edge contours) can be engaged and received dynamically. Weight normalization (WN) [39] was also introduced to TE-SAGAN to remove artifacts and increase the training network stability. The calculated texture loss by the Gram matrix was designed to assess the difference between the original HR and SR images in high-frequency features, which can help improve the internal details of the reconstructed image.

In conclusion, by the self-attention module and texture loss units, this work provides insight into refining the image edges and textures. The major contributions are as follows: (1) designing the self-attention mechanism module for extracting the global information, which can decrease the limitation of the local receptive field and increase the super-resolution network accuracy; (2) a weight normalization layer is introduced instead of batch normalization, which can reduce the interference of mini-batch data on reconstructed image quality and improve the image artifact; (3) a joint loss is designed combining the content loss, perceptual loss, adversarial loss, and texture loss, which take into account the consistency of the pixel content and texture details and promote the super resolution work to achieve a characteristic remote sensing image.

2. Method

2.1. Principle of the Proposed Method

The designed network framework proposed in this study is based on GAN [40] and comprises two parts: a generator network and a discriminator network. The generator network consists of feed-forward neural networks to form the feature extraction module, which is expressed by the function $G(I_{LR};{\theta }^G)$, where ${\theta }^G$ denotes the weights and biases $(w_L,b_L)$ of the neural network layer. The fully trained network was generated as follows:

$$I_{SR}=G_{{\theta }^G}(I_{LR}),$$

(1)

where $I_{SR}$ is the super-resolution image, and $I_{LR}$ is the corresponding low-resolution image. $I_{SR}$ is as close as possible to the sample distribution of $I_{HR}$, and the parameter optimization of the training network must satisfy the following rule:

$${\theta }^G=\arg \min \frac{1}{N}{\displaystyle \sum _{n=1}^N{L}_{(I_{SR},I_{HR})}},$$

(2)

where $L_{(I_{SR},I_{HR})}$ denotes the reconstruction error of a pair of data group $I^n$$(I_{SR},I_{HR})$.

The parameters of the discriminator D were defined accordingly as ${\theta }^D$. Their adversarial training is essentially a minimum–maximum optimization. We converted the training parameters corresponding to the LR-HR remote sensing image processing as follows:

$$\begin{array}{c}\underset{G}{\min } \underset{D}{\max }V\left(D_{{\theta }^D},G_{{\theta }^G}\right)=E_{I_{HR}~P_{data}\left(I_{HR}\right)}\left[\log D_{{\theta }^G}\left(I_{HR}\right)\right]+\\ \hfill E_{I_{LR}~pG\left(I_{LR}\right)}\left[\log \left(1-D\left(G_{{\theta }^G}\left(I_{LR}\right)\right)\right)\right]\end{array},$$

(3)

The basic idea expressed in the above equation is that the generator network G should be well-trained under the GAN (Figure 1). The discriminator network D has to distinguish the true and false states of the input remote sensing images. For the inputted $I_{LR}$, the generator in the latter term emulates the $G_{{\theta }^G}(I_{LR})$ sample from the true sample $P_{data}(x)$, thereby minimizing the probability that the discriminator distinguishes $G_{{\theta }^G}(I_{LR})$ as a fake sample. The network parameters are updated iteratively under strict monitoring conditions (e.g., loss constraints of mean square error [MSE] and mean absolute error [MAE]). The ability of the generator to fit the simulation is continuously enhanced, making it difficult for the discriminator to distinguish the reliability of the input samples. The well-trained model can predict the potential frequency distribution of the samples and generate new high-quality data sources.

2.2. Generator of TE-SAGAN

Owing to the limitation of “local perception” in conventional convolution modules, long-distance global information cannot be used for HR remote sensing images. To extract more edge and global information from remote sensing images, the generator of TE-SAGAN was designed with a low-frequency feature extractor, deep residual dense block, self-attention module, and WN (Figure 2). The self-attention module was designed to further enhance the ability of the network to extract global object features. Compared with the common batch normalization (BN) usually used in neural networks, the WN introduced in this study can stabilize network training and remove image artifacts.

In the generator network, the low-frequency feature extractor is a convolution layer with 3 × 3 kernels, following a WN layer. RRDB is an excellent nonlinear feature extraction module that has an interwoven ensemble with dense connectivity and residual skipping. RRDB can learn deep semantic features while improving low-level feature extraction. Here, we gathered the feature-flow output from RRDB with the previous residual layer, merging them into the SAM module to generate global contextual outlines. To achieve the SR processing for remote sensing images with upscaling factors (4×), these extracted features were recovered using an up-sample layer with nearest-neighbor interpolation. In this process, in addition to keeping 23 RRDBs with the residual scaling factor $\beta =0.2$ $(0<\beta <1)$ and a self-attention mechanism module, each regular convolution layer was followed by WN (Figure 3).

2.2.1. Self-Attention Mechanism

Traditional convolution operations associate regional pixels with neighboring pixels using a constant convolution kernel size. However, the limited convolution layer perception domain leads to the failure of global feature extraction, and consequently, the reconstructed image appears as a blurred overlay, or the recovered features do not correspond to the original image. Although expanding the convolution kernel size and deepening the convolution layers can help obtain the relevant global features, the computational complexity of the network increases exponentially. In recent years, attention mechanisms [41] have been widely used in natural language processing. In particular, the SAM module focuses selectively on significant location features to reduce network capacity and improve network computation performance. To obtain more global information, we designed a SAM module in the generator network (Figure 4).

The SAM module converts the previous feature nodes $x\in {\mathrm{R}}^{C\times H\times W}$ into three mapping sub-layers: a pixel feature extraction space $f(x)=W_{f}x$, a global feature extraction space $g(x)=W_{g}x$, and a convolution feature mapping layer $h(x)=W_{h}x$. To improve memory efficiency, the channel number $C$ was set to $C/k$ $\left(k=8\right)$. We first transposed the multiplication between $f(x)$ and $g(x)$. Subsequently, the transformed softmax classification function was used to obtain the feature attention map as follows:

$${\beta }_{j,i}=\frac{\exp (s_{ij})}{{\displaystyle {\sum }_{i=1}^N\exp (s_{ij})}},$$

(4)

where ${\beta }_{j,i}$ indicates the significance of the synthetic j-th region model for the i-th position, $s_{ij}=f{(x_i)}^{T}g(x_j)$, producing a series of attention map layers $o=(o_1,o_2,\dots ,o_j,\dots ,o_N)\in {\mathrm{R}}^{C\times H\times W}$, where one $o_j$ layer is shown as:

$$o_j={\displaystyle \sum _{i=1}^N{\beta }_{j,i}}h(x_i)$$

(5)

Finally, the attention map and feature mapping layer $h(x)$ were added together to merge the globally relevant features. The output of the layer is expressed as follows:

$$y_i=\gamma o_i+x_i,$$

(6)

To explore local spatial information, $\gamma$ was initialized as 0. Overall, self-attention can capture accurate image features beyond the convolution operation, and spatial connections between reference pixels across distances can help obtain more reference information. Furthermore, it also considers the key features of a high-resolution image with low computational complexity.

2.2.2. Weight Normalization

Batch Normalization (BN) [42] is a technique widely used in GAN-based methods. It balances the bias for image feature covariates inside the deep network and prevents overfitting during training. Nevertheless, BN tends to destroy low-frequency spatial features and increase erroneous noise estimates, causing artifacts in the SR processing results of remote sensing images. To overcome this issue, Salimans et al. have actively advocated for Weight Normalization (WN) [39] to stabilize the generative network, especially in GANs, by decoupling training weight vectors from their directions. WN decreases the dependence between mini-batch data and training performance, retains the contrast of image feature information, and has lower computational complexity than BN in equal time. Therefore, WN is more suitable than BN for performing SR tasks.

For nonlinear processing in the neural network, as seen in Equation (7), the former feature node $x$ was computed with convolutional filtering with parameters $w$ and $b$. New feature maps and nodes $y$ were obtained by the activation function. WN in the convolution layer can be calculated using Equation (8):

$$y=f\left(w\cdot x+b\right),$$

(7)

$$w=g\frac{v}{‖v‖},$$

(8)

where $v$ and $w$ have the same dimensional feature vector, and $\Vert v\Vert$ denotes the Euclidean norm of $v$, which represents the direction of $w$. Scalar g determines the length w. After WN, the neural network nodes can be calculated as follows:

$$y=f\left(\frac{g}{‖v‖}\cdot x+b\right)$$

(9)

2.3. Discriminator of TE-SAGAN

The role of the discriminator is to recognize the input images and judge their authenticity. The discriminator for TE-SAGAN is a VGGNet-type network structure [43] (Figure 5). This network has proven that multiple small convolution kernels (3 × 3) can learn more complex features at a lower computational cost than large convolutional kernels (e.g., 5 × 5, 7 × 7, and 11 × 11 in AlexNet) [44]. In this study, the fixed convolutional kernel size of VGGNet was modified to alternative kernel sizes of 3 and 4 to enhance the image feature recognition. Furthermore, the WN layers that replace all BN layers in the discriminator network can further improve the training network speed and stability.

The restored 128 × 128 SR and HR images were fed into the discriminator network. The image feature transferred through 10 layers of alternate convolution with kernel sizes of 3 × 3 and 4 × 4, step sizes of 1 and 2, and two linear dimensionality reduction operations. The channel numbers of the alternate convolution layers were 64, 128, 256, and 512. The generated feature map result was then sent into the final sigmoid function, outputting the authenticity of the remote sensing images.

2.4. Loss Function

The loss function measures the difference between the SR images generated by the network model and the HR images used for training the networks. In this study, to obtain a better result with more texture information and consistent style by TE-SAGAN, the loss function $L_{(I_{SR,HR})}$ during training network combined L1 norm content loss, training adversarial network loss, perceptual loss using a high-frequency network layer from VGGNet19, and texture loss computed using the Gram matrix.

2.4.1. Content Loss

Minimization of the MAE can enhance the sensitivity of abnormal image pixels. Therefore, it was used to describe the consistency of the low-frequency data between the recovered SR image and the original HR image. The definition of content loss is:

$$L_{\mathrm{con}}=L_{MAE}(I_{LR},\theta )=\frac{1}{M}{\displaystyle {\sum }_{i=1}^M\Vert G(I_{LR}^i;\theta )-I_{HR}^i\Vert },$$

(10)

where $I_{HR}^i$ and $I_{LR}^i$ jointly form the i-th pair of the training sample set, and M represents the number of training dataset pairs. $G(I_{LR}^i;\theta )$ is a mapping function between LR and SR images.

2.4.2. Adversarial Loss

The training stability of GANs is a measure used to ensure the quality of a generated image. The generator network “pretends to learn well” to cheat the discriminator and then yields terrible results. In this experiment, we employed the relativistic average standard GAN (RasGAN) [45], an adversarial training strategy with a relativistic discriminator, to compare the average input real data relative to faked data with a more realistic probability. The adversarial TE-SAGAN loss is unlike the rule of “either true or false” in the standard GAN discriminator. Instead, it focuses on the prediction of ground truth images to improve the network robustness. The generator and discriminator loss functions have mutual symmetry, and the corresponding loss functions of TE-SAGAN are defined in Equations (11) and (12):

$$L_D^{Ra}=-{\mathrm{E}}_{I_{HR}}[\log (D_{Ra}(I_{HR},I_{SR})-{\mathrm{E}}_{SR}[\log (1-D_{Ra}(I_{SR},I_{HR}))],$$

(11)

$$L_D^{Ra}=-{\mathrm{E}}_{I_{HR}}[\log (1-D_{Ra}(I_{HR},I_{SR})-{\mathrm{E}}_{SR}[\log (D_{Ra}(I_{SR},I_{HR}))],$$

(12)

where $I_{HR}$ denotes the original ground truth sample, $I_{SR}=G(I_{LR})$, which is the generated SR remote sensing image, and ${\mathrm{E}}_{SR}[*]$ represents the mean value of the generated remote sensing image for computation in the mini-batch. The adversarial loss of the generator is expressed as follows:

$$L_{\mathrm{adv}}=L_G^{Ra}=-{\mathrm{E}}_{I_{HR}}[(1-\log (D_{Ra}(I_{HR},I_{SR}))]-{\mathrm{E}}_{I_{SR}}[(\log (D_{Ra}(I_{SR},I_{HR}))],$$

(13)

2.4.3. Perceptual Loss

Typically, the pixel-wise loss (MSE or MAE) is restricted to the outcomes of low-frequency smooth features. However, it has some disadvantages for sparse and weak image feature values, leading to a low supervision capability for high-frequency features. To improve the quality of the generated HR remote sensing images and enhance the robustness of the deep network, perceptual loss was calculated using high-frequency feature maps before the activation operation, which can better represent the consistency of image texture information. In the pretrained VGG19 network [43], we calculated the Euclidean metric of high-frequency features extracted before the ReLU activation layers between the reference ground truth and the reconstructed SR image, defining their differences as perceptual loss:

$$L_{\mathrm{percep}}=\frac{1}{W_{ij}{H}_{ij}}{{\displaystyle {\sum }_{x=1}^{W_{i,j}}{\displaystyle {\sum }_{y=1}^{H_{i,j}}\left({\phi }_{ij}{\left(G(I_{LR})\right)}_{x,y}-{\phi }_{ij}{\left(I_{HR}\right)}_{x,y}\right)}}}^2,$$

(14)

where $W_{ij}$$H_{ij}$ represents the dimension of the feature map in the VGGNet, $I_{LR}$ represents the LR remote sensing image, $I_{HR}$ represents the HR remote sensing image, ${\phi }_{ij}$ indicates the activation operation of the feature map extracted before the j-th convolution layer and the i-th maximum pooling layer within the VGGNet19.

2.4.4. Texture Loss

Texture is also known as low-contrast fine detail. For SR remote sensing images, restoring the texture information to approach that of HR images is the primary objective. Consideration of perceptual loss improved the quality of the reconstructed image. However, the fine textures were still obscured. In this study, based on perceptual loss, the correlation between various feature channels was used to calculate texture loss. Texture loss can consolidate and correct the details of remote sensing images. Texture loss in HR-SR images can be calculated as follows:

$$L_{\mathrm{texture}}={‖G_r\left(\varphi \left(I_{SR}\right)\right)-G_r\left(\varphi \left(I_{HR}\right)\right)‖}_2^2,$$

(15)

where $\varphi (I)\in R^{n\times m}$ represents the extracted features from VGGNet19, and $G_r$ is the feature map matrix computed on the gram matrix with the property of $G_r(F)=F{F}^{\mathrm{T}}\in R^{n\times n}$.

2.4.5. Total Loss Function

Combined with the losses described in the previous sections, the total loss function is as follows:

$$L_G=\eta L_{\mathrm{cont}}+\lambda L_{\mathrm{adv}}+L_{\mathrm{percep}}+\beta L_{\mathrm{texture}},$$

(16)

where $L_{cont}=L_1=\left|\right|I_{SR}-I_{HR}{\left|\right|}_1$ and $L_{adv}$ denote adversarial loss. $L_{percep}$ denotes the perceptual loss mentioned in Section 2.4.3. $L_{texture}$ denotes the texture loss that describes the feature details. Accordingly, $\eta$, $\lambda$, and $\beta$ are the coefficients that balance different sub-loss terms.

3. Experiment and Analysis

3.1. Evaluation Indexes

The peak signal-to-noise ratio (PSNR), structural similarity (SSIM) [46], and Fréchet Inception Distance (FID) [47] scores were used as objective evaluation indices for the experimental results, whereas the visual effect provided a subjective judgment. The PSNR was used to measure the pixel difference between the compressed image or signal reconstruction image and the original image in dB. A higher PSNR score indicates higher quality and pixel fidelity of the reconstructed image. The PSNR was calculated as follows:

$$MSE=\frac{1}{mn}{\displaystyle \sum _{i=0}^{m-1}{\displaystyle \sum _{j=0}^{n-1}\Vert I(i,j)-K(i,j)\Vert }}{}^2,$$

(17)

$$PSNR=20\ast {\log }_{10}\left(\frac{MA{X}_I}{\sqrt{MSE}}\right),$$

(18)

where $MA{X}_I$ represents the maximum pixel value ($MA{X}_I$ is 255 for RGB images), and MSE represents the mean square error for images I and K with a size of m × n.

SSIM is a measure of luminance, contrast, and structure between two images in noise interference distortion. SSIM was calculated as follows:

$$SSIM\left(x,y\right)=\frac{\left(2{\mu }_x{\mu }_y+C_1\right)\left(2{\sigma }_{xy}+C_2\right)}{\left({\mu }_x^2+{\mu }_y^2+C_1\right)\left({\sigma }_x^2+{\sigma }_y^2+C_2\right)},$$

(19)

where ${\mu }_x$ and ${\mu }_y$ represent the pixel mean of images x and y, respectively; ${\sigma }_{xy}$ represents the covariance of images x and y; naturally, ${\sigma }_x$ and ${\sigma }_y$, respectively, represent the corresponding variance in images x and y. $C_1=(K_{1}L{)}^2$ and $C_2=(K_{2}L{)}^2$ are constants that avoid dividing by zero. Furthermore, $L$ is the range of image pixel fluctuation. $K_1$ and $K_2$ are set orderly to 0.01 and 0.03 by default. SSIM consists of the following three components:

Luminance: $l(x,y)=\frac{2{\mu }_x{\mu }_y+C_1}{{\mu }_x^2+{\mu }_y^2+C_1}$;

Contrast: $c(x,y)=\frac{2{\sigma }_x{\sigma }_y+C_2}{{\sigma }_x^2+{\sigma }_y^2+C_2}$;

Structure: $s(x,y)=\frac{{\sigma }_{xy}+C_3}{{\sigma }_x{\sigma }_y+C_3}$, with $C_3=C_2/2$.

In addition, the FID score, a proven systematic and convincing metric, was employed in this study to evaluate the accuracy of the designed model. FID computes the Fréchet distance (also known as Wasserstein-2 distance) between two Gaussian distributions (synthetic and real images) in the feature space [48]. FID constructs Gaussian model parameters around the matrix of two variables: the mean and covariance values $(m,C)$, which refer to the mean and covariance in HR $(m_r,C_r)$ and generated images $(m_g,C_g)$. The FID score was determined as follows:

$$\mathrm{FID}=d^2((m_r,C_r),(m_g,C_g))=\Vert m_r-m_g\Vert {}_2^2+ \mathrm{Tr}(C_r+C_g-2\sqrt{(C_r{C}_g)}),$$

(20)

where $\mathrm{Tr}$ indicates the sum of the image features on the main diagonal of the square matrix. Accordingly, a lower FID value indicates that the produced image is closer to the HR sample.

3.2. Experiment Preparation

3.2.1. Data Set

The study area is located in the port city of Portsmouth, eastern Virginia, USA, and comprises water systems, houses, roads, and other landscape features. We built a dataset with 60,000 HR satellite image patches of 128 × 128 pixels without overlapping areas. The final 10,000 images were available for testing the trained model (called the Test-10000 set). Because the designed model’s aim is to obtain the single images’ super-resolution, the 4× super-resolution image processing is applied in this research, which is widely recognized in the current deep learning remote sensing image super-resolution tasks [49,50,51]. We used bicubic down-sampling with a common factor (4×) to obtain corresponding LR image patches and to establish a training dataset pair with HR image patches. In addition, given that super-resolution tasks primarily involve scaling of spatial resolution, the training dataset was only enhanced by random horizontal left and right flips, as well as 90°, 180°, and 270° rotations. Additionally, a well-known and publicly available dataset, the UC Merced Land Use dataset [52], was used in the test experiments. This dataset consisted of 256 × 256 pixels original HR image patches with 21 categories of satellite image scenes, each containing 100 patches of images. We selected the harbor, runway, airplane, and building scenes to validate the performance of the proposed method. Corresponding LR image patches of 64 × 64 pixels were obtained through the same down-sampling method.

3.2.2. Experimental Settings

All SR processing algorithms were implemented based on the TensorFlow framework in an Ubuntu 20.04.0 LTS system equipped with an NVIDIA GeForce RTX 2080 Ti GPU, CUDA10.1, and CUDNN7.6.0. As in a previous study [35], the training process was divided into two stages: pretraining and training. First, the proposed networks were trained with the L1 loss to consolidate the low-frequency image content and enhance the generator simulation capability. After pretraining, formal adversarial training was initialized with the parameters obtained from the final pretrained model. The balancing loss term coefficients in Equation (16) were $\eta {=10}^{-2}$, $\lambda =5\times {10}^{-4}$, and $\beta {=10}^{-6}$. The batch size of the imported training data was set to 16 in the pretraining phase and 4 in the adversarial training phase. The initial learning rate was set at $2\times {10}^{-4}$, and half-decayed every $2\times {10}^5$ during the pretraining phase and every $5\times {10}^4$ during the training phase. Adam was used for the overall training optimization [53] in the training stage with default hyperparameters (${\beta }_1$ = 0.9, ${\beta }_2$ = 0.999).

3.3. Results and Analysis

3.3.1. Quantitative Evaluation

First, by comparing the results obtained using enhanced super-resolution GAN (ESRGAN) [35] and TE-SAGAN in the pretraining phase, we determined the generator network stabilization performance of the two networks. A visual example of PSNR values in the entire pretraining stage for the two methods is displayed in Figure 6. The results revealed that, compared with the baseline, the pretraining PSNR value for TE-SAGAN was slightly lower, and the network stability was considerably stronger without heavy fluctuations owing to the well-trained SAM module and WN layers.

To validate the pretrained model in detail, further experiments were conducted using remote sensing images of different scenes in the UC Merced Land Use dataset. As shown in

Table 1. Comparison of pre-ESRGAN and Pre-TE-SAGAN using part of the UC Merced Land Use dataset.

Dataset	Index	Pre-ESRGAN	Pre-TE-SAGAN
Harbor	SSIM	0.800	0.798
	PSNR	23.250	23.154
Runway	SSIM	0.819	0.821
	PSNR	30.574	30.724
Airplane	SSIM	0.819	0.821
	PSNR	28.748	28.800
Buildings	SSIM	0.805	0.806
	PSNR	26.124	26.138

Highest measures (PSNR [dB], SSIM) in bold [up-sampling scale 4×].

, our method yielded better results than ESRGAN. Simultaneously, as depicted in Figure 7, the current loss function only described the low-frequency content of the image, resulting in smooth outcomes that were still not entirely convincing. We further compared several reconstructed tree shadow textures in detail and concluded that the results of the proposed model were closer to the ground truth. However, both methods produced smooth and fuzzy images in general. For alignment with human visual perception, further training using an adversarial component based on the pretrained model is necessary.

To complement prior studies, bicubic interpolation and four representative deep-learning-based SR technologies, namely, ESPCN [30], EDSR [54], SRGAN [34], ESRGAN [35], and RFDNet [55], were used as comparison methods. The quantitative comparison results are presented in

Table 2. Comparison of Bicubic, ESPCN, EDSR, SRGAN, ESRGAN, RFDNet and TE-SAGAN on the Test-10000 set and part of UC Merced Land Use.

Dataset	Metric	Bicubic	ESPCN	EDSR	SRGAN	ESRGAN	RFDNet	TE-SAGAN
Test-10000	SSIM	0.558	0.500	0.564	0.539	0.575	0.600	0.583
	PSNR	22.854	22.141	22.855	22.695	23.500	23.650	23.700
	FID	136.312	148.199	95.226	55.252	32.587	81.866	23.771
Harbor	SSIM	0.716	0.635	0.668	0.618	0.659	0.749	0.707
	PSNR	20.314	19.339	19.302	18.814	19.765	21.760	20.770
	FID	147.225	169.257	205.605	227.557	135.349	115.247	121.527
Runway	SSIM	0.704	0.668	0.678	0.708	0.698	0.756	0.748
	PSNR	26.010	25.393	26.863	26.794	26.959	28.069	28.467
	FID	146.718	191.391	132.028	116.549	105.031	106.095	83.282
Airplane	SSIM	0.713	0.662	0.709	0.677	0.669	0.752	0.733
	PSNR	25.421	23.861	24.561	23.992	24.849	26.53	26.296
	FID	179.435	201.591	158.942	143.521	89.532	119.773	82.393
Buildings	SSIM	0.699	0.625	0.670	0.643	0.656	0.746	0.720
	PSNR	23.226	21.695	22.137	21.787	22.559	24.21	24.005
	FID	127.26	140.786	149.487	122.78	81.284	118.870	70.704
	Runtime (min)	/	107.8	3225.6	2292.0	3072.9	251.8	2288.7

For PSNR $\uparrow$ [dB] and SSIM $\uparrow$, the highest values are in bold, whereas the second-highest values are underlined. On the contrary, for FID $\downarrow$, the lowest values are in bold, whereas the second-lowest values are underlined.

. However, the mean PSNR values of our method were only 0.050 and 0.398 dB higher than the second-highest RFDNet algorithm for Test-10000 and runway, respectively. Moreover, the remaining mean PSNR values of our model were lower than RFDNet by 0.990, 0.234, and 0.205 dB for the harbor, airplane, and buildings sub-datasets, respectively, and all average SSIM scores of our model were lower than highest RFDNet by 0.017, 0.042, 0.008, 0.019, and 0.026. Instead, in the FID score term, where the lower score implies that generated images are closer to real reference images, most of the FID scores in our network were significantly lower than those in the second-lowest network, ESRGAN, by 8.816, 21.749, 7.139, and 10.580 for Test-10000, runway, airplane, and buildings, respectively, expect for the harbor sub-dataset, which was higher than RFDNet by 6.28. These findings indicate that compared with the other methods, TE-SAGAN focuses on sharpening the super resolution image while keeping a relatively high level of PSNR and SSIM image fidelity evaluations. Our method obtains a better objective rating for the four benchmark datasets. The program runtime is a non-negligible metric for evaluating network performance. ESPCN takes the shortest runtime with fast training using sub-pixel convolution, and the recent RFDNet belongs to an improved lightweight distillation network with a relatively low training runtime. Among the models with runtime lasting more than one day (1440 min), TE-SAGAN takes the shortest training time, which shortens around a quarter of the runtime than ESRGAN. The weight normalization (WN) layer introduced in this paper ensures the network stability in inputting mini-batch data to accelerate training time. Training EDSR takes the longest time. These findings indicate that TE-SAGAN shows improvement in network runtime.

To systematically elucidate how the introduced WN and SAM affect the network performance, several TE-SAGAN ablation experiments were performed using the following networks: baseline (ESRGAN), baseline combined with the WN layer (Baseline + WN), baseline combined with the WN layer and SAM (Baseline + WN + SAM), and the final combination of SAM, WN layer, and texture loss term (Baseline + WN + SAM + Texture). Based on the results presented in Figure 8, the following conclusions were drawn. First, the addition of the WN and SAM modules to the baseline network improved the PSNR (Figure 8a) and SSIM (Figure 8b) values for several sub-datasets in the UC Merced Land Use dataset. This implies that the added modules are effective in improving the quality and fidelity of the reconstructed image. Second, the introduced texture loss focuses more on restoring texture details (FID score in Figure 8c) than on enhancing PNSR and SSIM values. Furthermore, although our network parameter was slightly larger than that of the baseline model during the experiment, our network showed higher training efficiency. In summary, the ability of WN and SAM modules to rapidly and accurately extract image features allows TE-SAGAN to reconstruct images with high fidelity, and texture loss further adds details consistent with human observation to the reconstructed image. We further believe that the added modules are associated with improved network fitness.

3.3.2. Qualitative Evaluation

Faithful SR images has always been the focus of many image-processing tasks. As mentioned above, the PSNR and SSIM metrics obtained with the networks were different from human observations. Therefore, to further improve the texture and edge of reconstructed remote sensing images, qualitative visual comparisons were performed (Figure 9 and Figure 10). The overall quality of SR remote sensing images obtained using our method was significantly higher than those obtained via other algorithms. Figure 9 shows the processing results of our method for the test dataset (Test-10000), including the villa surroundings, car windows in the parking lot, and shade trees beside the road. In detail, bicubic interpolation [14] filled pixel information with a fixed computational paradigm, producing a blurry image. SRGAN [34] and ESRGAN [35] yielded more visually pleasant processing images than ESPCN [30] and EDSR [54] owing to their excellent depth feature extraction module and perceptual loss. However, the textures recovered by SRGAN and ESRGAN were inadequate and erroneous (e.g., car windows and tree shadows) because they lack expression and constraints on detailed and realistic textures. In particular, RFDNet is biased with respect to objective PSNR and SSIM showing higher scores, but generating super-resolution images with unclear textures. Our method was equipped with the WN layer and the Gram matrix for texture loss calculation, which significantly increased the calculation of high-frequency information correlation between SR and HR images and limited the pixel and texture deviation. Thus, our method yielded finer-grained and more realistic images.

To examine the recovery status of image outlines, experiments were conducted using four representative remote sensing images from the UC Merced Land Use dataset, including street corners, rooftops, harbor ships, traffic lines, and other scenes, and the results are presented in Figure 10. SRGAN and ESRGAN produced more realistic edges than bicubic, ESPCN, EDSR, and RFDNet. However, the textures recovered by SRGAN and ESRGAN still contained unreal artifacts, as they rarely consider the spatial information of remote sensing images. In particular, RFDNet obtained high scores in objective PSNR and SSIM but generated super-resolution images with unclear outlines. In contrast, the SAM module in our proposed TE-SAGAN enhanced the connection between distant pixels, resulting in clearer, sharper edges in the reconstructed remote sensing images. Interestingly, even though several of the original HR images were slightly blurry, our method produced images with clearer and more natural borders than the original.

4. Conclusions

SR images have been widely applied for extracting information from remote sensing images. In this study, we elucidated why some algorithms lose control over edges and internal textures in the reconstructed remote sensing images even after smoothing. We proposed an RRDB-based texture-enhancement network, TE-SAGAN, which integrates the SAM and WN into the GAN framework. The SAM module effectively combines wide-area and long-range image information training. The WN layer stabilizes the reconstructed image quality using mini-batch data, thereby reducing the contrast gaps and artifacts in the generated images. In addition, the Gram matrix imposes a texture loss constraint to refine the reconstructed texture. The results for PSNR, SSIM, and FID scores of TE-SAGAN during training using our own and public benchmark remote sensing image datasets indicate improved performance. This study also suggests that our proposed method yields clearer image edges and more realistic internal details compared to existing methods. Of note, the main limitation of our current study is that we focused on rebuilding texture details and borders by increasing the model parameters. Further exploratory research is needed to reduce the model size while achieving equivalent or even superior results.