SAR and Optical Image Registration Based on Uniform Feature Points Extraction and Consistency Gradient Calculation

Abstract

Synthetic aperture radar (SAR) satellites have an active sensor on board, which emits electromagnetic signals and measures the strength and time delay of the returned signal backscattered from ground objects. Optical images have rich spectral information, but it is easily affected by atmospheric attenuation and weather conditions. Thus, the study of the registration between these two images is of great significance. We present a novel method for SAR and optical image registration. In the stage of feature points extraction, the method combines phase consistency intensity screening and scale space grid division to obtain stable and uniform feature points from the image. During the stage of feature description, the method employs the extended phase consistency method to calculate the gradient amplitude and direction of the image, and improves the correctness of the main direction calculation and descriptor construction. Experimental results demonstrate its superior matching performance with respect to the state-of-the-art methods.

1. Introduction

Remote sensing technology is a critical means of earth observation from remote sensing devices mounted on artificial satellites, airplanes and other aviation or spacecraft [1], because it can collect electromagnetic information of ground targets on a large scale to help humans obtain observations that traditional technology cannot achieve. As a result, this technology is widely used in national economic and military aspects such as meteorological observation, map surveying and mapping and military investigation [2,3,4,5,6,7]. Remote sensing image registration is a method to establish a matching relationship between two or more remote sensing images captured by different angles or different sensors in the same scene at different times. It is a pioneer task for subsequent remote sensing image splicing, fusion or transformation detection.

With the rapid development of remote sensing technology, the acquisition methods of remote sensing images are gradually showing a diversified trend. In practical applications, the imaging methods of heterogeneous remote sensing images are different and the information contained is not the same. How to use heterogeneous remote sensing for effective fusion and complementation of information in images has attracted more and more attention in recent years. Synthetic Aperture Radar (SAR), as an active imaging radar, has the advantages of all-weather imaging due to its strong penetrating ability [8,9,10]. It can penetrate clouds and haze and other occlusions to break through the limitations of optical imaging, but is not easy to obtain the perceived characteristics of ground targets. In contrast, optical remote sensing images can obtain spectral information such as rich gray-scale textures of ground targets under good imaging conditions, good visual interpretation functions and have great advantages in ground target recognition and classification. Therefore, it has become an important research topic in the field of remote sensing image processing to realize the complementary advantages of multi-source images through effective heterogeneous remote sensing image registration and fusion technology [11]. At present, due to the great difference in imaging mechanism between SAR and optical images, the geometric and radiation characteristics of the images are different. In addition, the multiplicative speckle noise inherent in the SAR image brings many difficulties to the registration of SAR and optical images. Therefore, it is of great significance to study the registration of SAR and optical images.

The SIFT algorithm [12] is a classic local invariant feature extraction method. The algorithm mainly detects extreme points in the Gaussian difference scale space as stable feature points, and at the same time it calculates the gradient direction histogram in the neighborhood of the image to construct the feature vector. In terms of its similar algorithms, the extraction of robust feature points and the calculation of basic gradient features are of great significance to the performance of this method. As mentioned above, due to the different imaging mechanisms and acquisition methods of optical and SAR images, there is a large non-linear radiation difference between these two kinds of images [13]. At the same time, there is an inherent multiplicative noise in SAR images, which will cause different extractions. The source image has a small number of feature points with the same name and poor robustness [14]. In addition, the main direction of the calculated feature points and the constructed descriptor are unreliable due to gradient calculation errors, which leads to a lower correct matching rate in the feature point matching stage.

To solve the problems mentioned above, this paper conducts research from the two aspects of obtaining robust feature points with the same name and calculating the consistency gradient of heterogeneous images, and proposes a SIFT registration method based on uniform extraction of feature points and gradient consistency. In the stage of feature points extraction, the method combines phase consistency intensity screening and scale space grid division to obtain stable and uniform feature points from the image. In the feature description stage, the method employs the extended phase consistency method to calculate the gradient amplitude and direction of the image, and improves the correctness of the main direction calculation and descriptor construction compared with the original SIFT algorithm.

2. Proposed Method

Compared with the original SIFT algorithm, the method in this chapter first uses a combination of phase consistency intensity screening and uniformly distributed feature point detection to extract uniform and robust candidate points from the image as stable feature points. Secondly, the extended phase consistency method is used to calculate the gradient magnitude and direction of the image, and the main direction and descriptor of the feature points are calculated, accordingly. Finally, in order to improve the uniqueness of the feature point descriptor, a 136-dimensional GLOH-like descriptor is constructed by collecting the histogram of the neighborhood gradient around the feature point in polar coordinates. Finally, it is according to the dual-match and two-way matching and the eigenvector matching strategy combined with the RANSAC method, which obtains the deformed model parameters and realizes the registration between SAR and optical images. The workflow of the proposed algorithm is shown as Figure 1.

2.1. Uniform Robust Feature Point Extraction

When processing the registration of SAR and optical images, the distribution of feature points has a great influence on the registration result, which obtains a suitable number of evenly distributed feature points and retains stable and highly repeatable feature points in heterogeneous images. It can accurately estimate the geometric transformation model between images and improve the registration accuracy. In order to solve the problem of uneven distribution of feature points in a single image, this section introduces the scale space scale factor and coordinate space blocking strategy, and uses the phase consistency intensity value to filter the feature points, and obtain as much as possible from heterogeneous images with robust feature points of the same name. The steps of the algorithm are as follows:

• The upper limit of the total number of feature points is determined by the original image size N.
• Calculate the phase consistency response intensity map of the original image I_pc.
• Calculate the scale parameters of each group of each layer to construct a Gaussian pyramid.
• Calculate the feature point set of each group of each layer in the scale space I_ol: (a) Determine the upper limit of the number of feature points in this layer; (b) detect extreme points as initial candidate points of interest; (c) divide the layer of the Gaussian image into regular grid cells and determine the features that each cell needs to retain the number of points in the n_cell_i; (d) position the coordinates of the candidate points of interest and analyze the principal curvature to eliminate unstable points on the edge; (e) find the I_pc according to the coordinates to obtain the PC intensity value of the reserved interest point, and according to the PC intensity, the degree value and the absolute value of the Gaussian difference response are sorted, and the top n_cell_i are retained as the feature points in the cell; (f) summarize the feature points in all cells to obtain the feature point set P_ol of the current layer I_ol.
• Summarize the feature points of all scale layers in the scale space to obtain a set P.

According to the SIFT feature points obtained by the above feature point detection algorithm, because in the process of feature point detection, the number of feature points in the scale space distribution and coordinate space distribution is consciously restricted, and then the stability and saliency of the feature points can be characterized by the index screens of the feature points, and finally obtains a uniform and robust feature point set.

2.2. Main Direction Calculation and Descriptor Construction

In SIFT and its improved algorithm, specifying the main direction for the feature point can make it a rotation invariant, and the calculation of the main direction and the construction of the descriptor are all through the feature points in the differential pyramid space image to count the gradient histogram of the neighborhood. The image is obtained, so the gradient direction between the SAR and optical images should be consistent. However, as for the registration of multi-sensor images, due to the huge difference in radiation characteristics between each other, for example, the gradient direction of the matching area are often different. In order to avoid this problem, which may cause registration failure, in this paper, the gradient calculation method based on phase consistency is used to extract the consistent gradient features of SAR and optical images, and we use this basic feature to calculate the main direction of feature points by means of statistical histograms and build descriptors.

In this section, when calculating the main directions of feature points, we choose the method that approximates the original SIFT algorithm. After the precise coordinates of the feature points and the corresponding scale coordinates are obtained through the detection algorithm, the consistent gradient algorithm is used to calculate the neighboring neighbors of the feature points from the corresponding scale space. In the gradient direction histogram of the domain, the direction corresponding to the maximum peak value is selected as the main direction of the feature point, and the direction with the peak value in the histogram greater than 80% of the main direction peak value is used as the auxiliary direction to improve the stability of the algorithm.

When constructing the feature point descriptor, this paper makes some modifications to the selection and division of the neighborhood block of the SIFT feature point. The SIFT native structure description method divides the pixels in the radius neighborhood around the feature points into 4∗4 sub-blocks in space, and counts the gradient direction histograms of eight directions in each sub-block, and finally divides the directions of all sub-regions into 4∗4 sub-blocks. The gradient information combination is a 128-dimensional descriptor. According to the research of Krystian Mikolajczyk et al. this division method ignores the positional relationship between the pixels within the neighborhood block. Therefore, this article adopts the method that selects the image within a certain radius around the feature point as its neighborhood block. After rotating the neighborhood block to the main direction, the pixels in the neighborhood block are divided in the polar coordinate system according to their distance from the center point into different fan-shaped grids, so that 17 grid areas are obtained. For each fan-shaped sub-region and each pixel in the center circle, use the corresponding gradient algorithm to calculate the gradient magnitude and direction, and then assign the gradient value in the sub-block to eight directions, and finally after the gradient information of each direction in the 17 sub-regions is concatenated and normalized, 136 dimensions are obtained, so that the descriptors obtained have better robustness and uniqueness.

2.3. Feature Points Matching Strategy

After the feature point descriptors are constructed, the feature vector sets of SAR and optical images are obtained, respectively. It is necessary to establish a correct match based on the similarity measurement criterion between feature vector descriptors and the matching relationship. This chapter will first use the two-way matching algorithm to establish the initial matching relationship, and then use the random sampling consensus algorithm to remove the wrong matching point pairs, and calculate the parameters of the transformation model by retaining the correct matching point pairs. The main process is as follows:

2.3.1. Dual-Match

Assuming that the feature vector sets obtained in the previous feature extraction stage are V_s and V_o, respectively, the simplest way to calculate the similarity between feature vectors is to calculate the Euclidean distance. However, in the actual matching process, due to noise and scene occlusion problems, some feature points do not have correct matching points corresponding to them, and the points with the closest Euclidean distance found in the corresponding images may be mismatched points. Therefore, Lowe proposed the k-nearest neighbor ratio method to eliminate the false matches [10].

According to experiments, this method can achieve good results in most scenes. However, this one-way matching method is prone to “one-to-many” wrong matching phenomenon due to similar neighborhoods in different locations. A certain eigenvector in V_s has multiple eigenvectors that satisfy the k-nearest neighbor ratio method in the V_o set, which will cause a lot of trouble for the subsequent error matching elimination. Therefore, this paper adopts a two-way matching strategy [15] to improve the accuracy of matching point pairs.

The specific process is as follows: (a) For each feature vector L_a in V_s, calculate its Euclidean distance with each feature vector in V_o by traversing. Then, sort and find the closest Euclidean distance Dist_f1 and the next closest Euclidean distance Dist_f2 and the corresponding vectors L_b and L_b’. If Dist_f1/Dist_f2 ≤ Threshold, then L_a and L_b match. Traverse all the feature vectors in V_s to obtain the forward matching set of M_forward. (b) For each feature vector L_b in V_o, calculate its Euclidean distance with each feature vector in V_s by traversing. Then, sort and find the closest Euclidean distance Dist_f1 and the next closest Euclidean distance Dist_f2 and the corresponding vectors L_a and L_a’. If Dist_b1/Dist_b2 ≤ Threshold, then L_a and L_b match. Traverse all the feature vectors in V_o to obtain the forward matching set M_backward. (c) Set up two loops, traversing M_forward and M_backward, respectively. If the set of matching satisfies the relationship Dist_f1 = Dist_b1, then the set of matching point pairs is retained to obtain a set of bidirectional matching point pairs.

2.3.2. Random Sampling Consistency Method Purification

In the rough matching point set after the previous two-way matching, there are still some wrong matching point pairs. This includes two types. The first type of mismatched points come from unmatched regional scenes, which are mismatched due to approximate descriptors; the second type of mismatched point pairs come from matched regional scenes, however, due to noise and local areas of the image deformation leads to a deviation in positioning. If such matching point pairs are not eliminated, the calculated transformation model will deviate from the actual situation, resulting in a decrease in registration accuracy. For the above two considerations, it is necessary to adopt a certain method to purify the set of rough matching points. Among the many methods, the random sampling consensus algorithm proposed by Fischler et al. [16] is the most commonly used. This method is a robust parameter estimation method. Its main idea is to fit the model parameters in an iterative manner in the search set, set a threshold to filter the interior points, continuously expand the set of support points, and find all the data pairs that meet the model parameters to obtain the optimal solution.

3. Experimental Results and Analysis

To evaluate the performance of the proposed method, three pairs of SAR and optical images are experimented. The test data consists of different characteristics including different resolutions, incidence angles, seasons etc. Experimental results are shown in Figure 2, Figure 3 and Figure 4. To quantitatively evaluate the registration performances, we adopt the root-mean-square error (RMSE) [12] between the corresponding matching keypoints, and it can be expressed as:

$$\mathrm{RMSE}=\sqrt{\frac{1}{n}{\displaystyle \sum _{i=1}^n{(x_i-{x_i}^{\prime })}^2+}{(y_i-{y_i}^{\prime })}^2}$$

(1)

where (x_i,y_i) and (x_i′,y_i′) are the coordinates of the ith matching keypoint pair; n means the total number of matching points. In addition, the correct matching ratio (CMR) is another effective measure, which is defined as:

$$CMR=\frac{correctMatches}{correspondences}$$

(2)

“correspondences” is the number of matches after using PROSAC, “correctMatches” is the number of correct matches after removing false ones. The results of quantitative evaluation for each method are listed in

Table 1. Quantitative comparison of the proposed method with other SIFT-based algorithms.

Image No.	Method	CMR/%	RMSE/Pixel
1	SIFT	64.79	1.1344
	SAR-SIFT	79.65	0.9276
	Proposed	88.21	0.5128
2	SIFT	67.58	1.3153
	SAR-SIFT	71.29	1.1452
	Proposed	81.89	0.8127
3	SIFT	62.46	0.9008
	SAR-SIFT	75.34	0.7097
	Proposed	89.73	0.4982

.

It can be observed from Table 1 that the correct matching rate obtained by the SIFT and SAR-SIFT algorithms is relatively low. The proposed algorithm can obtain more correct matching point pairs, which shows that this algorithm is better in suppressing the radiation difference between optical and SAR images. Compared with the other two algorithms, the CMR of this algorithm is improved by 88.21%, 81.89% and 89.73%, respectively in the three groups of images, and the RMSE is reduced by 0.5128, 0.8127 and 0.4982, respectively.

4. Discussion

The original SIFT method uses the gradient calculation method of the difference operator to process the SAR and optical images. Due to the influence of the non-linear radiation difference of the heterogeneous image, the gradient magnitude and direction calculated in the SAR image by this calculation method are quite different from the calculation result in the optical image. At the same time, due to the SAR due to the influence of image noise, the feature points detected by this method from heterogeneous images are often poor in stability, and the ratio of control points with the same name is low, which is easy to cause mismatches. These all cause the phenomenon that the correct rate of feature points matching by this method is not high.

In this chapter, the SIFT registration method based on uniform extraction of feature points and gradient consistency has successfully dealt with the shortcomings of the SIFT algorithm. Using the combination of phase consistency intensity screening and uniform distribution feature point detection, it improves the robustness of feature points, while taking into account the uniformity of the feature points in the coordinate space. In addition, the extended phase consistency method is used to calculate the gradient amplitude of the image; thus, the accuracy of the main direction calculation of the original SIFT method is improved. In the descriptor construction process, the descriptor constructed by dividing the sector area under polar coordinates has better reliability. In this way, the feature point matching method is optimized from the feature point extraction to the feature point description link, so the highest matching success rate and registration accuracy can be obtained.

5. Conclusions

In this paper, we present a novel method for the SAR and optical image registration. In the stage of the feature points extraction, the method combines the phase consistency intensity screening and scale space grid division to obtain stable and uniform feature points from the image. In the feature description stage, the method employs the extended phase consistency method to calculate the gradient amplitude and direction of the image, and improves the correctness of the original SIFT algorithm of the main direction calculation and descriptor construction. Experimental results demonstrate its superior matching performance with respect to the state-of-the-art methods.