Next Article in Journal
Decoupling of Temperature and Strain Effects on Optical Fiber-Based Measurements of Thermomechanical Loaded Printed Circuit Board Assemblies
Previous Article in Journal
LPI Radar Detection Based on Deep Learning Approach with Periodic Autocorrelation Function
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Communication

Fast Automatic Registration of UAV Images via Bidirectional Matching

1
Yangtze Delta Region Institute (HuZhou), University of Electronic Science and Technology of China, Huzhou 313099, China
2
School of Electrical and Information Engineering, Panzhihua University, Panzhihua 617000, China
3
School of Automation Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
*
Author to whom correspondence should be addressed.
Submission received: 7 September 2023 / Revised: 1 October 2023 / Accepted: 16 October 2023 / Published: 18 October 2023
(This article belongs to the Section Sensing and Imaging)

Abstract

:
Image registration plays a vital role in the mosaic process of multiple UAV (Unmanned Aerial Vehicle) images acquired from different spatial positions of the same scene. Aimed at the problem that many fast registration methods cannot provide both high speed and accuracy simultaneously for UAV visible light images, this work proposes a novel registration framework based on a popular baseline registration algorithm, ORB—the Oriented FAST (Features from Accelerated Segment Test) and Rotated BRIEF (Binary Robust Independent Elemental Features) algorithm. First, the ORB algorithm is utilized to extract image feature points fast. On this basis, two bidirectional matching strategies are presented to match obtained feature points. Then, the PROSRC (Progressive Sample Consensus) algorithm is applied to remove false matches. Finally, the experiments are carried out on UAV image pairs about different scenes including urban, road, building, farmland, and forest. Compared with the original version and other state-of-the-art registration methods, the bi-matching ORB algorithm exhibits higher accuracy and faster speed without any training or prior knowledge. Meanwhile, its complexity is quite low for on-board realization.

1. Introduction

Because of its flexibility, high efficiency, and low cost, UAV (Unmanned Aerial Vehicle) remote sensing technology has gradually emerged in many fields, such as accurate agriculture, resources investigation, environment management, and disaster monitoring [1,2,3]. How to yield high-precision registered UAV images quickly has become an inevitable challenge to the wide application of UAV technology [4,5,6,7,8,9]. However, the high resolutions of UAV images have a great influence on the detection and matching of image feature points. Meanwhile, there is a series of parameters in collecting UAV images, such as flight speed, flight height, and weather conditions, which make detection and matching of image feature points more difficult to achieve. Hence, the automatic registration of UAV remote sensing images is worth studying comprehensively.
Recently, the issue of image registration has received considerable intensive attention from scholars. For instance, the SIFT (Scale Invariant Feature Transform) operator can obtain a sort of feature invariant to image resolution, rotation, and scaling [10]. However, the SIFT algorithm will yield 128-dimensional features, which is heavily time-consuming. Hence, the feature dimension in the SURF (Speeded-Up Robust Features) algorithm is reduced to 64 through wavelet transform, which efficiently enhances the registration speed [11,12]. Moreover, focused on matching different features, Zhang et al. combined the image enhancement technology with the SURF algorithm to obtain better feature points and matching efficiencies [13]. Wei et al. divided an image into super-pixels to calculate the information entropy of each region [14]. The redundancies in feature points are diminished according to the values of their information entropy. Zhang et al. proposed a threshold evaluation strategy to extract SURF rough matching points and employed the RANSAC (Random Sample Consensus) algorithm to eliminate gross errors [15]. Furthermore, in order to constrain the regions for fine matching, a similarity function of Delaunay triangles constructed by Harris points is introduced to obtain matched triangle pairs. At present, ORB (the Oriented FAST and Rotated BRIEF) is a fast local feature detection operator [16], which is improved based on the FAST (Features from Accelerated Segment Test) [17] and BRIEF (Binary Robust Independent Elemental Features) algorithm [18]. It overcomes the lack of scale and rotation invariances in FAST and is faster than most classical image registration algorithms. However, since it has the disadvantage of low accuracy, this work attempts to improve the comprehensive performances of ORB on UAV images. The main contributions of this work are presented below.
  • The ORB operator is utilized to extract image feature points since many fast registration methods cannot simultaneously provide high speed and accuracy for UAV images. Its complexity is quite low for on-board realization.
  • The KNN-based search and the Hamming distance are used in the initial matching of feature points; two bi-directional matching strategies are designed for their fine matching; and the progressive consistency algorithm (PROSAC) is exploited to remove false matches and fit a geometric transformation model to enhance registration accuracy.
  • The validation experiments were carried out on the UAV images from five typical scenes, which were acquired by different UAV image sensors. The results indicate that the proposed algorithm can achieve higher accuracy and faster speed than some existing fast registration algorithms.
The remainder of this article is organized as follows. Section 2 describes an improved ORB registration method. After images are down-sampled to speed up feature matching, an improved ORB operator and a bi-directional matching strategy are utilized to detect and match feature points in UAV images. After the initial matching, feature point pairs are obtained, and the PROSRC (Progressive Sample Consensus) algorithm is performed to screen out mismatched points to obtain an accurately matched point-pair set and final transform parameters [19]. In Section 3, our proposed algorithm is compared with some existing registration methods using UAV images acquired from different scenarios and sensors. The conclusion and discussion are presented in Section 4.

2. Methodology

Because of their high resolutions, it usually takes a long time for common image registration algorithms to process UAV images. However, there are often real-time requirements in many applications of UAV images. Therefore, based on an improved ORB operator, a bi-directional matching strategy is combined with PROSAC to pair extracted feature points, which can enhance the accuracies of the proposed registration algorithm. The registration procedure of our algorithm is illustrated in Figure 1.

2.1. Image Preprocessing

2.1.1. Down-Sampling

On account of the high resolutions of UAV images, an appropriate scaling factor is chosen to adjust the sizes of input images. If the initial size of an image is M × N, the width and height of the image are reduced to 1/n of the original one after being down-sampled by n. That is, the size of the obtained image becomes (M × N)/n, where the pixel values of each n × n region in the original image are replaced by the mean pixel value of the region. This operation can reduce the speeds of image loading and processing, which is beneficial to the real-time performances of registration algorithms.

2.1.2. Graying

It is known that the structure and texture characteristics of visible images usually will not vary with optical bands. For the sake of processing speeds, grayscale images are used as the basis of image registration in this work, and input RGB images are grayed according to the importance of three optical bands. The intensity of each pixel is generated from a weighted average of three different components. A graying instance of UAV images is given in Figure 2. Considering the sensitivity of human eyes to color information, the intensity of an arbitrary pixel can be denoted as
f ( i , j ) = 0 . 30 R ( i , j ) + 0 . 59 G ( i , j ) + 0 . 11 B ( i , j )
where R(i, j), G(i, j), and B(i, j) represent the three components at an arbitrary piont (i, j) in an input image [20].

2.2. ORB Brief

To ensure scale invariances, a Gaussian pyramid and patch centroid calculation based on FAST are combined in ORB. Moreover, an improvement on the BRIEF algorithm for rotation invariance is designed for constructing feature descriptors of ORB [21,22,23].

2.2.1. Feature Point Extraction

In terms of feature point detection, the kernel method of ORB is based on FAST [24,25,26]. If the difference in the gray level between the central pixel and one of its neighbors is significant enough, the point will be regarded as a feature point. This detection method of feature points not only preserves image features as much as possible but also greatly reduces computational complexity. It will bring about a significant improvement in the speed of the ORB algorithm. For an arbitrary detected feature point, the moment of its neighbor patch mpq is defined as
m p q = x , y x p y q I ( x , y ) , p , q { 0 , 1 }
where I(x, y) is the intensity of the pixel at (x, y). Then, the centroid coordinate C can be determined by
C = ( m 10 m 00 , m 01 m 00 )
Suppose that the feature point is O. The angle of the vector O C θ is regarded as the orientation of the feature point, as given below.
θ = arctan 2 ( m 01 m 10 )
where arctan2 is the quadrant-aware arctan [24].

2.2.2. Feature Descriptor Construction

After the feature points in images are found, an improved BRIEF algorithm, rBRIEF (rotation-aware BRIEF), is adopted in ORB to describe the features of feature points [27,28,29]. Binary coding is employed by BRIEF to construct feature description vectors, which is very helpful for accelerating calculation. Since BRIEF does not possess rotation invariance, the specific improvements on the BRIEF algorithm in ORB are as follows [24].
Give an M × M pixel smoothed patch P. A binary testing τ(P; u, v) can be defined as
τ ( P ; u , v ) = { 1 P ( u ) < P ( v ) 0 o t h e r w i s e
where P(u) and P(v) are the intensities of the pixels at two points u and v, respectively. For n pairs of points around a feature point, the generated feature descriptor fn(P) is an n-dimensional binary string, as follows.
f n ( P ) = 1 i n 2 i 1 τ ( P ; u i , v i )
According to the orientation of feature points, a corresponding matrix S is constructed as given in the following Equation.
S = [ u 1 , u 2 , , u n v 1 , v 2 , , v n ]
Since the orientation of the feature point is θ, the corresponding rotation matrix Rθ is defined by
R θ = [ cos θ sin θ sin θ cos θ ]
Then, if the matrix S is transformed by the rotation matrix Rθ, a new feature description matrix Sθ is obtained, which can be derived as below.
S θ = R θ S = [ cos θ sin θ sin θ cos θ ] [ u 1 , u 2 , , u n v 1 , v 2 , , v n ]
Consequently, the improved descriptor gn(P, θ) can be expressed as
g n ( P , θ ) = f n ( P ) | ( u i , v i ) S θ

2.3. Initial Matching and Screening

The initial matched point pairs are yielded by comparing the extracted feature points in the image to be registered and its reference image. In practice, the comparison of image point features is realized based on the similarities of feature points. The Hamming distance is chosen to describe the similarities between different points in this research [30,31,32]. To enhance the efficiency of initial matching, a fast search based on the K-Nearest Neighbor (KNN) algorithm is adopted in this work [33,34]. Unlike common violent matching methods, it searches for a corresponding point with the highest similarity for a point to be registered in a determined region of the reference image. Their Hamming distances to the point to be registered are calculated. Two matched points, optimal and suboptimal, are found by KNN. Lowe’s algorithm is adopted to compare whether the ratio of the two distances is less than a given threshold [35]. If so, the current matching relationship will be regarded as an acceptable matched point pair. Otherwise, the matching relationship will be eliminated.

2.4. Bi-Directional Fine Matching

The specific means of feature point detection and feature vector construction applied in ORB significantly contribute to processing speeds. Still, the algorithm’s performances are poor in terms of accuracy. Hence, it is necessary to accomplish the fine matching of feature points through an appropriate strategy. However, the matching procedures of the most common registration methods are unidirectional, which only search matching objects for feature points in reference images. There are many errors and omissions in their matching results. Focused on these problems, two bi-directional matching strategies are proposed in this work, as depicted in Figure 3.
Strategy 1: The primary idea of the first bi-directional matching strategy is to remove wrongly matched points in initial matching results. As presented in Figure 3a, its specific procedure is as follows. Suppose that the feature points detected in an image to be registered are marked as {p1, p2, …, pn}, and the feature points detected in its reference image are marked as {q1, q2, …, qm}. After forward matching, a set of matched feature points in the two images is obtained and denoted as PQ = {piqj, 1 ≤ In and 1 ≤ jm}. Similarly, after backward matching, a set of matched feature points in the two images can be represented by QP = {qspt, 1 ≤ sn and 1 ≤ tm}. Then, the backward matching results are traversed according to the forward matching result. If it is found that there are two identical pairs of matched points, i.e., piqj and qtps with i = s and j = t, the point pair piqj (or qtps) can be regarded as a correct matching relationship and will be preserved. Otherwise, the matched point pairs in the backward matching result will be eliminated. Thereby, a new set of matched feature points P ˜ Q ˜ can be built more reliably through this bi-directional matching strategy.
Strategy 2: The other bi-directional matching strategy is to compensate missed feature points into unidirectional results. The concrete realization is elaborated as follows. If there is a matched point pair qspt (1 ≤ sn and 1 ≤ tm) in the backward matching result, but qs and pt do not appear in the forward matching result PQ, qspt will be deemed as a matching pair missed in forward matching. Accordingly, the point pair ptqs will be added to the set PQ. Then, a new set P ˜ Q ˜ containing more matched feature points is obtained until all the point pairs in the backward matching result are examined. The detail matching flow is elaborated in Figure 3b.

2.5. False Match Elimination and Transform Model Fitting

In this work, PROSAC is employed to remove mismatched feature points further from bi-directional rough matching results [36]. It is known that the classical RANSAC method is realized by random sampling of rough matching results to fit transform models. This method is random and heavily depends on pre-set iteration times for fitting models. Nevertheless, PROSAC is constructed by sampling an increasing set of optimally matched point pairs, which have been ranked based on a certain similarity measure. Sampling and fitting models based on top-ranked point pairs help enhance the success rates of obtaining correct models. Meanwhile, the randomness of the algorithm decreases obviously. Specifically, matched point pairs in P ˜ Q ˜ are sorted from near to far according to Hamming distances of feature descriptors. Then, four point pairs are randomly chosen from the top m pairs in the reordered feature point-pair set, which can be used to fit a 3 × 3 homography matrix H, which can satisfy the following condition.
[ x y 1 ] H 3 × 3 [ x y 1 ]
where (x, y) and (x’, y’) correspond to point coordinates in the image to be registered and the reference image, respectively. According to the current parameters of the transform matrix H, the coordinates of all other points in the image to be registered are transformed by H into the coordinate system of the reference image. Then, these points are classified into inner or outlier points according to the differences between their projection coordinates and corresponding point coordinates in the reference image. These inner points are refined matched point pairs, and H will be renewed in light of them. Repeatedly, the inner point set and H will be updated until the pre-set times of iterations are reached.

3. Results

The experimental hardware platform includes an Intel Core i5-4590K processor with a main frequency of 3.5 GHz and 16 GB RAM. The experimental software environment contains a 64-bit operating system Windows 10, a programming tool Visual Studio 2017, and an open-source library OpenCV 3.20. The comprehensive performances of the proposed registration algorithm are verified in the experiments by using UAV images of different sensors acquired from different scenes, including urban, road, building, farmland, and forest. Moreover, the improved ORB algorithm is compared with the existing popular algorithms, including SIFT [37], SURF [38], KAZE (a Japanese word) [39], AKAZE (Accelerated KAZE) [40], and its original version in terms of speed and accuracy.

3.1. Determining the Number of Feature Points

Since the number of feature points extracted by the ORB operator is optional, a comparative experiment is carried out to determine an appropriate number of feature points for optimal registration effects. Hence, the UAV images of five scenes are registered through the different numbers of feature points extracted by the original ORB algorithm. The relationship between the number of feature points and running time or Root Mean Square Error (RMSE) are separately plotted in Figure 4a,b. RMSE can be denoted as follows [2].
R M S E = 1 n i = 1 n ( T ( y i , θ ) x i ) 2
where xi and yi (for i = 1, 2, …, n) stand for one of the n matching point pairs from the image to be registered and the reference image. T is a transform model, and θ is the model parameter vector. is the Euclidean distance between two points. Generally, the smaller the value of RMSE, the higher the registration accuracy. From Figure 4a, it can be found that the running time of the algorithm is directly proportional to the number of feature points. Meanwhile, it can be seen from Figure 4b that there exist minimum RMSE values individually at 1000, 2000, and 2500, and most of the values of RMSE for the five senses reach the lowest when the number of feature points is 1000. Thereby, considering running time and registration accuracy, the number of feature points to be extracted is uniquely set at 1000 for different scenes in the subsequent registration experiments.

3.2. Registration of Scene Images

The sources and parameters of UAV images used in the registration experiments are listed in Table 1. The urban and building scene images are, respectively, selected from the high-altitude and near-ground drone data of an ISPRS benchmark for multi-platform photogrammetry [41]. The image pairs of a road scene are the test images provided by Pix4Dmapper v.2.0 software. The farmland and forest scene images are self-collected. Particularly, the image pair of a farmland scene was acquired by the Parrot Sequoia multispectral camera with four bands, including green, red, red-edge, and near-infrared. Its red edge images are employed as blue components in the registration of this work. The two images of each scene image-pair were taken at different angles with an overlap. The image pair of the urban scene possesses many details and is quite complex, as shown in Figure 5. As mentioned before, 1000 pairs of feature points are extracted in the two images, and only 99 pairs of points are left after initial screening, as displayed in Figure 5c.
Based on the result of rough matching, two bi-directional matching strategies are adopted in fine matching, as shown in Figure 6. There are 60 matched point pairs preserved after screening out 39 point pairs using the first bi-directional matching strategy. After processing by the second bidirectional matching strategy, 121 matching point pairs remain, including 22 supplemental point pairs.
Then, the elimination results of false matches using the PROASC algorithm are shown in Figure 7, and the final registration results are presented in Figure 8. Since some pixel points in transformed images cannot be assigned directly to new grid coordinates, bi-linear interpolation is utilized to generate stitched images [42]. For the result of the first bi-directional matching strategy, 23 pairs of matched point pairs remain, and 37 pairs of matching point pairs are removed by the PROSAC algorithm. In this work, the running time and RMSE of the registration algorithm are chosen as the main indicators in performance comparisons. The running time of the registration process is 3.922 s, and the RMSE is 0.9926 pixels. The obtained transform matrix is as follows.
H = [ 1.000727295875549 0.000115774389996 675.5347900390625 0.000406843028031 1.000958800315857 118.6014556884766 0.000000566664652 0.000000197085213 1 ]
For the result of the second bi-directional matching strategy, 45 pairs of matching point pairs are left, and 76 pairs of matching point pairs are removed by the PROSAC algorithm. The registration time is 4.031 s, and the RMSE is 1.0750 pixels. The corresponding perspective matrix is denoted as
H = [ 1.000828385353088 0.000052663752680 675.6309814453125 0.000271332741249 1.001049280166626 118.7060470581055 0.000000538613107 0.000000276266348 1 ]
The registration accuracies of different registration algorithms for different scenes are listed in Table 2. For the five scene images, it can be seen that regarding accuracies, the proposed registration algorithms can provide obvious improvements on the original ORB algorithm, and the method based on the first bi-directional matching strategy exceeds all the other algorithms. For the scene images of urban areas, buildings, and farmlands, our algorithm even can achieve sub-pixel registration accuracies. In addition, it is revealed that the accuracy improvements of the proposed methods are related to the scenes of the experimental images. If there are rich details in images such as those found in urban, road, and building images, the accuracies can increase by more than 20.0% compared to the original ORB algorithm. If the images have fewer details, such as farmland and forest images, the accuracy enhancements are not outstanding, but still about 10.0%. The reason may be that the more details in images, the greater the differences in extracted feature points. Furthermore, the qualities of matched feature point pairs can be improved greatly after coarse and fine screening so that registration accuracies increase.
The time consumption comparison of different registration algorithms is presented in Table 2. As can be seen, the ORB algorithm has obvious advantages in running time over the other five algorithms. The reason may be that the high resolutions of these experimental UAV images result in more feature points being detected by the other five algorithms. Consequently, it takes a lot of time for these methods, especially for the KAZE algorithm, to detect and match feature points. However, it seems rational that the matching accuracy of the ORB algorithm is the lowest in Table 3 because its performance is closely related to the number of feature points. Our proposed registration methods cannot only inherit the advantage in speeds from the ORB algorithm but also achieve excellent registration qualities by combining the bi-directional matching strategies and a false-match elimination method. Hence, it can be drawn that our proposed registration algorithm realizes accuracy enhancement at the small cost of processing speeds.
In addition, the numbers of matched point pairs in the five scene images after different process stages are illustrated in Figure 9. It is known that there are abundant detail features in the UAV images of urban areas, buildings, and roads. In contrast, the contents of the UAV images of farmlands and forests are relatively simple, and their feature textures are not enough. In general, a sufficient number of feature points need to be extracted by matching methods effectively for fitting transformation model parameters. For the scene images of urban areas, buildings, and roads, a relatively large number of representative feature points can be detected. Although it benefits the increases in registration accuracies, it also spends more time in additional calculations. For the images of farmlands and forests, fewer and lower representative feature points are detected, which leads to lower registration accuracies but less calculation time.
The results of bi-directional matching for the other four scene image pairs are in Figure 10, and the final stitched images are displayed in Figure 11. It can be observed from Figure 11a that in the road scene, there are slight mismatches for one road in the mosaicking result of the first bi-directional matching strategy. The reason may be that the less matched point is extracted by this strategy. In Figure 11b, there are some obvious boundary effects in the body of a transmission tower, which may be due to its incomplete appearance at the edge of the reference image. From Figure 11c,d, it can be thought that though there are no significant edge features such as points and lines in the scenes of farmland and forest, the proposed fast registration provides good results for these image pairs. It can be noted that only a visible mismatch for one road exists in the result of the first bi-directional matching strategy for the farmland scene. This may be owing to its insufficient matched point pairs, as shown in Figure 10c. However, some areas for road and farmland scenes appear blurred owing to the quality of input images. Specifically, the red edge images of the farmland scene regarded as blue components in registration are inherently not clear enough. In addition, imaging conditions, such as shooting heights, sensor characteristics, atmospheric visibility and wind speed, light intensity, and the stability of drone platforms, may all affect the quality of drone images.

4. Conclusions

Since the ORB algorithm is not able to balance accuracy and time in the registration of UAV-visible images, in this work, a fast automatic registration method for UAV images is proposed by combining the ORB operator with bi-directional matching strategies. The KNN-based search method and a similarity measurement are assembled in initial matching. Two bidirectional matching strategies are designed for fine matching, and the PROSAC algorithm is employed to remove mismatches. In order to verify its performance, the proposed algorithm is compared with the existing fast algorithm for the registration of UAV images from different sensors and scenes. From the experimental results, it can be thought that the improvements of this work are effective. The proposed registration algorithm can enhance matching accuracy for UAV images without requiring any training or prior knowledge while maintaining the high registration speeds of the original ORB algorithm. The maximum accuracy improvement for the experimental images reaches 25.95% for the building scene. In terms of processing time and accuracy, the bi-directional matching method of reducing matched point pairs slightly outperforms the method of increasing matched point pairs. In addition, the proposed registration algorithm needs no training or prior knowledge with low complexities, which is quite suitable for onboard realization.

Author Contributions

Conceptualization and methodology X.L. and X.W. (Xiao Wang); software and validation, X.W. (Xiao Wang) and Z.W.; formal analysis, X.L. and Z.W.; resources, W.Z.; data curation and writing—original draft preparation, X.L. and P.L.; writing—review and editing, X.L. and X.W. (Xufeng Wei); visualization and project administration, W.Z. and Y.J.; funding acquisition, W.Z. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the Science and Technology Program of Sichuan, grant numbers 2022YFQ0052 and 2023ZHCG0013.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The source codes involved in this work are available from the authors upon reasonable request.

Acknowledgments

We would like to thank the journal’s editors and reviewers for their kind comments and valuable suggestions to improve the quality of this paper.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Etezadifar, P.; Farsi, H. A new sample consensus based on sparse coding for improved matching of SIFT features on remote sensing images. IEEE Trans. Geosci. Remote Sens. 2020, 58, 5254–5263. [Google Scholar] [CrossRef]
  2. Luo, X.; Lai, G.; Wang, X.; Jin, Y.; He, X.; Xu, W.; Hou, W. UAV remote sensing image automatic registration based on deep residual features. Remote Sens. 2021, 13, 3605. [Google Scholar] [CrossRef]
  3. Meng, S. A robust approach for bridge crack image mosaic based on SURF algorithm. Comput. Sci. Appl. 2019, 9, 375–383. [Google Scholar] [CrossRef]
  4. Goel, P.; Dayal, P.; Patra, T.K.; Gupta, C. Automatic co-registration of aerial and satellite imagery using optimized minimum distance thresholds for matching. In Proceedings of the 2022 International Mobile and Embedded Technology Conference (MECON), Noida, India, 10–11 March 2022. [Google Scholar] [CrossRef]
  5. Gong, X.; Liu, Y.; Yang, Y. Robust stepwise correspondence refinement for low-altitude remote sensing image registration. IEEE Geosci. Remote Sens. Lett. 2021, 18, 1736–1740. [Google Scholar] [CrossRef]
  6. Liu, Y.; Gong, X.; Yang, Y. A multilayer fusion network with rotation-invariant and dynamic feature representation for multiview low-altitude image registration. IEEE Geosci. Remote Sens. Lett. 2021, 18, 1019–1023. [Google Scholar] [CrossRef]
  7. Liu, Y.; Liu, Y.; Wang, J. RFM-GAN: Robust feature matching with GAN-based neighborhood representation for agricultural remote sensing image registration. IEEE Geosci. Remote Sens. Lett. 2023, 20, 1–5. [Google Scholar] [CrossRef]
  8. Silva, L.H.F.P.; Junior, J.D.D.; Mari, J.F.; Escarpinati, M.C.; Backes, A.R. Non-linear co-registration in UAVs’ images using deep learning. In Proceedings of the 35th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), Natal, Brazil, 24–27 October 2022. [Google Scholar] [CrossRef]
  9. Sun, Y.; Qi, H.; Wang, C.; Tao, L. Image registration method based on generative adversarial networks. In Proceedings of the 2020 Eighth International Conference on Advanced Cloud and Big Data (CBD), Taiyuan, China, 5–6 December 2020. [Google Scholar] [CrossRef]
  10. Hou, X.; Gao, Q.; Wang, R.; Luo, X. Satellite-borne optical remote sensing image registration based on point features. Sensors 2021, 21, 2695. [Google Scholar] [CrossRef] [PubMed]
  11. Qiao, M.; Liang, X.; Chen, M. Improved SIFT algorithm based on image filtering. In Proceedings of the Journal of Physics: Conference Series, Sanya, China, 29–31 January 2021. [Google Scholar] [CrossRef]
  12. Zhao, M.; Chen, H.; Song, T.; Deng, S. Research on image matching based on improved RANSAC-SIFT algorithm. In Proceedings of the 16th International Conference on Optical Communications and Networks (ICOCN), Wuzhen, China, 7–10 August 2017. [Google Scholar] [CrossRef]
  13. Li, X.; Luo, X.; Wu, Y.; Li, Z.; Xu, W. Research on stereo matching for satellite generalized image pair based on improved SURF and RFM. In Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Online, 26 September–2 October 2020. [Google Scholar] [CrossRef]
  14. Wei, W.; Cheng, A.; Zhao, Y.; Zhang, G. Image registration algorithm based on superpixel segmentation and SURF feature points. In Proceedings of the 8th International Conference on Image Processing Theory, Tools and Applications (IPTA), Xi’an, China, 7–10 November 2018. [Google Scholar] [CrossRef]
  15. Li, X.; Zhu, J.; Ruan, Y. Vehicle seat detection based on improved RANSAC-SURF algorithm. Int. J. Pattern Recognit. Artif. Intell. 2021, 35, 2155004. [Google Scholar] [CrossRef]
  16. Al Tamimi, A.; Qasaimeh, A.; Qaddoum, K. Offline signature recognition system using oriented FAST and rotated BRIEF. Int. J. Electr. Comput. Eng. (IJECE) 2021, 11, 4095–4103. [Google Scholar] [CrossRef]
  17. Zhai, Y.; Liu, S.; Guo, X.W.; He, P.; Zhang, Z.H. Lunar image matching based on FAST features with adaptive threshold. In Proceedings of the 7th International Conference on Communications, Signal Processing, and Systems (CSPS), Dalian, China, 14–16 July 2018. [Google Scholar] [CrossRef]
  18. Calonder, M.; Lepetit, V.; Strecha, C.; Fua, P. Brief: Binary robust independent elementary features. In Proceedings of the Computer Vision–ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, 5–11 September 2010. [Google Scholar] [CrossRef]
  19. Yang, Z.; Qin, X. Image stitching technology based on SIFT, FLANN, and RPOSAC algorithms. In Proceedings of the 2023 4th International Conference on Computer Vision, Image and Deep Learning (CVIDL), Zhuhai, China, 12–14 May 2023. [Google Scholar] [CrossRef]
  20. Saravanan, C. Color image to grayscale image conversion. In Proceedings of the 2010 Second International Conference on Computer Engineering and Applications (ICCEA), Bali, Indonesia, 26–29 March 2010. [Google Scholar] [CrossRef]
  21. Li, S.; Wang, Z.; Zhu, Q. A research of ORB feature matching algorithm based on fusion descriptor. In Proceedings of the 2020 IEEE 5th Information Technology and Mechatronics Engineering Conference (ITOEC), Chongqing, China, 12–14 June 2020. [Google Scholar] [CrossRef]
  22. Liu, Y.; Wang, Y. A new ORB feature matching image algorithm based on non-maximum suppression method and Retinal sampling model. In Proceedings of the 2021 International Conference on Control, Automation and Information Sciences (ICCAIS), Xi’an, China, 14–17 October 2021. [Google Scholar] [CrossRef]
  23. Sun, C.; Qiao, N.; Sun, J. Robust feature matching based on adaptive ORB for vision-based robot navigation. In Proceedings of the 2021 36th Youth Academic Annual Conference of Chinese Association of Automation (YAC), Nanchang, China, 28–30 May 2021. [Google Scholar] [CrossRef]
  24. Rublee, E.; Rabaud, V.; Konolige, K.; Bradski, G. ORB: An efficient alternative to SIFT or SURF. In Proceedings of the 2011 IEEE International Conference on Computer Vision (ICCV), Barcelona, Spain, 6–13 November 2011. [Google Scholar] [CrossRef]
  25. Ihmeida, M.; Wei, H. Image registration techniques and applications: Comparative study on remote sensing imagery. In Proceedings of the 2021 14th International Conference on Developments in eSystems Engineering (DeSE), Sharjah, United Arab Emirates, 7–10 December 2021. [Google Scholar] [CrossRef]
  26. Setyadi, R.; Ranggadara, I. Augmented reality using features accelerated segment test for property catalogue. Telecommun. Comput. Electron. Control 2020, 18, 140–147. [Google Scholar] [CrossRef]
  27. Heo, H.; Lee, J.; Lee, K.; Lee, C. FPGA-based implementation of FAST and BRIEF algorithm for object recognition. In Proceedings of the 2013 IEEE International Conference of IEEE Region 10 (TENCON 2013), Xi’an, China, 22–25 October 2013. [Google Scholar] [CrossRef]
  28. Li, Q.; Han, P.; Liu, D. Anti-fake legend registration algorithm based on improved ORB. In Proceedings of the 2020 IEEE 9th Joint International Information Technology and Artificial Intelligence Conference (ITAIC), Chongqing, China, 11–13 December 2020. [Google Scholar] [CrossRef]
  29. Taranco, R.; Arnau, J.; Gonzalez, A. LOCATOR: Low-power ORB accelerator for autonomous cars. J. Parallel Distrib. Comput. 2023, 174, 32–45. [Google Scholar] [CrossRef]
  30. Li, J.; Lin, S.; Yu, K.; Guo, G. Quantum K-nearest neighbor classification algorithm based on Hamming distance. Quantum Inf. Process. 2022, 21, 18. [Google Scholar] [CrossRef]
  31. Mitsuya, S.; Nakashima, Y.; Inenaga, S.; Bannai, H.; Takeda, M. Compressed communication complexity of hamming distance. Algorithms 2021, 14, 116. [Google Scholar] [CrossRef]
  32. Qin, M. Hamming-distance-based binary representation of numbers. In Proceedings of the 2018 Information Theory and Applications Workshop (ITA), San Diego, CA, USA, 11–16 February 2018. [Google Scholar] [CrossRef]
  33. Cunningham, P.; Delany, S.J. k-Nearest Neighbour Classifiers–A Tutorial. ACM Comput. Surv. 2021, 54, 1–25. [Google Scholar] [CrossRef]
  34. de Lima, R.; Cabrera-Ponce, A.A.; Martinez-Carranza, J. Parallel hashing-based matching for real-time aerial image mosaicing. J. Real-Time Image Process. 2021, 18, 143–156. [Google Scholar] [CrossRef]
  35. Wang, X.; Cao, W.; Yao, C.; Yin, H. Feature matching algorithm based on SURF and Lowes algorithm. In Proceedings of the 39th Chinese Control Conference (CCC), Shenyang, China, 27–29 July 2020. [Google Scholar] [CrossRef]
  36. Ma, S.; Ma, X.; You, H.; Tang, T.; Wang, J.; Wang, M. SC-PROSAC: An improved progressive sample consensus algorithm based on spectral clustering. In Proceedings of the 2021 3rd International Conference on Robotics and Computer Vision (ICRCV), Beijing, China, 6–8 August 2021. [Google Scholar] [CrossRef]
  37. Lopez, A.; Jurado, J.M.; Ogayar, C.J.; Feito, F.R. A framework for registering UAV-based imagery for crop-tracking in precision agriculture. Int. J. Appl. Earth Obs. Geoinf. 2021, 97, 102274. [Google Scholar] [CrossRef]
  38. Jhan, J.; Rau, J. A generalized tool for accurate and efficient image registration of UAV multi-lens multispectral cameras by N-SURF matching. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2021, 14, 6353–6362. [Google Scholar] [CrossRef]
  39. Kerkech, M.; Hafiane, A.; Canals, R. Vine disease detection in UAV multispectral images using optimized image registration and deep learning segmentation approach. Comput. Electron. Agric. 2020, 174, 105446. [Google Scholar] [CrossRef]
  40. Yan, J.; Wang, Z.; Wang, S. Real-time tracking of deformable objects based on combined matching-and-tracking. J. Electron. Imaging 2016, 25, 023019. [Google Scholar] [CrossRef]
  41. Nex, F.; Gerke, M.; Remondino, F.; Przybilla, H.-J.; Bäumker, M.; Zurhorst, A. ISPRS benchmark for multiplatform photogrammetry. In Proceedings of the ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Munich, Germany, 25–27 March 2015. [Google Scholar] [CrossRef]
  42. Ye, H.; Su, K.; Huang, S. Image enhancement method based on bilinear interpolating and wavelet transform. In Proceedings of the 2021 IEEE 5th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), Chongqing, China, 12–14 March 2021. [Google Scholar] [CrossRef]
Figure 1. The flow chart of the proposed fast registration algorithm for UAV (Unmanned Aerial Vehicle) images.
Figure 1. The flow chart of the proposed fast registration algorithm for UAV (Unmanned Aerial Vehicle) images.
Sensors 23 08566 g001
Figure 2. A graying instance of UAV images: (a) a visible image; (b) the grayed image.
Figure 2. A graying instance of UAV images: (a) a visible image; (b) the grayed image.
Sensors 23 08566 g002
Figure 3. The flow chart of bi-directionally matching UAV images: (a) Strategy 1; (b) Strategy 2.
Figure 3. The flow chart of bi-directionally matching UAV images: (a) Strategy 1; (b) Strategy 2.
Sensors 23 08566 g003
Figure 4. The relationship between the number of feature points and running time or RMSE (Root Mean Square Error) for five scenes: (a) the number of feature points vs. running time; (b) the number of feature points vs. RMSE.
Figure 4. The relationship between the number of feature points and running time or RMSE (Root Mean Square Error) for five scenes: (a) the number of feature points vs. running time; (b) the number of feature points vs. RMSE.
Sensors 23 08566 g004
Figure 5. A UAV image pair from an urban scene and the feature points after the initial screening: (a) the image to be registered; (b) the reference image; (c) the image pair after the rough screening. The corresponding points are connected by red lines.
Figure 5. A UAV image pair from an urban scene and the feature points after the initial screening: (a) the image to be registered; (b) the reference image; (c) the image pair after the rough screening. The corresponding points are connected by red lines.
Sensors 23 08566 g005
Figure 6. The urban image pairs after bi-directional matching according to two strategies: (a) Strategy 1; (b) Strategy 2. The corresponding points are connected by red lines.
Figure 6. The urban image pairs after bi-directional matching according to two strategies: (a) Strategy 1; (b) Strategy 2. The corresponding points are connected by red lines.
Sensors 23 08566 g006
Figure 7. The results of bi-directional matching after eliminating false matches: (a) Strategy 1; (b) Strategy 2. The corresponding points are connected by red lines.
Figure 7. The results of bi-directional matching after eliminating false matches: (a) Strategy 1; (b) Strategy 2. The corresponding points are connected by red lines.
Sensors 23 08566 g007
Figure 8. The stitched image of the urban scene by the proposed registration algorithms: (a) Strategy 1; (b) Strategy 2.
Figure 8. The stitched image of the urban scene by the proposed registration algorithms: (a) Strategy 1; (b) Strategy 2.
Sensors 23 08566 g008
Figure 9. The numbers of matched point pairs in registration processes: (a) the first bi-directional strategy; (b) the second bi-directional strategy.
Figure 9. The numbers of matched point pairs in registration processes: (a) the first bi-directional strategy; (b) the second bi-directional strategy.
Sensors 23 08566 g009
Figure 10. The different image pairs after bi-directional matching according to two strategies: (a) roads; (b) buildings; (c) farmlands; (d) forests. The corresponding points are connected by red lines.
Figure 10. The different image pairs after bi-directional matching according to two strategies: (a) roads; (b) buildings; (c) farmlands; (d) forests. The corresponding points are connected by red lines.
Sensors 23 08566 g010aSensors 23 08566 g010b
Figure 11. The stitched images of four scenes by the proposed registration algorithms. The left one is obtained according to the first bi-directional matching strategy, and the right one is generated by using the second bi-directional matching strategy: (a) roads; (b) buildings; (c) farmlands; (d) forests. Some obvious mismatches are marked by red circles.
Figure 11. The stitched images of four scenes by the proposed registration algorithms. The left one is obtained according to the first bi-directional matching strategy, and the right one is generated by using the second bi-directional matching strategy: (a) roads; (b) buildings; (c) farmlands; (d) forests. Some obvious mismatches are marked by red circles.
Sensors 23 08566 g011aSensors 23 08566 g011b
Table 1. The sources and parameters of UAV images used in the registration experiments.
Table 1. The sources and parameters of UAV images used in the registration experiments.
ScenesSensorsImage
Resolution
PlotsShooting TimeShooting Height (m)Focal Length (mm)
urbanHasselblad H3DII-502044 × 1533Toronto, CanadaJune 201450080
roadCanon
IXUS 220HS
2000 × 1500Brig, SwitzerlandSeptember 20132004
buildingSONY
NEX-7
3000 × 2000Dortmund, GermanyJune 20145016
farmlandParrot Sequoia2404 × 1728Dayi, ChinaSeptember 2017805
forestZENMUSE Z301920 × 1080Wusu Foshan Forest Park, ChinaJune 201915210
Table 2. RMSE comparison of registration accuracy of different algorithms (pixels).
Table 2. RMSE comparison of registration accuracy of different algorithms (pixels).
MethodsSIFTSURFKAZEAKAZEORBOurs
(Strategy 1)
Ours
(Strategy 2)
Scenes
urban1.23221.12421.18461.02061.32820.99261.0750
road1.18051.26701.29451.15461.35301.01241.1757
building1.17581.29181.21731.11631.32730.98281.0471
farmland1.37351.39051.26681.16061.20051.08530.9909
forest1.26921.33441.22871.23271.28711.04901.0776
The best results for different scene are bolded.
Table 3. Time consumption comparison of different registration algorithms (s).
Table 3. Time consumption comparison of different registration algorithms (s).
MethodsSIFTSURFKAZEAKAZEORBOurs
(Strategy 1)
Ours
(Strategy 2)
Scenes
urban125.25464.023128.408101.0532.2523.9224.031
road137.02586.799180.031144.1462.3243.4833.572
building163.348139.725194.028117.0543.4574.9484.992
farmland50.89946.65272.71235.7681.4143.5623.775
forest93.88942.79484.774103.4471.7273.3153.295
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Luo, X.; Wei, Z.; Jin, Y.; Wang, X.; Lin, P.; Wei, X.; Zhou, W. Fast Automatic Registration of UAV Images via Bidirectional Matching. Sensors 2023, 23, 8566. https://0-doi-org.brum.beds.ac.uk/10.3390/s23208566

AMA Style

Luo X, Wei Z, Jin Y, Wang X, Lin P, Wei X, Zhou W. Fast Automatic Registration of UAV Images via Bidirectional Matching. Sensors. 2023; 23(20):8566. https://0-doi-org.brum.beds.ac.uk/10.3390/s23208566

Chicago/Turabian Style

Luo, Xin, Zuqi Wei, Yuwei Jin, Xiao Wang, Peng Lin, Xufeng Wei, and Wenjian Zhou. 2023. "Fast Automatic Registration of UAV Images via Bidirectional Matching" Sensors 23, no. 20: 8566. https://0-doi-org.brum.beds.ac.uk/10.3390/s23208566

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop