An Intelligent System to Sense Textual Cues for Location Assistance in Autonomous Vehicles

Abstract

The current technological world is growing rapidly and each aspect of life is being transformed toward automation for human comfort and reliability. With autonomous vehicle technology, the communication gap between the driver and the traditional vehicle is being reduced through multiple technologies and methods. In this regard, state-of-the-art methods have proposed several approaches for advanced driver assistance systems (ADAS) to meet the requirement of a level-5 autonomous vehicle. Consequently, this work explores the role of textual cues present in the outer environment for finding the desired locations and assisting the driver where to stop. Firstly, the driver inputs the keywords of the desired location to assist the proposed system. Secondly, the system will start sensing the textual cues present in the outer environment through natural language processing techniques. Thirdly, the system keeps matching the similar keywords input by the driver and the outer environment using similarity learning. Whenever the system finds a location having any similar keyword in the outer environment, the system informs the driver, slows down, and applies the brake to stop. The experimental results on four benchmark datasets show the efficiency and accuracy of the proposed system for finding the desired locations by sensing textual cues in autonomous vehicles.

1. Introduction

Autonomous vehicles (AV) have gained significant popularity in recent years due to the vast revolution in modern transportation systems. An autonomous vehicle is a self-driving vehicle that is efficient at perceiving its outer environment and moving without or with very limited human involvement. The various renowned reports and surveys predict that by 2030, autonomous vehicles will be capable and reliable enough to replace maximum human driving [1,2]. In this scenario, many new methods are being proposed to facilitate autonomous vehicles’ vision perception, sensing the outer environment, safety aspects, traffic laws and regulations, accident liability, and maintaining the surrounding map [3,4,5].

An autonomous vehicle can rely on multiple sensors, complex algorithms, actuators, machine learning tools, computer vision techniques, and reliable processors to take effect [6,7]. The autonomous vehicle perceives the outer environment with the help of numerous sensors and makes the decision by perceiving with the assistance of computer vision [8,9]. Each sensor’s configuration and mechanism varies, as an example in [10], the sideslip angle estimation algorithm for autonomous vehicles is proposed. The algorithm is based on a consensus Kalman filter that fuses measurements from a reduced inertial navigation system (R-INS), a global navigation satellite system (GNSS), and a linear vehicle-dynamic-based sideslip estimator.

Since the last few decades, advanced driver assistance systems (ADAS) are equally appreciated to avoid traffic accidents and to improve driving comfort in autonomous vehicles [11]. The ADAS systems are safe and secure systems designed to decrease the human error rate. These systems assist the driver through the advanced technologies to drive safely and thus, improve the driving performance. Several state-of-the-art methods have employed the inertial measurement unit (IMU) and a global navigation satellite system (GNSS) for vehicle localization. In [12], the authors have proposed a method for estimating the sideslip angle and attitude of an automated vehicle using IMU and GNSS. The method is designed to be robust against the effects of vehicle parameters, road friction, and low-sample-rate GNSS measurements. In [13], the method is proposed for estimating the yaw misalignment of an IMU mounted on a vehicle.

The ADAS systems utilize a combination of multiple sensors to perceive the outer environment and then either offers useful information to the driver or take some necessary actions such as applying the brake, changing the lane, turning left/right, etc. These systems are very helpful to decrease traffic congestion and smoothing traffic movement [14,15]. In the last three decades, multiple features of ADAS systems have been proposed, including cruise control, antilock braking system, auto-parking, power steering, lane centering, collision warnings, and others [16,17,18]. In autonomous vehicles, cameras are generally used as vision sensors. The vision-based ADAS utilizes multiple cameras to capture the images, analyze them, and take the appropriate actions whenever needed.

In state-of-the-art methods, multiple advanced features of the ADAS system are proposed. In [19], Liu et al. proposed a framework using SVM-based trail detection to achieve trail directions and tracking in a real-time environment. The vision-based framework is capable to detect and track the trails as well as scene understanding using a quadrotor UAV operator. Yang et al. [20] proposed two frameworks that show how the CNNs perceive and process the driving scenes with distinguishing visual regions. Gao et al. [21] proposed a method for a 3D surround view for ADAS that covers automobiles around the vehicle. The method helps the driver to be aware of the outer environment.

Liu et al. [22] presented a novel algorithm for detecting tassels in maize using UAV-based RGB imagery. The algorithm named YOLOv5-tassel, is based on the YOLOv5 object detection framework and incorporates several modifications to improve its performance on tassel detection. The authors included the modifications of a bidirectional feature pyramid network to effectively fuse cross-scale features, introduced a robust attention module to extract the features of interest before each detection head, and added an additional detection head to improve small-size tassel detection. Xia et al. [23] proposed a novel data acquisition and analytics platform for vehicle trajectory extraction, reconstruction, and evaluation based on connected automated vehicle (CAV) cooperative perception. The platform is designed to be holistic and capable of processing sensor data from multiple CAVs.

Wang et al. [24] proposed a lane-changing model for making decisions to either change the lane or produce trajectories. The model analyzes the vehicle kinematics of different states, their distances, and comfort level requirements. Chen et al. [25] proposed an instructor-like assistance system in order to avoid collision risk. The driver and the assistance system both assure the recommendation to control the vehicle. Gilbert et al. [26] proposed an efficient decision-making model which selects the least possible collision for AV. The model combines vehicle dynamics and maneuver trajectory paths to produce simulation results and multi-attribute decision-making techniques. Gao et al. [27] proposed a new vehicle localization system that is based on vehicle chassis sensors and considers vehicle lateral velocity. The system is designed to improve the accuracy of vehicle stand-alone localization in highly dynamic driving conditions during GNSS outages.

Xia et al. [10] presented a new algorithm for estimating the sideslip angle of an autonomous vehicle. The algorithm uses consensus and vehicle kinematics/dynamics synthesis to enhance the accuracy of the estimation under normal driving conditions. The proposed algorithm uses a velocity-based Kalman filter to estimate the errors of the reduced inertial navigation system (R-INS) and a consensus Kalman information filter to estimate the heading error. The consensus framework combines a novel heading error measurement from a linear vehicle-dynamic-based sideslip estimator with the heading error from the global navigation satellite system (GNSS) course. Liu et al. [28] proposed a novel kinematic-model-based vehicle slip angle (VSA) estimation method that fuses information from a GNSS and an IMU. The method is designed to be robust against the effects of vehicle roll and pitch, a low sampling rate of GNSS, and GNSS signal delay.

Since the early days of mechanical vehicles, safety has been one of the key concerns in automotive systems. Several attempts have been made to address safety concerns by developing safe and secure systems to protect the driver as well as prevent injuring pedestrians [29,30]. It is one of the safety aspects of an autonomous vehicle when the driver is preoccupied with searching for the desired location to stop. With our proposed system, the safety of AV can be increased drastically since the AV will automatically realize the desired locations in its surrounding. Rather than continually searching for the desired locations, our proposed system will automatically realize the textual cues present in the outer environment and suggest the driver to stop.

While driving on the road, AV performs multiple operations such as lane change, lane keeping, overtaking, and following the traffic rules. Several studies proposed and developed numerous methods for ADAS systems [31,32]. It is equally important for an autonomous vehicle to be aware of the textual cues appearing in its outer environment to take some decision or assist the driver, either to stop or drive. Thus, the key concern of this paper is to reduce human intervention in an autonomous vehicles.

In this paper, we propose a novel intelligent system based on the driver’s instruction for finding the desired locations using textual cues present in the outer environment for advanced driver assistance systems. For this, we combine computer vision and natural language processing (NLP) techniques to perceive textual cues. Computer vision methods train the system to interpret and perceive the visual world around the autonomous vehicle and NLP techniques emphasize the system with the ability to read, recognize, and derive the meaning from textual cues appearing in front of an autonomous vehicle. The key contributions of this paper are as follows:

• A novel intelligent system is proposed for AVs to find unsupervised locations.
• The proposed system is capable of sensing the textual cues that appear in the outer environment for determining desired locations.
• The proposed system is a novel development in the list of ADAS features of an autonomous vehicle.
• With the proposed system, the driver’s efforts for finding the desired locations will drastically be decreased.

The remainder of the paper is organized as follows. In Section 2, we describe the proposed system for finding the desired locations with textual cues and their formation as keywords. In Section 3, the experimental results are defined to show the efficiency and accuracy of the proposed system. Section 4 concludes the proposed work and presents the future directions.

2. Proposed System

In this section, we propose a novel intelligent system to find the desired locations using textual cues for an autonomous vehicle. Firstly, the driver inputs one or more keywords to the proposed system to find the desired locations. Secondly, the proposed system detects and localizes the textual cues appearing in the outer environment. The system will generate the keywords localized from the outer environment with detection and recognition methods. Finally, the system will execute similarity learning to find the similarity between the input keywords and the localized keywords from outer environment images. The schematic diagram of the proposed intelligent system is shown in Figure 1.

2.1. Textual Cues Detection

In order to detect, localize, and form the keywords from the outer environment, we employ text detection and localization technique. Firstly, we use affine transformation to deal with global distortion appearing within an input image and to improve the accuracy of the text to a more horizontal text. It takes an input image $I_i\in {ℝ}^{C_i\times H_i\times W_i}$ with channel $C_i$, height $H_i$, and width $W_i$ to produce an output image $I_g$. The affine transformation based on the arguments between the input image $I_i$ and output image $I_g$ is given as:

$$\left[\begin{array}{l}{x}_i\\ y_i\end{array}\right]=T_{\theta }\left[\begin{array}{l}{x}_g\\ y_g\\ 1\end{array}\right]$$

(1)

where $(x_i,y_i)$ are the source coordinates of the input image and $(x_g,y_g)$ are the required coordinates for the output image $I_g$. The output image $I_g$ is further rectified from the input image $I_i$ using bilinear interpolation, given as:

$$I_g(x_g,y_g)={\displaystyle \sum _n^{H_i}{\displaystyle \sum _m^{W_i}{I}_{i(n,m)}}}\max \left(0,1-\left|x_i-n\right|\right)\max \left(0,1-\left|y_i-m\right|\right)$$

(2)

where $I_g(x_g,y_g)$ is the pixel value of the rectified image $I_g$ at the location $(x_g,y_g)$ and $I_{i(n,m)}$ is the pixel value of the input image $I_i$ at the location $(n,m)$.

2.1.1. Textual Candidates Detection

The textual candidates detection aims to extract the position of textual regions in the outer environment. Since the text appearing in the outer environment generally has diverse contrast to its relative background and uniform color intensity, the maximally stable extremal region (MSER) technique is the best approach as it is widely used and considered the best region detector [33]. In order to detect the textual candidates appearing in the outer environment, we adopt the MSER approach for finding the corresponding candidates within the input image $I_g(x_g,y_g)$. For finding the extremal regions in the input image, the intensity difference is given as:

$$\nu (R_x)=\frac{\left|R(+\Delta )-R\right|}{\left|R\right|}$$

(3)

where $\left|R\right|$ represents the extracted extremal regions area, $R(+\Delta )$ represents the extremal regions, $+\Delta$ specifies the increment of each extremal region $R$, and $\left|R(+\Delta )-R\right|$ shows the area difference between the two regions’ area. After applying the region detector, the obtained extremal regions are shown in Figure 2.

2.1.2. Textual Candidates Filtering

The textual regions detected in the previous step using the MSER technique are further refined and rectified. First, we validate the size and the aspect ratio using geometric properties for textual candidates filtering, which is given as:

$$h_{\min }\le h\le h_{\max }$$

$$r_{\min }\le h/w\le r_{\max }$$

where h and w are the height and width of the aligned bounding box of segmented axes, respectively, and $h_{\min }$, $h_{\max }$, $r_{\min }$ and $r_{\max }$ are components to finetune.

The input image $(x_d,y_d)\in {\ell }_d$ having the size $W\times H$ and the predicted categorized result $\alpha$ and ${\alpha }^{\prime }$ with uncertain probability sequence $p$ and $p^{\prime }$ is given as:

$${\Theta }_t(x_d)=(\alpha ,p)$$

(4)

$${\Theta }_s(x_d)=({\alpha }^{\prime },p^{\prime })$$

(5)

where

$$\alpha =({\alpha }_1,{\alpha }_2,{\alpha }_3,\dots ,{\alpha }_D)$$

(6)

$$p=(p_1,p_2,p_3,\dots ,p_D)$$

(7)

$${\alpha }^{\prime }=({{\alpha }^{\prime }}_1,{{\alpha }^{\prime }}_2,{{\alpha }^{\prime }}_3,\dots ,{{\alpha }^{\prime }}_K)$$

(8)

$$p^{\prime }=({p^{\prime }}_1,{p^{\prime }}_2,{p^{\prime }}_3,\dots ,{p^{\prime }}_K)$$

(9)

where D and K represent the character sequence length.

The input vector $V$ is combined using the following properties:

$$\min (p_1,p_2,p_3,\dots ,p_D)$$

$$\min ({p^{\prime }}_1,{p^{\prime }}_2,{p^{\prime }}_3,\dots ,{p^{\prime }}_K)$$

$$\mathrm{mean}(p_1,p_2,p_3,\dots ,p_D)$$

$$\mathrm{mean}({p^{\prime }}_1,{p^{\prime }}_2,{p^{\prime }}_3,\dots ,{p^{\prime }}_K)$$

The above four properties are probability characteristics in which mean represents the overall confidence score and min represents the least likely character.

Furthermore,

$$D^{\prime }=\frac{D}{\theta }$$

(10)

$$K^{\prime }=\frac{K}{\theta }$$

(11)

where $\theta$ is the constant parameter. The above two properties are used to normalize the number of characters between 0 and 1, and the following two properties are used for character width calculated as per geometric properties:

$$\kappa =\frac{W}{D}$$

(12)

$${\kappa }^{\prime }=\frac{W}{K}$$

(13)

where $\kappa$ represents the character width.

The localized regions which satisfy the above properties are then further processed and the remaining regions are discarded, as shown in Figure 3. The obtained localized regions consist of non-textual regions and may produce a false result for recognition. We further segment textual regions with the stroke responses of each image pixel. The corner points are used as the edges of two strokes. The corner points and stroke points establish the distortion of strokes. For this, we follow the corner detection approach [34], which applies the following selection criteria. Firstly, the matrix M for each pixel is calculated as follows:

$$M={\displaystyle \sum _{(x,y)}w(x,y)}\left[\begin{array}{cc}{I}_x^2& I_x{I}_y\\ I_x{I}_y& I_y^2\end{array}\right]$$

(14)

where $w(x,y)$ represents the weight at position $(x,y)$ for window center, $I_x$ and $I_y$ denotes the gradient value of pixel at position $(x,y)$. The eigenvalues ${\lambda }_1$ and ${\lambda }_2$ of $M$ matrix are calculated as:

$$R=\min ({\lambda }_1,{\lambda }_2)$$

(15)

To compute the turning point of outer stroke endpoints, we use the following equation [35]:

$$x^{\ast },y^{\ast }=\underset{x,y}{\arg \max }\left(\frac{(X_2-X_1)\times y+(Y_2-Y_1)\times x+X_2{Y}_1-X_1{Y}_2}{{(X_2-X_1)}^2+{(Y_2-Y_1)}^2}\right)$$

(16)

where $X_1$, $X_2$, $Y_1$, and $Y_2$ are the coordinates of the endpoints of the strokes; $x^{\ast }$ and $y^{\ast }$ are coordinates of the outermost points; and $x,y$ denotes coordinates of every single point at the curve. The following equation is given to determine outer stroke points:

$$P(x_{out},y_{out})=2P(x^{\ast },y^{\ast })-\frac{1}{2}(P(X_1,Y_1)+P(X_2,Y_2))$$

(17)

where $P(x,y)$ denotes a single point at the x-axis and y-axis in word image.

Given the corner point $P(X_c,Y_c)$ along with its adjacent corner $P(X_n,Y_n)$, the height $h_{area}$ and width $w_{area}$ of a moving window is determined as:

$$h_{area}=2\alpha \times \left|Y_c-Y_n\right|$$

(18)

$$w_{area}=2\alpha \times \left|X_c-X_n\right|$$

(19)

where $\alpha$ is a coefficient to normalize the area of the moving region among the corner points and is set between 0 and 1. Moreover, the moving area of outer strokes $P(X_o,Y_o)$ for the side length area $s_{area}$ is given as:

$$s_{area}=\beta \times \left(\frac{(X_2-X_1)\times Y_o+(Y_2-Y_1)\times X_o+X_2{Y}_1-X_1{Y}_2}{{(X_2-X_1)}^2+{(Y_2-Y_1)}^2}\right)$$

(20)

where $\beta$ is a coefficient to normalize the moving regions among the outer strokes and is set between 0 and 1. The final filtered localized textual regions are shown in Figure 4.

2.1.3. Keywords Grouping and Recognition

The localized textual regions in the previous steps consist of individual text characters. In order to recognize and understand the meaning of these textual regions, these individual characters must be combined into text lines. This way, the localized textual regions may represent more meaningful information about the outer environment as compared to the individual characters. For example, the localized textual region consists of the “SCHOOL” versus the individual character set {C,O,L,O,S,H} where its meaning is lost due to the unordered sequence of the word [36,37,38].

In order to form the ordered keywords, we employ the grouping approach [39]. The key idea is to apply a rectangle $\omega s_p\times h{s}_p$ for each connected region having the center $(x_p,y_p)$ and orientation ${\theta }_p$. Each associated region is considered to be a keyword candidate. The initial candidate regions having $w_1=0.4$ are refined to be the keyword with the following properties:

(1)

The two adjacent textual candidates are associated with a new $w_i$ value.

(2)

The achieved keyword candidate which is the combination of two candidates is obtained with curvilinear.

If the centers of connected regions in $\tau$ are estimated normally with a kth order polynomial, then the candidate keyword $\tau (\subset \zeta )$ is determined as curvilinear:

$$\sqrt{\underset{f}{\min }{\displaystyle \sum _{c_p\in \tau }\frac{1}{\left|\tau \right|}{\left(({y^{\prime }}_p-f({x^{\prime }}_p)\right)}^2}}\le \frac{\overline{s}}{4}$$

(21)

where $({x^{\prime }}_p,{y^{\prime }}_p)$ is the rotated point of $(x_p,y_p)$ and $\overline{s}=\frac{1}{\tau }\sum s_p$ is the average score. The bounding boxes are applied to the character set of textual regions, as shown in Figure 5.

The grouped keywords from localized textual regions are further processed for recognition purposes. The cropped word images $I\in {ℝ}^{W\times H}$ having width $W$ and height $H$ consisting of the textual cues are recognized individually. The inputs are the 2D maps resulting in a $W\times H$ map for the individual character supposition.

Given the metrics $X,Y,Z$ and the confidence score for individual word supposition $w=(c_1,c_2,c_3,\dots ,c_L{}_{{}_w})$, let $b^w=(b_1^w,b_2^w,b_3^w,\dots ,b_{L_w+1}^w)$ represent the breakpoints amid individual characters, where $b_1^w$ initializes the first character and $b_{L_w}^w$ ends the last character. The breakpoint hypothesis $(w,b^w)$ for the word confidence score is given as:

$$s(w,b^w,X,Y,Z)=\frac{1}{\left|b^w\right|}\left({\displaystyle \sum _{i=1}^{\left|b^w\right|}{m}_i(b_i^w,R)+{\displaystyle \sum _{i=2}^{\left|b^w\right|}\varphi (b_i^w,b_{i-1}^w,X,Y,Z)}}\right)$$

(22)

Each individual hypothesis word $w$ is optimized for breakpoints, and a word having an optimal score is recognized as:

$$s(w,I)={\max }_{b^w}s(w,b^w,X,Y,Z)$$

(23)

The unary fraction scores given in Equation (22) are determined with the following properties: the distance from outside the image boundaries, the distance from the estimated breakpoint location, the binary fraction score, the non-text class score, and the distance of the first and last breakpoints from the edge of the image. The pairwise score given in Equation (23) is determined with the following properties: non-text scores at character centers, character scores at midpoints amid breakpoints, eccentricity from the normalized character width, and active contributions of the left and right binary responses comparative to character scores.

The bounding boxes are applied to recognized words in order to match the evaluated breakpoints, and the recognized bounding boxes are added to the queue of recognized words. The recognized keywords are shown in Figure 6.

2.2. Textual Cues Keywords

The localized textual regions are optimized with the OCR and the formal words are recognized, thus providing a sensible meaning. In this step, we utilize the recognized formal words to establish a words model that will be responsible for sequences and boundaries. Since the recognized textual cues may still be missing some characters and may affect finding the desired locations, we employ an n-gram probabilistic language model that will provide evidence for the presence of the actual cues [40].

An n-gram model is generally used to predict the probability of a given n-gram in any contiguous sequence of words. A better n-gram model predicts the next word in a sentence. For example, given the word ‘park’, the first recognized trigram is ‘par’ and the second recognized trigram is ‘ark’, and then its overlapping characters ‘ar’ suggests that the correctly recognized word is likely to be ‘park’.

Given the word $w$ of length $N$ as a sequence of characters $w=(c_1,c_2,c_3,\dots ,c_N)$ where each $c_i\in C=\{1,2,3,\dots ,36\}$ denotes a character at $i$ position in word $w$ from 26 letters and 10 digits, each recognized word has a varying length $N$ that can be determined at the run time. Therefore, the number of characters in a single word is fixed to 22 with a null character and a maximum length class, which is given as:

$$c_i^{\ast }=\arg {\max }_{c_i\in C\cup \left\{\varphi \right\}}P(c_i|\vartheta (x))\cdot P(c_i|\vartheta (x))$$

(24)

For two strings and $w\in C^M$, the $s\subset w$ represents $s$ as a substring of the word $w$. An $N$-gram of $w$ is assumed as substring $s\subset w$ having the length $\left|w\right|=N$. The dictionary of all grams of word $w$ of length $N$ is given as:

$$G_N(w)=\{s:s\subset w\wedge \left|s\right|\le N\}$$

(25)

$$G_N={\cup }_{w\in \omega }{G}_{N(w)}$$

(26)

As an example, the dictionary for the word ‘cafe’ is $G_3(\mathrm{cafe})=\left\{\mathrm{c},\mathrm{a},\mathrm{f},\mathrm{e},\mathrm{ca},\mathrm{af},\mathrm{fe},\mathrm{caf},\mathrm{afe}\right\}$. Given the recognized ith n-gram $w_{n,i}$ and its consistent confidence score $c_{n,i}$, in order to determine the sequence of n-grams with the most confident prediction for the entire sequence of recognized words, the objective function can be given as:

$$C_{n,i}=\{\begin{array}{l}{c}_{n,i},\mathrm{if}n=1\\ \min (c_{n,i},W_{n,i}),\mathrm{otherwise}\end{array}$$

(27)

where

$$W_{n,i}={\min }_{m\in [1,n-1]}\left\{\frac{(n-m)\cdot C_{n-m,i}+m\cdot C_{m,n+i-m}}{n}\right\}$$

(28)

Here, $W_{n,i}$ is used to achieve the optimal n-gram separation of the given word, and each n-gram word image is recursively recognized.

2.3. Similarity Learning

Similarity learning finds and matches similar images as the user-input keywords [41,42,43]. The proposed intelligent system matches the user-input keywords with the outer environment textual cues. For this, we create a feature vector of user input keywords and the recognized textual cues from the outer environment images.

Given the input keywords $Q$, the word $q_i$ is treated as a sequence of characters $(y_1,y_2,y_3,\dots ,y_{\left|q_i\right|})$, where $\left|q_i\right|$ denotes the total number of characters in word $q_i$, and $y_j$ is considered as the optimal representation of the $j^{th}$ character of word $q_i$. Each sequence is interpolated and concatenated with a fixed-length feature ${\widehat{f}}_i\in {ℝ}^{T\times 2C}$ and all the features ${\left\{{\widehat{f}}_i\right\}}_{i=1}^N$ are signified as output features $\widehat{F}\in {ℝ}^{N\times T\times 2C}$.

The recognized textual cue proposals $E\in {ℝ}^{K\times T\times C}$ and the input keywords $F\in {ℝ}^{N\times T\times C}$ are formed, and the similarity is computed as a similarity matrix $\widehat{S}(Q,P)\in {ℝ}^{N\times K}$ between the input keywords $Q$ and recognized textual cues. The score ${\widehat{S}}_{i,j}(Q,P)$ between both the feature vectors $F_i$ and $E_j$ is given as:

$${\widehat{S}}_{i,j}(Q,P)=\frac{\tanh (V(F_i))\tanh {(V(E_j))}^T}{\Vert \tanh (V(F_i))\Vert \times \Vert \tanh (V(E_j))\Vert }$$

(29)

where $V$ represents the operator that converts the 2D matrix into a 1D vector. The required similarity matrix $\widehat{S}(Q,P)$ is maintained by the target similarity matrix $S(Q,P)$. The target similarity ${\widehat{S}}_{i,j}(Q,P)$ is computed as the Levenshtein distance between corresponding textual pairs $(q_i,q_j)$ and is given as:

$$S_{i,j}(Q,P)=1-\frac{Distance(q_i,q_j)}{\max (\left|q_i\right|,\left|q_j\right|)}$$

(30)

Meanwhile, during implementation for the ranking, the similarity between the input keywords and recognized textual words equals to the maximum value of ${\widehat{S}}_{i,j}(Q,P)$.

3. Experimental Results and Discussion

This section presents evaluation results to show the efficiency and accuracy of the proposed intelligent system. The evaluation protocols and benchmark datasets used for experimentation are given as follows.

3.1. Datasets

As the proposed intelligent system is capable to find and locate the desired locations using textual cues, we evaluated our system on four different benchmark datasets comprising the outer environment images. In these datasets, different textual cues can be found and located for the autonomous vehicle’s driving assistance system.

Street View Text (SVT): The dataset [44] consists of outer environment images and certain textual cues appear on the different objects such as walls, shops, billboards, buildings, etc. This dataset contains 250 trained images and 100 test images, and each image has a varying dimension from 1024 × 768 to 1920 × 906.

ICDAR 2013: The dataset [45] comprising the outer environment images and textual cues can be found on multiple different objects such as shops, cafes, signboards, banners, posters, etc. This dataset contains 229 trained images and 233 test images. The dimension of each image varies from 3888 × 2592 to 350 × 200.

Total-Text: The dataset [46] contains curved, orientated, and horizontal textual cues that are very challenging to detect and recognize. The text appears on multiple objects present in the outer environment. The dataset contains 300 test images and 1255 trained images and each image’s dimension varies from 180 × 240 to 5184 × 3456.

MSRA-TD500: The dataset [47] contains challenging outer and inner environment images in which the textual cues appear on doorplates, caution plates, signs, boards, etc. This dataset contains 300 trained images and 200 test images. The dimension of each image varies from 1920 × 1280 to 1296 × 864.

3.2. Evaluation Measures

The evaluation measures for the proposed intelligent system are given as follows.

3.2.1. Textual Cues Evaluation

The detection and localization of the textual cues is one of the main entities for a robust intelligent system. We evaluate the textual cues’ detection and localization with standard evaluation protocols [48]—precision p, recall r, and frequency f measures defined as:

$$p=\frac{{\Sigma }_{r_e\in E}m(r_e,T)}{\left|E\right|}$$

(31)

$$r=\frac{{\Sigma }_{r_t\in {\rm T}}m(r_t,E)}{\left|T\right|}$$

(32)

$$f=\frac{1}{\frac{\alpha }{p}+\frac{(1-\alpha )}{r}}$$

(33)

3.2.2. Location Retrieval Evaluation

The user-input keywords are matched with the recognized textual cues and similar location images are retrieved for the necessary actions such as applying the brake. For location retrieval, the mean average precision mAP is a commonly used evaluation measure which is the average of all queries. The mean average precision can be given as follows:

$$P(R_k)=\frac{\#(A\cap B)}{\#(B)}$$

(34)

where A denotes the number of relevant locations, B is the number of retrieved locations, $R_k$ denotes the top similar images consisting of the same textual cues as the user-specified keywords. Given the set of user keywords $q_i\in Q$ as $\{w_1,w_2,w_3,\dots ,w_m\}$, where Q denotes the set of all the keywords specified by the user, the mAP for the proposed system is formulated as:

$$mAP(Q)=\frac{1}{Q}{\displaystyle \sum _{i=1}^{\left|Q\right|}\frac{1}{m}}{\displaystyle \sum _{k=1}^{m}P(R_k)}$$

(35)

where k is the number of retrieved location images having the most similar textual cues.

3.3. Implementation Results

In this section, we briefly describe the implementation details and discuss the output results. Firstly, the proposed intelligent system asks the driver to input the keywords of the desired location. For this purpose, we randomly select the different keywords as input from the test images of all the datasets defined in Section 3.1. Secondly, the proposed system detects and localizes the textual cues from the trained images of each dataset and applies OCR recognition. Thirdly, similarity learning is applied to compute the similarity between the input keywords and recognized textual cues. Lastly, whenever any similar textual cue is found, the intelligent system informs the driver, slows down, and applies the brake to stop. The detailed experiments are given as follows.

3.3.1. Textual Cues Detection

Since textual cues detection is one of the key phases for a robust intelligent system, we first evaluate the textual cues detection on the benchmark datasets.

In this experiment, we evaluate the efficiency and accuracy of the proposed intelligent system for detecting and localizing the textual cues on the different datasets. We use the textual evaluation protocols: precision p, recall r, and frequency f. The obtained results of the proposed system are given and compared with the state-of-the-art methods in

Table 1. Textual cues detection and localization result on SVT dataset.

Method	Precision	Recall	f
Unar’18 et al. [49]	54.0	51.0	52.0
Wei et al. [50]	18.2	41.2	25.2
Unar’19 et al. [51]	47.0	42.0	44.0
Yu et al. [52]	27.0	35.0	30.0
Our method	61.52	58.31	59.87

,

Table 2. Textual cues detection and localization result on ICDAR’13.

Method	Precision	Recall	f
Wei et al. [50]	83.5	77.2	80.2
Zhong et al. [53]	90.8	86.1	88.4
Unar’18 et al. [49]	81.0	79.0	79.0
Lyu et al. [54]	92.0	84.4	88.0
Neumann et al. [36]	82.0	71.0	76.0
Unar’19 et al. [51]	83.0	82.0	82.0
Hou et al. [55]	89.6	81.3	85.3
Our method	92.4	87.63	89.95

,

Table 3. Textual cues detection and localization result on Total-Text.

Method	Precision	Recall	f
Wang et al. [56]	76.2	80.9	78.5
Yuxin et al. [57]	83.9	86.9	85.4
Wang et al. [58]	75.1	81.8	78.3
Liao et al. [59]	82.5	87.1	84.7
Zhang et al. [60]	75.7	88.6	81.6
Our method	84.2	86.9	85.5

and

Table 4. Textual cues detection and localization result on MSRA-TD500.

Method	Precision	Recall	f
Wang et al. [56]	82.1	85.2	83.6
Zhong et al. [53]	85.5	81.3	83.3
Lyu et al. [54]	87.6	76.2	81.5
Hou et al. [55]	81.8	78.7	80.2
Shi et al. [61]	86.0	70.0	77.0
Our method	86.2	86.71	86.45

for SVT, ICDAR’13, Total-Text, and MSRA-TD500 datasets, respectively.

The proposed method outperformed state-of-the-art methods for the SVT and ICDAR datasets in textual cues detection and localization. For the Total-Text dataset, our proposed method achieved better precision as compared to state-of-the-art methods. For the MSRA-TD500 dataset, the proposed method achieved remarkable results with a better f score. Our main target is to detect the textual cues from the low-quality road and street images such as the SVT dataset that are really challenging to detect.

3.3.2. Locations Retrieval

This section describes the experimentations on the datasets for finding the desired locations. In order to show the better performance and accuracy of the proposed intelligent system, we conducted the experimentation on one-to-one and one-to-many location frames. The details of the experiments are given as follows:

Experiment 1.

One-to-one: In this experiment, the proposed intelligent system first asks the driver to input the keywords of desired locations to trace and proceed. Once the driver inputs one or more keywords, the proposed system creates a feature vector of those keywords and finds a similar location image with the same textual cues. In this experiment, the proposed system keeps finding similar textual cues and asks the driver to confirm. The feature vector for input keywords continually matches with each image and produces the score. The retrieval time for this experiment is much faster than Experiment 2 as it rapidly compares the similarity and presents the outcomes to the driver. The obtained results of Experiment 1 are given in

Table 5. One-to-one location retrieval evaluation on different datasets.

Datasets	mAP
SVT	69.2
ICDAR’13	74.8
Total-Text	59.1
MSRA-TD500	63.4

for SVT, ICDAR’13, Total-Text, and MSRA-TD500 datasets.

Experiment 2.

One-to-many: In order to show the efficiency and accuracy of the proposed intelligent system, we perform the one-to-many experiment. In this experiment, the proposed intelligent system takes input keywords from the driver for the desired location and traces the top-rank location images possessing similar textual cues. The actual purpose of this experiment is to show the robustness of the proposed system by retrieving the top ten similar images with similar textual cues. The obtained results of Experiment 2 are compared with the state-of-the-art methods in

Table 6. One-to-many location retrieval evaluation on different datasets.

Datasets	SVT	ICDAR’13	Total-Text	MSRA-TD500
Unar’18 et al. [49]	63.0	71.0	-	-
Mishra et al. [62]	56.0	65.0	-	-
Neumann et al. [63]	23.0	-	-	-
Unar’19 et al. [51]	59.0	74.0	-	-
Our method	66.8	75.6	57.4	61.7

for SVT, ICDAR’13, Total-Text, and MSRA-TD500 datasets.

It is worth mentioning that the proposed method outperforms the majority of the previous methods in the one-to-many experiment. However, for the ICDAR’13 dataset, our proposed method could not compete for the ICDAR’13 dataset with the method [51] but still presented a remarkable performance with other datasets.

3.3.3. Retrieval Time Comparison

The time consumed during the computation of textual cues and finding the similarity is truly a critical parameter to be considered. The proposed intelligent system finds the textual features and discards the non-textual features during the localization step. The system maintains a good balance between textual and non-textual features. The retrieval time of the proposed system is compared for both experiments in

Table 7. Locations’ retrieval time (s) comparison.

Datasets	One-to-One	One-to-Many
SVT	0.433	2.644
ICDAR’13	0.391	2.309
Total-Text	0.534	3.215
MSRA-TD500	0.482	2.962

. For the one-to-one experiment, the retrieval time is better and more robust since the similarity is compared between the two entities, i.e., driver-input keywords and targeted outer environment location image. For the one-to-many experiment, the retrieval time is higher since the similarity is computed to index top-rank location images.

3.4. Results Impact and Discussion

In this section, we discuss and compare the results of the proposed approach with state-of-the-art methods. The details are given as follows.

Textual cues: Since the textual cue detection plays a vital role in order to find the semantic locations, we first compared the textual cue detection rate for four different datasets. The output results are given on the SVT dataset for textual cues detection and localization in Table 1. The proposed approach outperformed the other methods with respect to p, r, and f values. The method achieved 61.52 precision, 58.31 recall, and 59.87 f values. Here, the precision and recall values of the method [52] are the least inferior, having 27.0 precision, 35.0 recall, and 30.0 f values. For the ICDAR’13 dataset, the proposed method achieved 92.4 precision, 87.63 recall, and 89.95 f values. The proposed method also outperformed state-of-the-art methods for the ICDAR’13 dataset. The close competent method [54] has 92.0 precision, 84.4 recall, and 88.0 f values.

For the Total-Text dataset, our proposed method achieved 84.2 precision, 86.9 recall, and 85.5 f values. The proposed method outperformed most of the existing methods for precision and f values. However, the method in [60] has a slightly higher recall value, i.e., 88.6 and the method in [59] has an 87.1 recall value. For the MSRA-TD500 dataset, the proposed method achieved 86.2 precision, 86.71 recall, and 86.45 f values. Here, the method in [54] has the highest precision value of 87.6. However, the proposed method outperformed state-of-the-art methods for the highest recall and f score.

Locations retrieval: For retrieving the semantic locations, we have evaluated the proposed method for two different experiments. For the one-to-one mode, the proposed method achieved 69.2 mAP on SVT, 74.8 mAP on ICDAR’13, 59.1 mAP on Total-text, and 63.4 mAP on MSRA-TD500 datasets. The mAP is inferior due to the textual cue localization step and can be improved if the textual cues are further improved. For the one-to-many mode, the proposed method has achieved 66.8 mAP on SVT, 75.6 mAP on ICDAR, 57.4 mAP on Total-Text, and 61.7 mAP on MSRA datasets. Here, the close competent method [49] has 63.0 mAP on SVT and 71.0 mAP on ICDAR datasets. The method in [63] has the most inferior mAP 23.0 on SVT and the method in [62] has 65.0 mAP on ICDAR datasets. The proposed method has achieved an overall better mAP for all the datasets and outperformed state-of-the-art methods.

The proposed method has certainly a few limitations. Due to its simple feature description, it is robust and able to handle only a small number of data rather than millions of images. Another disadvantage is that our method might not work well with more complicated and complex images since it might not be able to generalize to the new data type. The proposed approach, however, is exempt from both extensive training and any expensive hardware needs. The proposed approach is often easier to adopt and can be used more quickly. As a result, the proposed method performed better than the majority of the current methods and achieved an overall greater mean average precision score. The datasets included images from outer environment scenarios; however, the system’s performance can be further improved by more complicated conditions such as low light, occlusion, or a partial visibility of textual cues.

4. Conclusions

In this work, a novel intelligent system is proposed to sense the textual cues available in the outer environment for finding the locations in autonomous vehicles. The proposed system first asks the driver to input the keywords of the desired locations. Next, the system proceeds with the detection and recognition of certain textual cues appearing on different objects such as billboards, shops, signboards, walls, buildings, banner, etc. Whenever the system finds a location composed of similar keywords to the driver’s input keywords, the system notifies the driver, slows down, and applies the brake to stop. The experimental results on four datasets show the robustness of the proposed intelligent system for autonomous vehicles to sense the textual cues appearing in the outer environment scenario. The proposed system has lesser computation complexity and does not require any specific hardware. The system is free from a tremendous amount of training due to its simple feature description. In the future, we intend to improve the retrieval accuracy of the proposed system. We will further improve the methodology for live tracking and perform the experimentations on real-time scenario with video frames.