Multimodal Feature-Assisted Continuous Driver Behavior Analysis and Solving for Edge-Enabled Internet of Connected Vehicles Using Deep Learning

Abstract

The emerging technology of internet of connected vehicles (IoCV) introduced many new solutions for accident prevention and traffic safety by monitoring the behavior of drivers. In addition, monitoring drivers’ behavior to reduce accidents has attracted considerable attention from industry and academic researchers in recent years. However, there are still many issues that have not been addressed due to the lack of feature extraction. To this end, in this paper, we propose the multimodal driver analysis internet of connected vehicles (MODAL-IoCV) approach for analyzing drivers’ behavior using a deep learning method. This approach includes three consecutive phases. In the first phase, the hidden Markov model (HMM) is proposed to predict vehicle motion and lane changes. In the second phase, SqueezeNet is proposed to perform feature extraction from these classes. Lastly, in the final phase, tri-agent-based soft actor critic (TA-SAC) is proposed for recommendation and route planning, in which each driver is precisely handled by an edge node for personalized assistance. Finally, detailed experimental results prove that our proposed MODAL-IoCV method can achieve high performance in terms of latency, accuracy, false alarm rate, and motion prediction error compared to existing works.

1. Introduction

In intelligent transportation systems (ITSs), driver behavior analysis is the most exploited topic in terms of emergency events reduction [1]. If any abnormal behavior occurs on the roadside, the service not only highlights the wrong information to the driver, but also affects the transportation system around the vehicle [2]. For instance, accidents and road risks are caused by high volumes of traffic and the speed of vehicles [3].

Recently, driver behavior analysis based on the continuous monitoring of visual features (driver’s actions) has been used for managing risks in the roadside environment [4,5]. However, determining the driver’s status based on visual facial features is inadequate for driver behavior analysis [6]. In-built vehicle sensors are used to monitor steering wheel movement, and the standard deviation of lane changes and position [7]. Furthermore, physiological signal parameters, such as a trocardiograph (ECG), electroencephalogram (EEG), and electromyogram (EMG), are used to measure health issues [8]. Therefore, driver, vehicle, and roadside parameters are utilized to continuously monitor driver behavior [9,10].

Most of the previous works in this field lack in identifying the various driver and vehicle parameters [11]. Such driver behavior monitoring suffers by not alerting roadside pedestrians and other vehicles that are traveling on the non-safe driver’s path [12]. Some of the input parameters for monitoring driver behavior are gravity, throttle, speed, revolutions per minute, and acceleration. The types of driver behavior are normal, aggressive, distracted, drowsy, and drunk driving [13,14]. This information is measured by sensors that are placed on the vehicle. Each vehicle is connected to the roadside units (RSU) and each vehicle has microphones, frontal cameras, and road cameras (as shown in Figure 1).

In driver behavior monitoring, vehicle motion prediction and lane change detection are often utilized since drivers frequently change lanes with high motion [15]. This causes any risks to avoid nearby vehicles and roadside pedestrians [16]. As a result, vehicle velocity, distance, and the position of the vehicle are observed to predict the sudden change of motion from past sequences [17]. There are two successive procedures that are implemented if any abnormal behaviors are detected: route planning and personalized assistance [18]. Route planning based on drivers’ current preferences was not discussed earlier. This is significant in monitoring and providing assistance to drivers to ensure traffic and driver safety [19]. Hence, both assistance and route planning is provided in this study. Moreover, minimal research has been presented regarding disseminating warning messages to nearby vehicles [20].

1.1. Research Aim and Objectives

The main research aim is to design a safe roadside environment (i.e., ensure driver safety, pedestrian safety and neighboring vehicle safety). This can be achieved with the continuous monitoring of vehicle motion and lane change deviation, and processing other multimodal features is important for accurate driver behavior analysis. Another aim of the study was to reduce the overhead of the whole network by introducing a multiaccess edge environment—an end-to-end architecture for driver behavior analysis and solving roadside risks via personalized assistance and route planning. The common issues that are faced in driver behavior analysis are as follows:

• High overhead: The huge volume of data processing in RSU increases the message overhead; the cloud-based computing environment that is used for warning message generation for driver behavior increases the latency in alert generation; the prevention of road traffic is not possible;
• Lack of optimum recommendation: Most of the existing works have focused on driver behavior analysis/prediction; however, if the behavior was abnormal, solving it was not discussed in the current research papers. Recommendations for solving abnormal behavior and reducing damage (economic loss, accidents to pedestrians and passengers (inside the vehicle), and health issues of driver) are lacking.

1.2. Research Contribution

The contributions of our multimodal feature-based driver behavior analysis method are defined as follows:

• Vehicle motion and lane change behaviors are predicted for improving the performance of feature extraction. For this, HMM is proposed in this paper. After completing vehicle motion and lane change detection, the HMM classifies them into three classes—namely, frequent change, no change, and mild change for improving the performance of the proposed MODAL-IoCV method;
• Then, RSU extracts multiple features from four classes, which are provided by SqueezeNet, which is a lightweight model. It extracts three types of features, including visual features, vehicular features, and smart IoT device features, which increases the speed and accuracy of feature extraction compared to AlexNet;
• We construct a perfect roadside virtual graph by considering vehicle mobility speed, moving direction, and the distance between vehicles for broadcasting the warning messages to the nearby vehicles and roadside pedestrians, which helps to reduce the risk of accidents;
• Then, we provide a recommendation to the drivers by disseminating the message to each driver using edge nodes. This assistance is recommended based on the past and future behavior of the vehicles using the tri-agent-based soft actor critic (TA-SAC) algorithm. Here, three agents are utilized for driver behavior and this provides recommendations such as take a rest and drive defensively based on the current state. This increases the detection accuracy and prevention accuracy of the proposed MODAL-IoCV method.

1.3. Paper Organization

The rest of this paper is organized as follows: Section 2 explains the related work and its limitations, which are related to our work. Section 3 describes the problem statement that existed in the previous work. Section 4 provides a detailed explanation of the proposed methodology of the MODAL-IoCV method, which includes pseudocode, algorithms and mathematical representations. Section 5 explains the results section, which shows how the proposed system achieves better performance compared to the existing work. Section 6 discusses the results of our proposed model. Finally, Section 7 concludes the research work, which also includes the future work of this research.

2. Related Work

In this section, the literature survey regarding the detection and classification of driver behavior in IoCV is presented. The research gap in designing an effective driver behavior analysis is analyzed.

Using driving behavior detection to reduce the occurrence of traffic accidents was proposed in [21]. Driving behavior analysis is conducted based on the serial feature network (SF-NET) and smart phone inertial sensor, which consider the continuity of driving events and utilize multitime data to analyze the driving status. Data are collected from GPS data, acceleration, and gyroscope sensors in smartphones. The input vector not only consists of the current sensor information, but also the relevant information for the adjacent time. In the serial feature network (SF-NET), 10 different behaviors of drivers are identified, which uses multitime and multilevel features information. The SF-NET model must be simplified to reduce the overhead in driver behavior analysis because it is difficult to analyze the multiple features in a single model.

Using vehicle motion prediction for driver behavior monitoring and design was proposed in [22]. Attention span is incorporated in this paper for deriving the relative importance of nearby vehicles with respect to the future moving direction. The weighted sum model is added for fusing the related motion patterns to the surrounding vehicles. By incorporating the historical information of vehicle motion, the future trajectories of vehicles are computed. The vehicle motion is different depending on the number of intersections, vehicle densities, and traffic signals. The consideration of historical information does not reflect higher accuracy in driver behavior detection.

The prediction of driving distraction based on a hybrid deep learning model was proposed in [23]. Visual analysis of the driver is monitored continuously by a camera, for which CNN is presented. A hybrid CNN is presented for behavior detection. For driver behavior extraction, three different deep learning models are used, including ResNet50, Inception V3, and XCeption, and then the extracted features are applied in the hybrid CNN. In this final softmax layer, anomalies in driver behavior are assessed to manage safe driving habits. The set of driver behaviors classified by the hybrid CNN are safe driving, right-handed texting, right-handed phone use, left-handed texting, left-handed phone use, operating the radio, drinking, glancing behind, hair and makeup, and talking to the passengers. However, this paper only considers visual images when the camera is placed in a different location, and it is not effective in handling driver behavior continuously.

Image analysis-based driver status monitoring using lightweight deep learning algorithms was proposed in [24]. In the first step, the driver’s facial behavior is analyzed to analyze the time series information for driver status behavior analysis. In the second step, the driver’s face is continuously tracked and PydMobileNet is used for facial analysis-based driver behavior detection. However, visual analysis can predict driver behavior using facial emotions and actions, although vehicle wheel movement and lane change deviation must be computed to manage the risk values of a particular vehicle. Otherwise, the ITS system is affected by roadside loss, and accidents to pedestrians and nearby vehicles, that are caused by the abnormal driver.

An approach for the risk assessment of drivers in near risk conditions was proposed in [25]. The data about vehicle crashes and driver behavior was observed to guarantee safety, which avoids emergency events in ITS. A real-world dataset was used, which collected driver, vehicle, and road information for driver behavior modeling. A modified version of rough set theory was used—namely, variable precision rough set—which makes decisions based on real-world information. The mutual information entropy function was used to find the significant road attributes and to determine the risk values. A safety distance was computed for giving a warning message to the following vehicles. Furthermore, there were four factors that were used to analyze the crash risk: driver behavior, vehicle motion, road traffic, and weather conditions. However, lane change deviation was not predicted in this approach, and the health parameters of the driver were not extracted, which increases the misclassification rate.

Using an advanced driver assistance system for increasing safety was proposed in [26]. Driving security and quality for both passengers and drivers are ensured in this paper. A driver complex state (DCS) is predicted using non-obstructive sensors, and artificial intelligence (AI) is used to uncover the driver state. This paper considers vehicle information, driver data through a smartphone, smart wearables, and external data (i.e., road condition, weather condition, lane occupation, and actions of other vehicles). However, personalized recommendation is not effectively designed in this paper, in which the emotional state of the driver is not predicted. Vehicle moving patterns become a more powerful input parameter that aids driver behavior analysis.

An approach for the effective monitoring of drivers current state of action was proposed in [27]. In this paper, driver distraction was predicted by designing driver monitoring systems. The driver’s historical travel data was collected based on two aspects: short-distance and long-distance drivers. Then, driving time, average speed, and frequency and length of breaking time were predicted to analyze the driver’s behavior. However, this paper does not provide the specific distraction type of the driver, which must be detailed to correlate with the environment and to generate the corresponding warning messages to the driver and to other vehicles in the ITS. Drivers’ physiological parameters must be observed through smart wearables to monitor the health issues of the driver. With the lack of information about drivers, the presented study is not able to determine the behaviors of the different profiles/characteristics of the driver.

A cloud-based approach for the detection of driver fatigue was proposed in [28]. In this paper, driver fatigue detection was implemented by using multiple sensors and the reports were processed in the cloud environment. Multimodal feature processing was investigated in this paper, such as facial PERCLOS features, mobile sensors, and inbuilt vehicle sensors, were used to analyze the performance of the driver. There were several driver styles that were predicted and studied in this paper using machine learning with rule-based and deep learning-based approaches. For a real-world environment, it is necessary to compute all of the different parameters of the driver in a DL model.

A deep learning model based on analyzing of interaction between the user and the road was proposed in [29]. The proposed stacked autoencoder (SDAE) was used to extract the features of driver behavior from a dataset that included real driving test and GPS sensor records. The dataset included high-dimensional raw driver behaviors. Cubic representation was used to extract the 3D features, in which the color trajectories represented the driving path. The RGB color features were extracted for trajectory analysis. Then, a windowing process was applied to normalize the data, which converts the dataset into a number of frames based on the time.

In [30], the authors proposed a mobile app for monitoring driver behavior and provided recommendations for preventing accidents. The proposed method used ontology for collecting driver and vehicle information, which helped to provide a recommendation for dangerous states. For this, the mobile phone was used as a front camera, gyroscope, accelerometer, GPS, navigation map, and microphone to collect the information. The main objective of this research was to provide a recommendation to reduce the number of accidents. The proposed system included two processes: dangerous state classification and detection. The classification model collected information regarding the driver’s stress, pulse rate, drunk driving, and distraction. Additionally, the fame images were collected from the driver, including head angle and pitch, eye openness, mouth openness; based on this information, the dangerous states were detected. In the reference model, the recommendations were provided to the driver.

In [31], the authors proposed braking behavior and driver behavior for reducing the probability of accidents using time pressure conditions. The proposed system considered two events—namely, brake pedal force and brake to maximum brake transition time. Additionally, the time conditions included: no pressure, low time pressure, and high time pressure. Based on this information, the braking behavior was analyzed. The simulation result showed that the proposed system achieved better performance compared to the existing systems.

Asymmetric driving theory was proposed for capturing driving characteristics using traffic oscillation [32]. An unmanned aerial network was used to record videos to collect vehicle trajectory data. Initially, the author illustrated the theory of asymmetric behavior, and then the proposed system collected the individual vehicle trajectories using driver behaviors which represented the characteristics of drivers. The data was collected in Nanjing, China. The simulation result shows that the proposed model achieved better performance, in terms of driver reaction in oscillation, compared to the existing model.

Table 1. Summary of the contribution of the approach and results achieved by other research.

Existing Work	Problems Addressed	Proposed Solutions
[33]	Visual result of environment (scenes by images) is considered and processed in DRN, but weather information is not of focus, which causes low accuracy in monitoring driver behavior. System error in modelling driver stress becomes very high due to the usage of the XGBoost algorithm, and training time is large compared to the Catboost and light GBM algorithms.	Visual features of driver’s vehicle are considered in SqueezeNet for modelling facial features and head position. Furthermore, weather information is considered and learned by SqueezeNet for modelling the driver’s current behavior based on past sequences. SqueezeNet is a lightweight deep learning architecture that is able to process both large and small volumes of data to produce the corresponding outcomes.
[34]	Environmental and health status is not considered in this paper, which causes a higher accident risk and high economic damage. This paper does not discuss the alert messages for the personalized monitoring of drivers during driving. CNN does not effectively handle all image features converted from signals, and behavior modelling takes a large amount of time to generate predictions. This increases the chance of accidents and does not ensure traffic safety through early alert message transmission.	Alert messages are forwarded to the drivers and personalized driver monitoring is implemented in this work. SqueezeNet is used to handle individual type of features. Based on the features processed in SqueezeNet, the driver behavior is modelled.
[35]	Cloud-based driver behavior monitoring increases latency. It is not effective to handle the monitoring of a large number of drivers’ behavior for real-time warning message updates. The health (physiological) parameters of the driver are not considered, which are important in emergency situations. The lack of smart sensor deployment increases pandemic cases. Recommendations are not personalized for drivers since environment parameters (events) and weather information is not considered, which causes low accuracy in recommendations.	Edge is deployed in monitoring all RSUs deployed on the road side, which decreases latency. It can handle the number of vehicles inside the region and multiaccess edges coordinate with the corresponding RSUs. Smart wearables, smart sensors, and smart phone devices are used to continuously monitor the physiological parameters and sense the health issues of the driver for accurate behavior modelling. Personalized assistance is recommended for drivers in which the driver’s next action is presented. Based on the action, route planning is initiated for drivers.
[36]	Vehicle motion is not focused, which measures the lane change deviation using image processing techniques. Hence, it is not suited for all types of drivers. High acceleration, continuous speeding, frequent lane changes, and large brakes must be considered to monitor driver behavior. Dangerous driver’s behaviors are not classified, such as drunk, distraction, and fatigue. Hence, warning messages are not optimal in this case (i.e., the decision must be properly made and sent to the pedestrians and other road side vehicles).	Vehicle motion and lane change detection is implemented using the hidden Markov model, which uses vehicle sensor information of past and current sequences. With this, we find vehicle motion of three classes: high, medium, and low. On the other hand, lane change is detected and classified into three classes: frequent change, no change, and mild change. There are several classes of dangerous driver that are identified using multimodal feature processing in the SqueezeNet algorithm.

summarizes the contributions of the approach and the results that were achieved in other research.

3. Problem Statement

For ITS safety provision, driver behavior analysis is essential to provide continuous assistance and route planning for drivers. The specific problem statements that were encountered in the detection and classification of driver behavior are as follows.

The machine learning models that were used for the accurate detection and classification of driver behavior for minimizing road casualties were proposed in [33,34]. The driver behavior was described by the acceleration and speed of the vehicle [33]. The environment was modelled using dilated residual networks (DRN) that split the video image into a number of patches for the driver’s attention. For stress analysis, extreme gradient boosting (XGBoost) predicted the stress values of the driver into three aspects: high, low, and medium. In [34], various drivers’ behavior signals such as gravity, acceleration, throttle, speed, and revolutions per minute (rpm) were used to categorize drivers as normal, aggressive, distracted, drowsy, and drunk driving. A 2D convolutional neural network (CNN) was used to classify driver behavior based on vehicle moving trajectories. For this, the temporal dependencies of driving signals were converted into spatial dependencies. The major problems that are faced in these approaches are:

• The visual result of the environment (scenes by images) was considered and processed in DRN, but the weather information was not considered, which causes low accuracy in monitoring the driver’s behavior;
• System errors in modelling driver’s stress become very high due to the usage of the XGBoost algorithm, and training time is long compared to the Catboost and light GBM algorithms;
• Environmental and health status was not considered in this paper, which causes a higher risk of accidents and high economic damage. Lane change deviation is also taken into account for modelling the behavior of drivers;
• This paper does not discuss the alert messages for the personalized monitoring of drivers during driving. Hence, driver behavior modelling must be investigated using other significant parameters;
• CNN does not effectively handle all of the image features that are converted from signals, and behavior modelling takes a long amount of time to generate a prediction. This increases the chance of accidents and also does not ensure traffic safety through early alert message transmission.

In [35,36], driver monitoring was utilized to provide personalized assistance. This system was developed for the internet of vehicles (IoV), in which on-board image parameters and wearable sensor parameters were collected to compute the deviation degree of vehicles, as well as the head motion of drivers. Based on the observed parameters, an abnormality level of driver behavior was computed. If the abnormality was high, alert messages were forwarded to the neighboring vehicles and the pedestrians that travelled at the roadside. Frequent alert message transmission avoids accidents and economic damage, and also ensures traffic safety. The cloud environment keeps track of drivers’ reports and allows the monitoring of drivers’ behavior [36]. The problems in these approaches are presented as follows:

• Vehicle motion, which measures lane change deviation using image processing techniques, is not a focus. Hence, it does not suit all types of drivers. High acceleration, continuous speeding, frequent lane changes, and large brakes must be considered to monitor the driver’s behavior;
• Dangerous drivers’ behavior is not classified, including drunk, distracted, and fatigued. Hence, the warning messages are not optimal in this case (i.e., decisions must be properly made and sent to the pedestrians and other roadside vehicles);
• Cloud-based driver behavior monitoring increases latency. It is not effective to monitor a large number of drivers’ behaviors to provide real-time warning message updates;
• The health (physiological) parameters of the driver are not considered, which are important in emergency situations. The lack of smart sensor deployment increases pandemic cases;
• Recommendations are not personalized for drivers, since environment parameters (events) and weather information are not considered, which causes low accuracy in the recommendations.

The proposed MODAL-IoCV approach overcomes the above stated problems to effectively analyze driver behavior and provide route assistance. The proposed approach considers the visual features of driver’s vehicles in SqueezeNet for modelling facial features and head position. Furthermore, weather information is considered and integrated into SqueezeNet for modelling the driver’s current behavior based on past sequences. SqueezeNet is a lightweight deep learning architecture that is able to process both large and small volumes of data to produce the corresponding outcomes. Vehicle motion and lane change detection is implemented using the hidden Markov model, which uses the vehicle sensor information of past and current sequences. With this model, we categorize vehicle motion into three classes: high, medium, and low. On the other hand, lane change is detected and classified into three classes: frequent change, no change, and mild change. Edge nodes are deployed for monitoring all of the RSUs that are deployed on the roadside, which decreases latency. It can handle the number of vehicles that are inside the region, and multiaccess edges coordinate with the corresponding RSUs. Smart wearables, smart sensors, and smartphone devices are used to continuously monitor the physiological parameters and to sense the health issues of the driver for accurate behavior modelling. Personalized assistance is recommended for drivers, in which the driver’s next action is presented. Based on the action, route planning is initiated for drivers.

4. Proposed Work

In this study, we concentrated on driver behavior modelling and road accidents reduction by providing early warning alerts using an Internet of Connected Vehicles (IoCV) environment. For that, we designed multimodal feature processing in edge assisted IoCV environment, as shown in Figure 2. The proposed work is organized by the following entities: Vehicles Along with Drivers, Edge Devices, and RSUs. Furthermore, it is organized into two tiers as follows,

• Layer 1 (IoCV)—This layer consists of intelligent vehicles that are connected with the internet. Each vehicle consists of sensors for monitoring speed, acceleration, etc. OBU also contains sensors which are equipped to capture vehicles’ information. This layer also consists of RSUs for data collection from vehicles.
• Layer 2 (Edge Computing)—This layer consists of a number of edges that are responsible for monitoring separate regions of layer 1. Each edge node has a processing capacity that is higher than cloud computing platforms. These two layers are functioned for accidents prevention and generate early warning alerts.

4.1. Data Gathering

Typically, to experiment with the proposed system, we executed a set of driving maneuvers under real conditions. A real scenario rather than a simulation has been chosen to avoid possible deviations from the real behavior of drivers. Three drivers were involved in the experiments. They were driving the same vehicle in real traffic conditions. Participants were encouraged to use all types of driving including (1) normal, (2) aggressive, (3) distracted, (4) drowsy, and (5) drunk. They knew the characteristics of all driving behaviors and imitated the different driving styles by performing certain predetermined jobs. We collected data including acceleration, gravity, RPM, speed, and throttle of the vehicle during the experiment.

All data on different driving styles in different road and transport conditions were carefully collected.

4.2. Driving Vehicle Motion and Lane Change Detection

In its range, RSU is responsible for vehicle monitoring and vehicle motion. Lane change behaviors are predicted. Vehicle motion is described by two sets of attributes as Nearby Vehicle Report $\left(V_R\right)$ and Past Mean Velocity $\left(M_V\right)$. This information is collected and forwarded to the Hidden Markov Model (HMM) that predicts the vehicles motion accurately and also Lane Change Deviation is measured by Vehicle Direction $\left(V_D\right)$, Distance to neighboring vehicles $\left(N_D\right)$ and past sequences of lane change $\left(L_C\right)$, The proposed HMM is defined as follows,

$$H=\left(n,m,\pi ,a,b\right)$$

(1)

where n represents the number of states, and the value of n is 3. In our work, we have three states (frequently state, no change and mild change). $m$ represents the observation symbol and $\pi$ represents the initial state distribution. a represents the transition probability of the state and b represents the probability of observation.

In our work, HMM consider three states with the feature parameters such as $V_R$ and $M_V$ for motion prediction and $V_D, N_D,L_C$ for lane detection. Then, the HMM is defined as follows,

$$H=\left(\pi ,a,b\right)$$

(2)

If the condition is expressed as Q ($i$=1, 2…n), n represents the condition limit of the Markov chain. The observed values are represented as $R \left(j=1,2\dots ,m\right)$ where m represents the number of observed values as $y_t$ and the Markov sequence is represented as $x_t$. The probability matrix of the HMM for vehicle motion detection and lane detection is defined as follows,

$$B={\left[A_{ij}\right]}_{n\times n}$$

(3)

where,

$$A_{ij}=P\left(x_{t+1}=Q|x_t=Q\right),1\le i, j\ge n$$

(4)

The observation probability matrix is defined as follows,

$$D={\left[C_{ij}\right]}_{n\times m}$$

(5)

where,

$$C_{ij}=P\left(y_t=R|x_t=Q\right),1\le i\le n, j\le m$$

(6)

The initial condition probability vector is defined as follows

$$\pi =P\left(x_t=Q\right), 1\le i\le n$$

(7)

Therefore, the probability of motion and lane change detection is defined using HMM, based on the probability, and the HMM provides three classes (frequently change, no change and mild change). Figure 3a illustrates the conceptual diagram of motion and lane change detection, Figure 3b represents the probability matrix which shows the current position and previous position, and Figure 3c represents the process of HMM, where I represents the initial state and E represents the end state.

4.3. Multimodal Features-Based Driver Behavior Modelling

Once the RSU has classified the vehicle, then driver behavior analysis is performed for four classes of vehicles (i.e., high motion, low motion, frequently change, and mild change). Multiple features are extracted using SqueezeNet, which is a lightweight deep learning model that has a more compact architecture than other models. It is a good replacement for AlexNet. The performance of SqueezeNet is increasing due to a smaller number of parameters (50X) and speed (3X). In total, 10 layers are used in SqueezeNet including Conv_Layers, Fire Layers, Max Pool Layers, and a Softmax Layer. The initial layer of the squeeze is a convolutional layer that extracts the features from $3\times 3$ regions. The feature sets are collected from the fire modules, which include driver visual features $\left(V_f\right)$, head position and facial emotion, vehicular features $\left(F_v\right)$, and smart IoT device features $\left(S_f\right)$. These are all considered to predict driver behavior. The visual features conditions re shown in Pseudocode 1.

Pseudocode 1: Visual features conditions

Initialize parameters $N_y, CP,N_{Hn}, S_v, H_r$ While $S_v>0$: { If $N_y>3$ in $T_y=1$|| $CP\ge 25\%$|| $N_{Hn}>4$ in $T_{Hn}=2$ { Current state = drowsy } Else if $H_r\ge {18}^{°}$ in $T_{H_r}=3$ { Current state =distracted } Else { Current state =normal } end }

Table 2. Features considered for driver behavior analysis.

Feature Type	Feature Name	Range
Smart IoT device	Pulse range	50–110 (ppm)
	Rhythm	yes
	Respiratory rate	12–20 (bpm)
	ST elevation	0.13
	Q wave	0.15
	EEG range	8–13 Hz
	ST depression	0.16
Vehicular	Moving direction	(North, South, East, West)
	Position	(Latitude, Longitude)
	Longitudinal acceleration	$0.1–0.22\mathrm{m}/{\mathrm{s}}^2$
	Speed	40-0 km
	Rate of yaw angle	${20}^{°}/\mathrm{s}–{27}^{°}/\mathrm{s}$

and

Table 3. Attributes of driver behavior analysis.

	Attribute Type	Attribute	Description
Driver behavior	Boolean (0-no, 1-yes)	Brake Switch	Keep constant Acceleration Deceleration Steering
	Boolean (0-no, 1-yes)	Acc Pedal
	Boolean (0-no, 1-yes)	Turn indicator
Road obstacles	Continuous	Obstacles in longitudinal direction	Calculated by TTC and classified into 3 levels 1>5, 2=1–5 3=0–2
Kinematic status of vehicle	Continuous	Velocity	Calculate by km/h and classified into 4 levels 1=0–40, 2=41–50, 3=51–60,4=>60

present the condition for determination for driver behavior based on the visual features, wherein $N_y,\mathrm{CP},N_{\mathrm{Hn}},H_r$ denote the number of yawns, closure percentage of eye, and the count of head nods by the driver, respectively, and $S_v$ denotes the speed of the vehicle, which is considered in order to determine whether the vehicle is in motion or parked state. The pulse range rhythm, respiratory rate, ST elevation, Q wave, EEG range, and ST depression are used to detect driver drowsiness. Driving behavior information includes deviations from lane position, vehicle speed, steering movement, pressure on the acceleration pedal, etc. We facilitate the classification of driver behavior based on the significant features such as visual features of driver, vehicular features, and physiological features acquired from the smart wearables and the driver behavior is extensively influencing by the physiological features which are using to accurately detect the driver behavior.

For each kind of feature, an individual SqueezeNet program is used. It has nine fire blocks whose size is $13\times 13\times 512$. Each feature vector is extracted from a vector size of 512. And then the attention branch includes the global average pooling layer which also has the vector size 512. The fully connected layer has a sigmoid function with the size of 169 $\left(13\times 13\right)$ that extracts the weight of the feature vector. The features of the input $\left(I_i\right)$ block are represented as follows,

$$O_i=F\left(I_i\right)$$

(8)

where $O_i$ represents the feature block and $F$ represents the function of convolutional layer in the nine fire blocks. Here, we consider many features such as visual features, vehicular features, and Smart IoT device features. The weight of the features is defined as follows,

$$W_i=\rho \left[M_w{G}_P{O}_i\right]$$

(9)

where $M_w$ represents the weight matrix of the FC layer, $G_P$ defined global average pooling operation, and $\rho$ is a sigmoid function with the range of [0, 1]. The attention is given to the highest weight values. Based on these features, the Softmax layer performs classification with the activation function.

$$X_i=F\left[G_P\left(W_i\times O_i\right)\right]$$

(10)

From this information, the Softmax layer predicts the driver’s current behavior (i.e., drowsy, distracted, emergency, speeding, normal driving, bad pedaling, and bad steering). The overall process is shown in Figure 4 and the layers of input-output and filters information are shown in

Table 4. Layers of SqueezeNet.

Name of the Layer	Filters	Depth	Output Size
Conv 1	$7\times 7/2(\times 96)$	1	$111\times$$111\times$96
Maxpool 1	$3\times 3$/2	0	$55\times$$55\times$96
Fire 2	-	2	$55\times$$55\times$128
Fire 3	-	2	$55\times$$55\times$128
Fire 4	-	2	$55\times$$55\times$256
Maxpool 4	$3\times 3$/2	0	$27\times$$27\times$256
Fire 5	-	2	$27\times$$27\times$256
Fire 6	-	2	$27\times$$27\times$384
Fire 7	-	2	$27\times$$27\times$384
Fire 8	-	2	$27\times$$27\times$512
Maxpool 8	$3\times$3/2	0	$13\times$$12\times$512
Fire 9	-	2	$13\times$$12\times$512
Conv 10	$1\times 1/1(\times 1000)$	1	$13\times$$13\times$1000
Avgpool 10	$13\times 13$/1	0	$\times$$1/1\times$1000

.

After the abnormal behaviors are classified, the current state of behavior is forwarded to the Edge Nodes. The warning messages are broadcasted to nearby vehicles and roadside pedestrians by constructing the virtual graph from source nodes using the ‘perfect roadside graph’, which is constructed by vehicle mobility speed $\left(M_s\right)$, moving direction $\left(M_d\right),$ and distance $\left(d\right)$ between vehicles. Here, the graph is constructed as follows,

$$G=\left(V,E\right)$$

(11)

where, V represents the vertices and E represents the edges. Here the vehicles are considered as vertices and the edges are used to connect two edges by considering the parameters of $M_s,M_d,d.$

$$E\dot{\subseteq }\left\{\left\{u,v\right\}\right|u,v\mathrm{and}u\ne v\}$$

(12)

where u and v represent the endpoints in the edges. For all the forwarding vehicles, the warning message is forwarded. Further, edge node forwards this message to the nearby edge nodes for warnings about abnormal behavior. The Pseudocode 2 shows the multimodal features-based driver behavior modelling.

Pseudocode 2: Multimodal features-based driver behavior modelling

Input: Four classes {C1, C2, C3, C4} Output: Driver behavior { Begin Initialize C1, C2, C3, C4 Initiate SqueezeNet Ensure relevant information is taken as input for i $\leftarrow o to n$ do { Extract $V_f$ from the input Extract $F_v$ from the input Extract$S_f$ from the input F$\leftarrow${$V_f,F_v,S_f\}$ $F_S\leftarrow$Extract squeezeNet features Construct feature map $O_i$ using Equation (8) Calculate the weight values $W_i$ for features using Equation (9) Classifying driver behavior using Equation (10) $B\leftarrow$driver behavior } end for if (B==abnormal) { Forward driver behavior to edge node Calculate $M_s$ Calculate $M_d$ Calculate $d$ Construct virtual graph using Equation (11) } else consider as normal behavior Return B; end }

4.4. Personalized Recommendation and Route Planning

In parallel to the message dissemination, each vehicle of the driver is precisely handled by Edge Node for provisioning of personalized assistance. This precise assistance is recommended by the past, present, and future behavior of the vehicles. We can also send one or more assistance provided according to the current state by using a tri-agent-based soft actor critic (TA-SAC) algorithm. Each agent purpose can be as follows,

• Agent 1: It is used to disseminate the warning messages to nearby vehicles and edges
• Agent 2: It is used to determine personalized assistance for behaviors according to the current state
• Agent 3: It is used for route planning according to the drivers’ preferences

This process is constructed as a Markov decision process (MDP) with corresponding state $(ℌ)$, action $(Å)$, and reward $(Rw)$. Let the policy of process be denoted as ${\pi }_{\varphi }({Å}_t|{ℌ}_t)$. Then, the soft Q function and state value function can be denoted as $Q_{\theta }\left({ℌ}_t,{Å}_t\right)$ and $V_{\psi }\left({ℌ}_t\right)$, respectively. The state represents the current state of the driver and the action represents the dissemination of warning message, providing personalized assistance and route planning which is performed by three different agents.

The squared residual error is minimized by introducing a separate function approximation. The training of soft value function can be expressed as,

$$K_v\left(\psi \right)=E_{{ℌ}_t~D}\left[\frac{1}{2}{\left(V_{\psi }\left({ℌ}_t\right)-E_{{\Omega }_t~{\pi }_{\varphi }}\left[Q_{\theta }\left({ℌ}_t,{Å}_t\right)-\log {\pi }_{\varphi }({Å}_t|{ℌ}_t)\right]\right)}^2\right]$$

(13)

The estimation of gradient function can be formulated as,

$${\widehat{\nabla }}_{\psi }{K}_v\left(\psi \right)={\nabla }_{\psi }{V}_{\psi }\left({ℌ}_t\right)\left(V_{\psi }\left({ℌ}_t\right)-Q_{\theta }\left({ℌ}_t,{Å}_t\right)+\log {\pi }_{\varphi }({Å}_t|{ℌ}_t)\right)$$

(14)

The current policy is considered to perform sampling of actions. The training of soft Q function is optimized by using a stochastic gradient, which can be formulated as,

$${\widehat{\nabla }}_{\theta }{K}_Q\left(\theta \right)={\nabla }_{\theta }{Q}_{\theta }\left({Å}_t,{ℌ}_t\right)(Q_{\theta }\left({ℌ}_t,{Å}_t\right)-Rw\left({ℌ}_t,{Å}_t\right)-{\delta V}_{\psi }\left({ℌ}_{t+1}\right)$$

(15)

The learning of parameter policy is carried out to achieve the optimal policy, which can be formulated as,

$$K_{\pi }\left(\varphi \right)=E_{{ℌ}_t~D,{ϵ}_t~N}\left[\log {\pi }_{\varphi }\left(f_{\varphi }\left({ϵ}_t;{ℌ}_t\right)|{ℌ}_t\right)-Q_{\theta }\left({ℌ}_t,f_{\varphi }\left({ϵ}_t;{ℌ}_t\right)\right)\right]$$

(16)

where $f_{\varphi }\left({ϵ}_t;{ℌ}_t\right)$ denotes the action ${Å}_t$ and by doing so, the optimal policy is achieved for the three agents. For instance, if the current state of the driver is detected as distracted and drowsy, then the assistance recommended to the driver is to take a rest and drive defensively. The current location of the vehicles is used to provide assistance to driver such as “There is a coffee shop in 100 m ahead, take a break and drive safely”. The audio system of the vehicle is utilized to provide these recommendations. Figure 5 presents the architecture of the TA-SAC algorithm in providing precise recommendation, warning message dissemination, and route planning. The Pseudocode 3 of our proposed TA-SAC algorithm is provided below. Figure 6 presents the overall flow chart of MODAL-IoCV approach.

Pseudocode 3: TA-SAC algorithm

Parameter initialization ($\theta ,\varphi ,\psi ,\dot{\psi })$ For each agent do For each iteration do ${Å}_t~{\pi }_{\varphi }({Å}_t|{ℌ}_t)$ ${ℌ}_{t+1}~l({ℌ}_{t+1}|{ℌ}_t,{Å}_t)$ $D\leftarrow D{\displaystyle \cup }\left\{{ℌ}_t,{Å}_t,Rw\left({ℌ}_t,{Å}_t\right),{ℌ}_{t+1}\right\}$ End for For each step of gradient do $\psi \leftarrow \psi -{\alpha }_v{\widehat{\nabla }}_{\psi }{K}_v\left(\psi \right)$ ${\theta }_n\leftarrow {\theta }_n-{\alpha }_Q{\widehat{\nabla }}_{{\theta }_n}{K}_Q\left({\theta }_n\right)$ $\varphi \leftarrow \varphi -{\alpha }_{\pi }{\widehat{\nabla }}_{\varphi }{K}_{\pi }\left(\varphi \right)$ $\dot{\psi }\leftarrow \tau \psi +\left(1-\tau \right)\dot{\psi }$ End for End for

5. Experimental Study

In this section, the experimentation of our proposed MODAL-IoCV approach is carried out in order to evaluate the performance. This section is divided into two sub-sections, namely a simulation study and its corresponding comparative analysis.

5.1. Simulation Study

The proposed MODAL-IoCV approach is simulated using tools such as objective modular network (OMNET++) and simulation of urban mobility (SUMO). Here, SUMO is implemented for traffic simulation and OMNET++ is utilized for network simulation, respectively. The system configurations required for the simulation of the approach are presented in

Table 5. System requirements.

Software Requirements	Network simulator	OMNET++
	Traffic simulator	SUMO
	OS	Ubuntu
Hardware requirements	RAM	8 GB
	CPU	2.90 GHZ
	Hard disk	1 TB
	Processor	Intel core

. The simulation parameters considered for the simulation of our approach are provided in

Table 6. Simulation parameters.

PARAMETERS	DESCRIPTION
Network Parameters
Area of simulation	500 × 500 m
Simulation time	300 s
Number of RSUs	4
Number of Edge nodes	6
Number of cloud node	1
Number of vehicles	100
Type of traffic	Traffic control interface model
Rate of transmission	200 Mbps
Range of transmission	200–250 m
Transport protocol	TCP
Size of packet	512 bytes
Total number of packets	10,000 (approx.)
Mobility model	Random way point

.

5.2. Comparative Analysis

In this section, the validation of our proposed MODAL-IoCV approach is carried out by comparing with existing works such as DRN (XGBoost algorithm-based monitoring model for urban driving stress: combining driving behavior, driving environment, and route familiarity) [33] and CNN (driver behavior detection and classification using deep convolutional neural networks) [36] by means of performance metrics such as motion prediction error, accuracy, latency, and false alarm rate with respect to both number of vehicles and number of edges.

5.2.1. Impact of Motion Prediction Error

The motion prediction error is the measure of rate of error acquired during the prediction of motion of the vehicle. Figure 7 depicts a comparison of the motion prediction error of our proposed MODAL-IoCV approach and existing approaches, with respect to the number of vehicles. The motion prediction error increases with the increase in number of vehicles. The proposed approach possesses low motion prediction error due to the effective prediction of vehicle’s motion using HMM by considering both the past mean velocity and report by the nearby vehicles. By doing so, even if the speed of the vehicle increases the prediction of motion is not interrupted. The increase in speed affects the motion prediction accuracy of the existing approaches due to a lack of consideration of important parameters, thereby resulting in an increased prediction error.

The motion prediction error of our proposed approach is compared with the existing approaches with respect to the number of edge nodes, as illustrated in Figure 8. The motion prediction error decreases with an increase in the number of edges due to the fast computations provided by the edge nodes. The motion prediction error of our proposed approach is low than the existing approaches due to the increased efficiency of monitoring the specific regions by the respective edge nodes. In existing approaches, the error decreases with an increase in edge nodes, but the inefficient prediction of motion still results in an increased error.

5.2.2. Impact of Accuracy

The accuracy is referred to as the measure of correctness in analyzing the behavior of the driver. The comparison of accuracy of the proposed approach and the existing approaches with respect to number of vehicles is presented in Figure 9. The accuracy of our proposed approach in classifying the behavior of the driver is high due to the implementation of SqueezeNet, which possesses increased efficiency in performance.

The classification of driver behavior is performed based on the integration of visual features, vehicle features and features provided by the smart wearables, which provides the drivers’ health information. The consideration of a variety of essential features and efficacy of the proposed SqueezeNet resulted in increased classification accuracy. The existing approaches utilized traditional deep learning models which are inefficient in differentiating between complex features resulting in reduced classification accuracy.

Figure 10 depicts the comparison of accuracy of our proposed approach and existing approaches with respect to the number of edge nodes. The increase in number of edge nodes contributes to the increased accuracy. The reduced error achieved by the proposed approach in predicting the motion of the vehicle also contributes to increased accuracy in the classification of driver behavior. The lack of consideration of necessary features in the classification of driver behavior resulted in reduced accuracy of the existing approaches even with the increase in the number of edge nodes.

5.2.3. Impact of Latency

The latency is defined as the delay associated with the execution of operations involved in analyzing the behavior of driver. The latency is an important metric to be considered in analyzing the driver behavior and generation of personalized assistance in emergency situations. The latency of the proposed MODAL-IoCV approach is compared with the existing approaches with respect to the number of vehicles, as illustrated in Figure 11. The latency increases with an increase in the number of vehicles. The proposed approach possesses reduced latency due to the execution speed of the proposed SqueezeNet, which requires only a fewer number of parameters. The existing approaches implemented more complex methods resulting in increased latency, thereby affecting the performance of those approaches in emergency situations.

Figure 12 illustrates the comparison of latency of the proposed approach and the existing approaches with respect to the number of edge nodes. The latency decreases with an increase in the number of edge nodes. The latency of the proposed approach increases with the increase in the number of edges is found to be low due to the efficacy of the SqueezeNet in the classification of driver behavior. The lightweight nature of the proposed model makes it possible to execute with less consumption of time. The conventional methods possessed by the existing approaches resulted in increased latency even with the increase in the number of edge nodes.

5.2.4. Impact of False Alarm Rate

The false alarm rate is the measure of rate of unwanted warnings provided by an approach due to the inefficiency in the classification of driver behavior. The false alarm rate of the proposed approach is compared with the existing approaches with respect to the number of vehicles as shown in Figure 13.

The false alarm rate increases with an increase in the number of vehicles. The proposed approach possesses a lower false alarm rate than the existing approaches due to precise classification of driver behavior and providing personalized assistance to the driver. This is achieved by implementing TA-SAC in which the three agents are utilized to provide precise assistance and dissemination of warning messages based on the environment. The existing approaches possess low accuracy in detection of driver behavior, thereby providing a high number of false alarms (which affects the regulation of traffic).

Figure 14 presents the comparison of the false alarm rate of our proposed approach and the existing approaches with respect to the number of edge nodes. The increase in edge nodes influences the speed of transmission of warning messages to nearby vehicles. The proposed approach possesses less false alarm rate due to the efficacy of both driver behavior detection and generation of personalized assistance. The increase in the number of edge nodes in the existing approaches, which possess an increased false alarm rate, result in frequent dissemination of false alarms, thereby resulting in wastage of bandwidth.

6. Discussion Section

6.1. Discussion

Context aware technologies employ sensors and other informational sources to inform users and systems about the operating environment and activities within a given domain. Advanced implementation of these systems will increasingly seek to integrate information concerning the physical, physiological, and psychological dimensions of the user. The integration of these three dimensions enables the output of context aware, and other ambient intelligence systems, to provide enhanced monitoring and reporting to a third party on the well-being or behavior of a target population in fixed locations. This context rich information can better inform the management of activities or resources and ultimately the support and motivation of certain behaviors that improve overall wellbeing.

Although previous research results such as [33] driver’s stress is assessed during driving which is based on three different factors as behavior, environment and route familiarity, the system error in modelling driver’s stress becomes very high due to the usage of the XGBoost algorithm and the training time is large as compared to the Catboost and light GBM algorithms. In [36], CNN is presented for driver behavior detection and classification, but CNN is not effectively handle in all image features converted from signals. Behavior modelling also takes a large amount of time for prediction. This increases the chance of accidents and does not ensure traffic safety through early alert message transmission. Thus, in this work the proposed MODAL-IoCV approach is validated with respect to two different metrics, namely the number of vehicles and the number of edges, respectively. The driver behavior analysis is carried out and the generation of personalized assistance based on the current state of the driver is performed in the proposed approach. The classification of driver behavior is facilitated based on the significant features such as visual features of driver, vehicular features, and physiological features acquired from the smart wearables. The driver behavior is extensively influenced by the physiological features which are used to accurately detect the driver behavior. The integration of visual features and physiological features contributed to the increased accuracy of the classification of driver behavior. The existing approaches implemented conventional models and considered either any of the feature set which affects the accuracy of detection. The proposed SqueezeNet performs the classification of driver behavior with reduced time, thereby reducing the overhead associated with the computation. The warning message dissemination and personalized assistance generation are executed by TA-SAC, which provides assistance based on multiple factors. Through this, the proposed approach is able to achieve increased accuracy and reduced latency, motion prediction error, and false alarm rate of about 90%, 0.16 s, 10%, and 0.37, respectively.

The proposed approach possesses reduced latency due to the execution speed of the existing approaches, which requires only a fewer number of parameters. The lightweight nature of the proposed model makes it possible to execute with less consumption of time.

Table 7. Performance analysis.

Performance Metrics	Scenario	Proposed and Existing Approaches
		DRN	CNN	MODAL-IoCV
Accuracy (%)	#of vehicles	$71.7\pm 4$	$78.6\pm 3$	$90.5\pm 2$
	#of edges	$72.6\pm 3$	$78.5\pm 2$	$90.1\pm 1$
Latency (Sec)	#of vehicles	$0.39\pm 0.4$	$0.29\pm 0.3$	$0.16\pm 0.2$
	#of edges	$80.5\pm 3$	$66.1\pm 2$	$44.6\pm 1$
Motion prediciton error (%)	#of vehicles	$22.8\pm 4$	$18.5\pm 3$	$11\pm 1$
	#of edges	$25.6\pm 3$	$15.5\pm 2$	$9.5\pm 1$
False alarm rate	#of vehicles	$0.62\pm 0.4$	$0.50\pm 0.3$	$0.37\pm 0.1$
	#of edges	$0.82\pm 0.4$	$0.65\pm 0.3$	$0.37\pm 0.2$

presents the numerical comparison of the performance metrics of our proposed MODAL-IoCV and the existing approaches with respect to both the number of vehicles and the number of edge nodes. From this, we can conclude that our proposed approach outperforms the existing approaches in all of the metrics to provide precise, personalized assistance and warning message dissemination based on the analysis of the behavior of the driver.

6.2. Safety and Security Challenges

As we have mentioned above, the main research aim is to design of safe roadside environment (i.e., to ensure driver safety, pedestrian safety, and neighboring vehicle safety). Safety and security properties to be taken into account in our work addressing sensitive data can be summarized as follows.

• Confidentiality, which is the property through which data is disclosed only as intended by the data owner;
• Integrity, which is the property guaranteeing that critical assets are not altered in disagreement with the owner’s wishes;
• Availability, which is the property according to which critical assets will be accessible when needed for authorized use;
• Accountability, which is the property according to which actions affecting critical assets can be traced to the actor or automated component responsible for the action.

All above security properties can be achieved when continuous monitoring of vehicle motion and lane change deviation and other multimodal features processing is important for accurate prediction driver behavior analysis. We set our aim in this study to reduce the overhead of the whole network by introducing a multiaccess edge environment. An end-to-end architecture for driver behavior analysis and solving roadside risks via personalized recommendation of assistance (and route planning) is therefore proposed.

7. Conclusions and Future Work

In this paper, a MODAL-IoCV method is proposed for driver behavior analysis using a deep learning approach. 5G is a high-speed and reliable technology, and thus the proposed model using edge computing possesses reduced latency due to the execution speed of the proposed SqueezeNet, which requires only a fewer number of parameters. The proposed method includes two layers, namely the IoCV layer and edge layer. The first layer includes the number of vehicles and RSU for monitoring the behaviors of the vehicles. The second layer consists of a number of edges which is responsible for monitoring the vehicles and driver behaviors. These layers are used for accident prevention and can provide early warning alerts to the drivers. The HMM predicts vehicle motion and lane changes for identifying the behavior of the vehicles. Then, the RSU classifies the driver behaviors into set of classes (i.e., high motion, low motion, frequently change and mild change). From these classes, the features are extracted using SqueezeNet which can classify the behaviors into drowsy, distracted, emergency, speeding, normal driving, bad pedaling, and bad steering, thereby improving the accuracy of the process. Afterward, abnormal behaviors are forward to the nearest vehicles by constructing a virtual graph, which increases traffic prevention. In parallel, the message dissemination of each vehicle of the deriver is precisely handled by the edge node. For that, we proposed TA-SAC, which provides the recommendation based on the current state of the driver behavior. Finally, the performances are evaluated and proved that the proposed model achieves better performance.

In ongoing and future work, we have planned to focus on security in driver behavior analysis with different varieties of attacks.