The Detection of Black Ice Accidents for Preventative Automated Vehicles Using Convolutional Neural Networks

Abstract

Automated Vehicles (AVs) are expected to dramatically reduce traffic accidents that have occurred when using human driving vehicles (HVs). However, despite the rapid development of AVs, accidents involving AVs can occur even in ideal situations. Therefore, in order to enhance their safety, “preventive design” for accidents is continuously required. Accordingly, the “preventive design” that prevents accidents in advance is continuously required to enhance the safety of AVs. Specially, black ice with characteristics that are difficult to identify with the naked eye—the main cause of major accidents in winter vehicles—is expected to cause serious injuries in the era of AVs, and measures are needed to prevent them. Therefore, this study presents a Convolutional Neural Network (CNN)-based black ice detection plan to prevent traffic accidents of AVs caused by black ice. Due to the characteristic of black ice that is formed only in a certain environment, we augmented image data and learned road environment images. Tests showed that the proposed CNN model detected black ice with 96% accuracy and reproducibility. It is expected that the CNN model for black ice detection proposed in this study will contribute to improving the safety of AVs and prevent black ice accidents in advance.

1. Introduction

As discussions on the fourth industrial revolution become more active, there is a movement to utilize big data, artificial intelligence, and 5G. Among them, Automated Vehicles (AVs), a collection of various technologies, are attracting attention in the transportation field. AVs are expected to bring effects such as improving mobility for the vulnerable and reducing traffic congestion costs and are expected to minimize human and material losses in terms of preventing traffic accidents caused by driver negligence [1,2]. Currently, various companies such as Google, NVIDIA, and Tesla are developing and experimenting with AV systems, and each country is reorganizing its institutional foundation to prepare for the commercialization of AVs. Despite these efforts, however, traffic accidents continue to occur in autonomous driving situations, and the social acceptability of AVs has emerged due to Uber’s pedestrian deaths in 2018 [3,4,5]. In order to solve these problems fundamentally, Germany and the United States have issued an ethics guideline for AVs [6,7]. The guidelines specify the need to develop principles to cope with dilemma situations, along with information on the preventive design of the AVs to avoid accidents. Preventive design of AVs is an issue about risk management that can occur in a realistic driving environment, changing from passive safety systems to active safety systems research [8]. In addition, recently, there has been a change in the tendency toward preventing accidents themselves by learning all the accident situations related to AVs [9]. While various preventive design studies are being carried out, there is a lack of research on preventing black ice accidents, which are the main cause of large-scale traffic accidents in winter. Black ice is a thin ice film formed on the road by combining rain and snow with pollutants such as dust, which is likely to lead to fatal accidents because it is difficult to identify with the naked eye. As black ice is considered to be a potential accident factor even in the era of commercialization of AVs, it is expected that technologies that can detect it in advance, and thus prevent accidents, will be required. Therefore, we will adopt the Convolutional Neural Network method, which is known to detect object’s images most effectively, to present measures for preventing AV-related black ice accidents in the study.

This study is conducted in the following order: Section 2 discusses the research on the use of Convolutional Neural Networks (CNN) in the field of transportation and derives the differentiation of this research, while Section 3 sets up the CNN model learning environment for the detection of black ice. Section 4 identifies and analyzes learning results through models, and Section 5 presents implications and future studies with a brief summary.

2. Literature Review

In this chapter, we consider existing detection methods for black ice and studies using CNN in the transportation field to derive the differentiation of this study.

2.1. Black Ice Detection Methods

Methods for detecting black ice include sensors [10,11,12], sound waves [13,14], and light sources [15]. Habib Tabatabai et al. (2017) [10] conducted a study to detect black ice, ice and water in roads and bridges using sensors embedded in concrete. In this study, a sensor that detects the road surface condition was proposed through the change of electrical resistance between stainless steel columns inside concrete. As a result of conducting experiments under various surface conditions, it was suggested that the proposed sensor can effectively detect the road condition, thereby preventing various accidents. Nuerasimuguli ALIMASI et al. (2012) [11] conducted a study to develop a black ice detector consisting of an optical sensor and an infrared thermometer. The study was conducted in Route 39 around Sekihoku Pass, Hokkaido, and was conducted on a total of six road conditions (dry, wet, “sherbet”, compact snow, glossy compacted snow, black ice) and diffuse reflection and reflection. As a result of the experiment, black ice had a large specular reflection ($R_s$) and a small diffuse reflection ($R_D$) was measured, resulting in a low ($R_s$/$R_D$) value. Youngis E. Abdalla et al. (2017) [12] proposed a system for detecting black ice using Kinect. The types of ice (Soft Ice, Wet Snow, Hard Ice, Black Ice) were classified and the thickness and volume of the ice were measured using Kinect. Experiments have shown that the types of ice formed in the range of 0.82 m to 1.52 m from the camera can be distinguished, and the error rate of measured thickness and volume is very low, suggesting that black ice can be detected by utilizing Kinect. Xinxu Ma et al. (2020) [15] studied a black ice detection method using a three-wavelength non-contact optical technology. The study conducted an experiment to distinguish dry, wet, black ice, ice and snowy conditions using three wavelengths (1310 nm, 1430 nm, 1550 nm). As a result of the experiment, it was confirmed that black ice was detected through the reflectance of each wavelength, and it was suggested that it can be used as basic data for the development of equipment to detect road conditions.

2.2. Deep Learning Applications to Intelligent Transportation

Artificial Intelligence (AI) methodologies are currently being used in various fields, and CNN studies using image data for the detection of vehicles and pedestrians, detection of traffic signs, and detection of road surface are actively carried out in the transportation field.

First of all, for vehicle and pedestrian detection, studies using AlexNet [16,17], VGG (Visual Geometry Group) 16 [18], Mask R-CNN [19], and Faster R-CNN [20,21] existed, and a comparative analysis of the performance of Faster R-CNN and YOLO (You Only Look Once) [22,23]. Lele Xie et al. (2018) [24] conducted a vehicle license plate detection study at various angles using CNN-based Multi-Directive YOLO (MD-YOLO). The study proposed an ALMD-YOLO structure combining CNN and MD-YOLO, and compared the performance of various models (ALMD-YOLO, Faster R-CNN, SSD (Single Shot multibox Detector), MD-YOLO, etc.) and found that the newly proposed ALMD-YOLO had the best performance. It also suggested that the simple structure of the model reduced the computational time and that a high-performance multi-way license plate detection model could be established. Ye Yu et al. (2018) [25] proposed a CNN-based Feature Fusion based Car Model Classification Net (FF-CMNET) for the precise classification of vehicle models. The above study utilized FF-CMNET, which combines UpNet to extract the upper features of the car’s frontal image and DownNet to extract the lower features. Experiments have shown performance better than traditional CNN methodologies (AlexNet, GoogLeNet, and Network in Network (NIN)) in terms of extracting the car’s fine features. M. H. Putra et al. (2018) [26] conducted a study using YOLO to detect people and cars. Unlike the traditional YOLO structure, the above study proposed a modified YOLO structure using seven convolutional layers and compared their performance. As a result of the study, the modified YOLO’s 11 × 11 grid cells model had a lower mAP compared to the traditional YOLO model, but had better processing speed. In addition, tests with actual images showed that small-sized people and cars could be extracted.

Second, in the case of traffic sign detection, there have been many studies using the basic CNN structure [27,28,29,30], Mask R-CNN [31,32], and Faster R-CNN [33,34]. Rongqiang Qian et al. (2016) [35] conducted a study using Fast R-CNN to recognize traffic signs on road surfaces. To enhance the performance of the model, the experiment was conducted by utilizing MDERs (Maximally Stable Extremal Regions) and EdgeBoxes algorithms in the object recognition process. The results of the experiment showed that the Recall rate was improved, with an average precision of 85.58%. Alexander Shustanov and Yakimov, P. (2017) [36] conducted a CNN model design study for real-time traffic sign recognition. The study used modified Generalized Hough Transform (GHT) and CNN, with 99.94% accuracy. It was also confirmed that the proposed algorithm could process high-definition images in real time and accurately recognize traffic signs farther away than similar traffic sign recognition systems. Lee, H.S. and Kim, K. (2018) [37] conducted a study using CNN to recognize the boundaries of traffic signs. They designed CNN based on SSD architecture, and unlike previous studies, they proposed a method of estimating the positions of signs and converting them into boundary estimates. Experiments have confirmed that various types of traffic sign boundaries can be detected quickly.

Finally, studies of road surface detection were reviewed to identify road surface conditions and to detect road cracks. Juan Carrillo et al. (2020) [38] and Guangyuan Pan et al. (2020) [39] are both studies that identify road surface conditions. The above studies divided the data into three and four classes and compared the performance of the CNN model. Studies found that up to 91% accuracy was derived, and CNN showed excellent performance in road surface identification. In the road crack detection study, Janpreet Singh et al. (2018) [40] conducted a study using Mask R-CNN to detect road damage in images taken with smartphones. The data utilized 9053 road damage images taken with smartphones, and the CNN’s structure utilized Mask R-CNN. Experiments have confirmed that road damage is detected effectively, showing high accuracy and 0.1 s processing speed. Zheng Tong et al. (2018) [41] conducted a study to classify the length of asphalt cracks using DCNN (Deep Convolutional neural networks). Data collection was conducted in various places and weather conditions, and the data were divided into eight classes from 0 cm to 8 cm in 1 cm increments. As a result of the experiment, the accuracy was 94.36% and the maximum length error was 1 cm, and it was suggested that the length of the crack can be classified as well as the existence of a simple crack. Baoxian Li et al. (2020) [42] conducted a study using CNN to classify road crack types. Road cracks were classified into a total of five types (non-crack, transverse crack, longitude crack, block crack, alligator crack) and four models were designed using the basic CNN structure. As a result of the experiment, the accuracy of the four models designed was 94% or more. It was also confirmed that CNN, which has a 7 × 7 size of the reactive field, was the best choice for crack detection.

2.3. Summary

In summary, it was confirmed that various studies using black ice detection research and CNN in the transportation field are in progress. In addition, a study on CNN in the transportation sector has been conducted to detect the most important objects that make up road environment, such as pedestrians, vehicles, traffic signs, and road surfaces, and object detection using CNN shows a fast processing speed and high accuracy. In spite of such studies, it is expected that there is a limit to preventing black ice accidents in advance due to problems such as the installation of traditional black ice detection systems. Accordingly, this study proposes a method to detect black ice by identifying road conditions based on the CNN technique to prevent black ice accidents in AVs

3. Learning Environment Setting

CNN is a type of AI that uses convolutional computation, which emerged in 1998 when Yann LeCun proposed the LeNet-5 [43]. CNN is one of the most popular methodologies in image analysis, with features that maintain and deliver spatial information on images by adding synthetic and pooling layers to existing Artificial Neural Networks (ANN) to understand dimensional characteristics. As we considered earlier, there are various studies using CNN in the transportation sector, but the study of black ice detection on the road has only thus far been conducted using other methodologies (sensors and optics) [10,11,12,13,14,15] other than research using AI. Black ice is reckoned to be a potential accident factor in the future era of AVs as it leads to large-scale collisions in winter due to features that are hard to distinguish with the naked eye. Accordingly, we will perform the detection of black ice by utilizing the CNN technique, which is considered to have excellent performance in object detection using images, rather than the traditional black ice detection methods.

The proposed learning environment of the CNN model for black ice detection consists largely of data collection and preprocessing, model design and the learning process. In this chapter, we set up the data collection, 1st preprocessing, and 2nd preprocessing, and the model was designed and learning undertaken (see Figure 1).

3.1. Data Collecting and Preprocessing

This chapter consists of data collection for learning black ice detection, 1st preprocessing, and 2nd preprocessing.

3.1.1. Data Collection

In data collection, the method of data collection and the splits on the collected data were performed.

• Data Collection

For learning, image data was collected using Google Image Search, and data was collected in four categories: road, wet road, snow road and black ice. During the collection process, image data taken in various regions and road environments were obtained, and a total of 2230 image data were collected as shown in

Table 1. The Number of Image Data.

	Road	Wet Road	Snow Road	Black Ice	Total
Number	730	610	570	320	2230

.

2.

Data Split

We split various regional and road environment data collected through Google Image Search. This is the process of removing objects that interfere with the extraction of features, such as road structures, lanes, and shoulder, within the image so that the characteristics of each category can be clearly identified. In this process, there were pros and cons depending on the size of the data (see

Table 2. (Dis) advantage by data size.

	256 × 256 px	128 × 128 px
Advantage	Easy to identify image characteristics	Large number of images Deep neural network can be implemented
Disadvantage	Small number of images Unable to implement deep neural network	Hard to identify image characteristics

). Smaller data sizes have disadvantages in identifying image characteristics compared to larger cases, but they have the advantages of having a large number of images and having deep neural network implementations. On the other hand, when the data are large, feature extraction can be more accurate because it can clearly identify the characteristics of the image compared to the smaller image, but the disadvantage is that the number of images is reduced and the deep neural network is difficult to implement. Accordingly, in this study, data crop was carried out in 128 × 128 px size to proceed with learning through deep neural network and large number of images. The results of the data split are shown in Figure 2.

3.1.2. 1st Preprocessing

In the 1st Preprocess, the channel setup and data padding for learning were performed.

• Channel Setup

The color image of 128 × 128 px obtained earlier through data split has the advantage of being easy to identify the characteristics of the data in the form of three channels. However, since there are three channels of data, the size of the data is large, which limits the number of learning data and the implementation of deep neural networks, this study has transformed the data into black and white image data to conduct learning (see

Table 3. Features of RGB, GRAYSCALE.

	RGB	GRAYSCALE (Black and White)
Number of Channels	3 Channels	1 Channel
Feature	Large data size	Small data size
Advantage	Easy to identify image characteristics	No limit on the number of learning data Deep neural networks can be implemented
Disadvantage	Limited number of learning data Deep neural network impossible to implement	Hard to identify image characteristics

).

2.

Data Padding

Data padding is one of the ways to resize learning images by adding spaces and meaningless symbols to the end of existing data. As a result of learning without data padding in the augmentation conducted during the 2nd preprocess of this study, very low accuracy (25%) and high loss values were identified (

Table 4. Data Augmentation and Learning Results.

		Original Data	Padding Data
Data augmentation results
Learning results	Loss	1.39	0.26
	Accuracy	0.253	0.891

). This is because the edges of the image data are distorted by the data enhancement. Accordingly, in this study, the image data were padded to prevent distortion of the edges of the data.

During the 1st preprocessing, during which channel setup and data padding were performed, image data of 150 × 150 px in GRAYSCALE format were obtained as follows: There are 4900 road and wet road image data and 4900 snow road and black ice image data (

Table 5. Number of data through 1st preprocessing.

Class	Size	Number
Road	150 × 150 px	4900
Wet road		4900
Snow road		3900
Black ice		3900
Total		17,600

).

3.1.3. 2nd Preprocessing

In the 2nd preprocessing, there was a limit to collecting more diverse image data through Google Image Search, so data sets were built through data augmentation to increase their accuracy.

For AI learning models, large amounts of data are essential for high accuracy and prevention of overfitting [44]. In particular, black ice, which is intended to be detected in this study, is characterized by seasonal characteristics and unusual forming conditions that do not occur in many places. As a result, the data collection process did not collect a large amount of data compared to the other data. Accordingly, to improve the accuracy of CNN proposed in this study, the ImageDataGenerator function provided by the Keras library [45] was used to augment the data under the conditions in (

Table 6. Setting Data Augmentation Value.

Transformation Type	Value
Rotation	20
Width shift	0.15
Height shift	0.15
Zoom	0.1

).

The process of building a data set through data augmentation is as follows. From the previously obtained 17,600 sheets of data, 1000 were randomly extracted per class and set as test data. Since then, the data have been augmented using the ImageDataGenerator function for the rest data, which has built 10,000 data per class. The train data and validation data were then set at 8:2. Accordingly, the final data set was set to 8:2:1 in all classes to proceed with the learning (Figure 3 and

Table 7. Dataset for class.

Class	Train Data	Validation Data	Test Data	Total
Road	8000	2000	1000	11,000
Wet road
Snow road
Black ice

).

3.2. CNN Design and Learning

The structure of the CNN model used in this study consists of the Feature Extraction and Classification as shown in Figure 4. In the feature extraction, two convolutional layers, two max-pooling layers, and one dropout layer were arranged to conduct two iterations (Figure 4a). Each layer was then arranged once and repeated twice (Figure 4b). In addition, we used ReLU (Rectified Linear Unit), which has a fast learning speed and prevents gradient vanishing [46], as the activation function. In the case of kernel size of the convolutional layer, (3,3) was applied because the repetition of (3,3) has a fast learning speed and extracts features well [47]. The Stride of the max pooling layer was (2,2), and the Dropout rate of the dropout layer was experimentally applied 0.2. In the classification, the Fully-Connected layer and the Dropout layer were alternately placed, Softmax was applied to the output layer (Figure 4c), and the SGD (Stochastic Gradient Descent) Optimizer was used for high experimental accuracy. In addition, we applied 200 epochs, 32 batch size and the earlystopping function for optimizing the model and preventing overfitting. The earlystopping function terminates learning when there is no room for improvement in the model. Therefore, we designed this study to stop learning if the validation loss does not update the minimum value within 20 times (see

Table 8. CNN Model Setup Value.

Class	Value
Activation Function	ReLU
Kernel size	(3,3)
Strides	(2,2)
Dropout rate	0.2
Optimizer	SGD
Epoch	200
Batch size	32
Earlystopping	20

)

4. Result

4.1. Result

Based on the CNN model, the loss of train data and test data was found to be 0.008 and 0.097, respectively, and the accuracy was 0.998 and 0.982, respectively (see Figure 5 and

Table 9. Training Result.

Class	Loss	Accuracy
Train	0.008	0.998
Test	0.097	0.982

).

For more detailed learning results analysis, performance indicators by class were identified with train data and test data. First, we analyzed a confusion matrix for train data and checked the classification results by class. Confusion matrix is a matrix for comparing the predicted class with the actual class to measure the prediction performance through training, with the x-axis representing the predicted class and the y-axis representing the actual class. The results of the Confusion matrix (Figure 6) showed that some mutual confusion occurred between black ice and snow road, and when the actual class was wet road, it was predicted as road.

Secondly, the calculation and analysis of accuracy, precision, and recall of each class was conducted on test data. The calculation results of each performance indicator are shown in

Table 10. Accuracy, precision, recall results by class.

Class	Accuracy	Precision	Recall
Road	0.996	0.99	1.00
Wet road	0.989	0.99	0.99
Snow road	0.981	0.97	0.98
Black ice	0.961	0.98	0.96
Average	0.982	0.983	0.983

. This shows that the accuracy of black ice, wet road and snow road is measured as relatively low, which is estimated to be the result of loss of light characteristics in the same way as the Confusion matrix analyzed earlier. However, the average values of accuracy, precision and call were 0.982, 0.983, 0.983, and 0.983, which are considered to have produced significant learning outcomes, even though the data are not relevant to learning.

4.2. Discussion

The 69-car collision on the Virginia Expressway in December 2019 and the traffic collision on the Yeongcheon Expressway in Korea in December of the same year were all caused by black ice. As such, casualties from black ice accidents continue to occur worldwide. To prevent this, measures such as installing grooving (a construction method that makes small grooves on the surface of the road to reduce braking distance), installing LED signs, and installing heat wires on the road are currently being proposed. However, since the proposed measures are not a preliminary measure to prevent accidents, a proactive strategy to prevent black ice accidents is being devised. Accordingly, the study was conducted to detect black ice in advance using AI methodologies to extract high accuracy. As further confirmed earlier, the existing black ice detection method is expected to present a prevention measure using light [10,15], as a methodology using light sources is used. This strategy is due to the light reflection feature of black ice, and the image data used in this study also needs to be considered. As confirmed by the analysis results, we have checked that the black ice and snow road appear in the Confusion matrix, and we will explain the cause through the train data of two categories. Figure 7 shows the RGB version of train data, with black ice data showing reflections of light, and snow road data showing snow crystals and slush. However, in this study, because the neural network was designed low due to the limitations of computing and the transformation to GRAYSCALE was carried out (Figure 8), it was difficult to clearly identify these characteristics, resulting in confusion between the two classes and relatively low black ice accuracy of the test data.

4.3. Application Method

We propose an application that can take advantage of the CNN-based black ice detection method we proposed. It will offer a means of preventing accidents by being mounted on AVs and CCTVs. First of all, it is expected that the camera attached to the vehicle will be able to detect black ice in advance by pre-learning the detection of black ice in accordance with the engineering characteristics of AVs. In addition, CCTV is expected to become an important medium for C-ITS (Cooperative Intelligent Transport System) in the future, so it is expected that it can be installed in a proper black ice accident prediction area to prevent accidents. CCTV and cameras installed in AVs are used to forward information on the presence or absence of black ice to nearby vehicles and take measures to undertake a detour of the area or reduce vehicle speeds to prevent major accidents caused by black ice.

5. Conclusions

This study conducted a study using a CNN to detect black ice that is difficult to judge visually in order to prevent black ice accidents in AVs. Data were collected via classification into four classes, and each class’s train, validation, and test data were set through pre-processing of split, padding, and augmentation. Unlike the DCNN model, the CNN model proposed in this study was designed to be relatively simple but showed an excellent performance with an accuracy of about 96%. This suggests that it is more effective to optimize the neural network depth according to the object to be detected rather than to detect black ice by increasing the amount of computation through a complex neural network model. In addition, in this study, a neural network was designed and learned through GRAYSCALE as a feature of black ice mainly formed at dawn, but it was found that some specific classes were confused due to the loss of light characteristics. Accordingly, we plan to conduct research on neural network design that is more optimized for black ice detection by utilizing RGB images in the future. Additionally, since the data were collected through Google Image Search, only images detected close to the object are classified. Accordingly, we plan to construct a CNN model applicable to various situations by setting the distance and angle to the object to be detected in various ways [48,49,50] in the future.

This study is significant in that black ice, which is deemed a potential risk factor even in the era of AVs, was detected using AI, not sensors and wavelengths. It is expected that this will prevent black ice accidents of AVs and will be used as basic data for future convergence research.