Using Video Replay of Simulated Driving to Estimate Driving Safety and Cognitive Status

Abstract

Cognitive decline resulting from Dementia of Alzheimer’s Type (DAT) can lead to reduced ability to perform complex daily tasks required for independent living, including driving an automobile. This study explores the ability of untrained observers to classify driving safety using short video clips of simulated driving through intersections; it also examined whether untrained observers could predict whether the driver was cognitively healthy or cognitively impaired. Participants (n = 54) were shown a series of 30 video clips arranged in an online survey and asked to answer questions following each clip regarding the safety of the maneuver and the cognitive status of the driver. Results showed that participants’ subjectively rated DAT drivers as significantly less safe in comparison to control drivers, F (1, 52) = 228.44, p < 0.001. Participant’s classification of DAT drivers and controls was also significantly higher than chance (i.e., >50% correct). Findings provide preliminary support for the development of a clinical decision-making aid using video replay of driving simulator performance in fitness-to-drive assessments for individuals with cognitive impairment.

1. Introduction

Driving requires a wide range of cognitive abilities including integrating sensory information from multiple sources, perceptual and spatial processing, motor skills, decision making, and planning [1]. Cognitive functioning is highly associated with driving performance and for older adults who experience significant cognitive changes, decrements in attention, memory, and executive functions can reduce their driving safety [1,2].

Dementia of Alzheimer’s Type (DAT) is the most common form of dementia, accounting for between 60 and 80 percent of all dementia cases [3]. The cognitive decline associated with DAT leads to a decreased ability to perform complex tasks required for independent living, such as driving. However, research clearly shows that not all drivers living with dementia are at an increased risk. Even if the level of risk is greater in older individuals living with dementia, research supports the argument that a diagnosis of dementia should not be the sole basis for revoking driving privileges [4,5]. This is especially crucial in regard to the importance of driving and mobility for many older individuals. Driving a motor vehicle remains an integral component to multiple domains of daily life for many older adults: it represents mobility and independence, and is associated with increased health and quality of life [6,7,8,9]. Conversely, in research focused on the experiences of persons with dementia, driving cessation has been linked to negative outcomes, including physical and mental health declines, reduced mobility, and reductions in quality of life [10]. Driving cessation can lead to a decrease in perceived autonomy and independence, a reduction in opportunities for social engagement, and an increase in depressive symptoms [11,12,13,14]. It is therefore of critical importance to balance the potential harms associated with driving cessation with the potential increases in risk to safety when considering fitness-to-drive decisions for persons with dementia.

Despite persons with dementia being at increased risk for motor vehicle collisions as a result of their cognitive deficits, an appreciable proportion of drivers with mild to moderate dementia are deemed safe to continue to drive their car [15]. While drivers in the mild stage of dementia may continue to drive safely, all drivers living with dementia will have to cease driving as their disease progresses and cognitive declines affect their ability to drive [16]. Considering the importance of driving for the majority of older adults, a challenge involves determining the point at which, during the disease course, one’s safety risk is no longer acceptable.

Current practice to address the issue of safety in drivers with dementia involves clinical assessment to determine an individual’s fitness to drive. This process can vary widely, and there is no clear consensus on the best practice, though there are some consistent findings that suggest a combination of assessment techniques including neuropsychological measures and clinical interviews and examination (including functional mobility testing) are better predictors of driving safety than any one measure alone [17]. In many jurisdictions, physicians are required to evaluate dementia patients’ safety and suitability to drive, and make a recommendation regarding driving fitness [18,19,20]. Further complicating this process, research shows that some health professionals tasked with making decisions regarding fitness to drive may not feel confident in making this assessment, and they may lack the appropriate tools to make this determination, including knowledge regarding the minimal requirements for fitness to drive for older drivers [21,22].

Common practice may include structured clinical interviews, neuropsychological testing, and/or an on-road driving test [23]. On-road driving tests can be burdensome in terms of time commitment and financial impact. Comprehensive neuropsychological assessments may identify specific cognitive deficits that increase crash risk; these assessments can not only be costly and time consuming [24], but may also lack ecological validity in terms of driving safety. Shorter cognitive screens that can be performed in a doctor’s office (e.g., Trails A + B, Clock drawing) may be less resource intensive, but are less sensitive than more comprehensive testing and also lack ecological validity. As clinical assessment of older drivers can be costly and lengthy, one alternative to extensive neuropsychological assessment and on-road testing is the use of a driving simulator [25,26].

Research examining the use of driving simulators in fitness to drive assessments has demonstrated promising results and has been found to be a valid measure of on-road performance for both healthy [27] and at-risk drivers [28,29]. Performance on a simulated driving task can be used to discriminate drivers with diagnosed cognitive impairment from those without [30], and may be useful for discriminating individuals with different forms of dementia [31]. In particular, drivers with Dementia of Alzheimer’s Type and Lewy Body type obtain significantly lower simulated driving scores (due to speed exceedances, lane positioning errors, and crashes; [32,33]).

In addition to overall scores of driving performance on simulated driving tasks, drivers with dementia demonstrate difficulties in specific driving scenarios and tasks that may be indicators of risk for collision. In particular, when compared to healthy older drivers, those with DAT show greater difficulties navigating intersections in a simulated driving environment; the greatest difficulties are evident in the preparation for navigating an intersection [30,34]. These preparatory behaviors include adjusting speed, estimating distances, keeping proper lane positioning, and obeying verbal commands. Further, evaluation by a trained scorer, blind to the driver’s cognitive status (i.e., healthy vs. DAT), has shown that DAT drivers have significant difficulty engaging in intersections, which is consistent with previous research [34,35].

Reviewing video clips of an individual’s performance navigating intersections in the driving simulator is one potentially useful method of using simulator data as a clinical decision-making aid. The purpose of this study is to test the use of short driving simulator video clips of maneuvers deployed at intersections as a means to facilitate the determination of driving safety. We were also interested in determining whether untrained observers could discriminate healthy older drivers from those living with dementia. We predicted that drivers with dementia would be rated as less safe. Moreover, we expected observers to be able to discriminate whether the maneuvers that they rated were executed by a healthy older driver or a driver with dementia.

2. Materials and Methods

Fifty-four English-speaking students enrolled in a first-year psychology course were recruited through the Integrated System for Participation in Research (ISPR) at the University of Ottawa to complete an online survey. This research was approved by the University of Ottawa Research Ethics Board (REB#H10-16-09; Approved 30 January 2017).

A series of 360 video clips was created, showing drivers navigating intersections on a pre-determined simulated driving course, guided by computerized voice directions. The recorded videos of driving reactions were gathered during a previous study, in which seventeen patients diagnosed with mild Alzheimer’s, and nineteen cognitively healthy controls—all aged over 65 years-old and holding a valid driver’s license—completed a simulated driving scenario designed to emulate an on-road evaluation (see [31] for information regarding the simulator technical features and the procedure).

In the previous study, older adults with a diagnosis of mild DAT (7 women, 10 men) were recruited from a hospital-based memory clinic. The DAT participants ranged in age from 67 to 90 years old, with a mean age of 79.5 years old (SD = 6.8). All of these participants received their diagnosis from a neurologist. Severity of dementia was assessed using the Global Deterioration Rating Scale (GDS, [36]). Drivers in the DAT group had a mean Mini-Mental State Exam (MMSE, [37]) score of 25.06 (SD = 3.83, range from 13 to 30) and a mean GDS score of 3.05 (SD = 0.24, range from 3 to 4).

The neurologically healthy control participants (10 women, 9 men) were recruited through a newspaper announcement and were subject to an additional 20 min screening call to determine study eligibility. These participants were an average of 77 years of age (SD = 5.9, range from 68 to 86 years) and had a mean MMSE score of 29.00 (SD = 1.3, range from 25 to 30) and a mean GDS score of 1.19 (SD = 0.4, range from 1 to 2). Exclusion criteria for both groups consisted of any serious health problems, including visual or hearing impairments, medication that could alter cognitive functioning, and a history of substance abuse or learning disorder. Control participants were also excluded if they had a score less than 25 on the MMSE, as scores below 25 on this measure may indicate cognitive impairment [38].

The STISIM Drive software (version 2.08.004; Systems Technology, Inc, Hawthorne, California) was used to create the simulated route. In the current study, we used the play back function of the software to replay the maneuvers executed by the drivers. All the drives were taped on a digital camera and were edited using a movie editing software (Quick Time Player).

Before completing the simulated driving scenario, all participants completed a comprehensive training session. In addition to a thorough explanation of the task, this training session included an accommodation phase in order to allow participants to practice operating the pedals and steering wheel, followed by a 20 min training course in which they practiced accelerating, braking, maintaining their speed, making left and right turns, and negotiating traffic [39].

The simulated driving course was designed to emulate a standardized on-road assessment used by a provincial licensing body and was designed based on a real segment of road found in Thunder Bay, ON, Canada [40]. This 12.3-km course included sixteen intersections and segments of urban, highway, and residential environments. The course also presented variable driving conditions to capture a range of road design, traffic conditions, speed, and intersection types (road signs and/or traffic lights, etc.), as well as three different verbal prompts for driver behavior (“turn left”, “turn right”, go straight”). This simulated test course has been used in several other studies [33,41,42] and has demonstrated important similarities between simulator and on-road performance [42].

From each driver’s simulated driving course, ten intersections of interest were selected based on their relative complexities and variations in left/right turns and red/yellow/green traffic lights [30]. Twelve out of the total sixteen intersections were lighted, and two of these were excluded due to a lack of complexity (e.g., go straight at a green light). The final ten intersections selected included three left turns, four right turns, and three “go straight” instructions and were therefore representative of the overall route. The video clips of these intersections were created so that each scene began 250 m before each intersection, and ended 250 m after the intersection, or upon a collision. The clips ranged from 11 s to 99 s in duration. One clip of each intersection, for each driver resulted in a pool of 360 clips (i.e., 10 intersections × 36 drivers). For the purpose of creating the online survey, the clips were uploaded to YouTube as unlisted files, which were only able to be accessed through a direct web address. The clips shown only depicted the front view of the driver’s car from the driver’s seat perspective. Information from behind the car projected in the central mirror was also available in each clip.

Ten surveys of thirty clips each were created by randomly selecting clips from the group of 360. The number of clips presented was decided based on feasibility (a survey of less than an hour) and decreasing the fatigue factor. Two constraints were applied to the randomization process: (1) of the thirty clips, fifteen were drawn from DAT patients and fifteen from the age-matched control participants; (2) a maximum of two clips from the same driver was allowed for each survey. This was done to mitigate the effect of outliers. The survey was administered via Qualtrics Surveys.

Participants were given a detailed set of instructions at the beginning of the survey. They were asked to watch a series of thirty clips of simulated driving through intersections, and answered two questions after each clip. The first question asked them to classify individuals in terms of cognitive status (i.e., cognitively healthy vs. cognitively impaired). The second question asked participants to provide a safety rating on a 10-point Likert scale ranging from “Very unsafe—0” to “Very Safe—10”. A still example of a clip can be seen in Figure 1.

Data for four participants who completed the survey in less than 20 min were excluded from analyses. A mixed Analysis of Variance (ANOVA) was conducted to compare participants’ survey responses after viewing videos of DAT and control drivers. This comparison also sought to examine differences in ratings of the twenty participants who completed the survey in person and the 34 participants who completed the survey online in order to control for potential bias associated with administering the survey remotely. We used descriptive statistics (mean percent [%] correct) and single sample t-tests to determine if participants were able to accurately classify drivers in terms of cognitive impairment.

3. Results

Fifty-four participants (15 women and 38 men; mean age = 19.30 years SD = 1.21), were given the option of completing the survey online or in person. Twenty participants completed the survey in person at the University of Ottawa campus, and 36 completed the survey online.

Statistically significant differences were observed in safety ratings across DAT and control groups, whereby participants rated drivers with DAT as less safe (M = 2.43, SD ± 0.184) than controls (M = 4.77, SD ± 0.200), F (1, 52) = 228.44, p < 0.001. The ANOVA also yielded no effect of location, such that the differences in safety scores between participants who completed the survey online or in person did not reach significance, F (1,52) = 0.78, p = 0.381. Mean safety ratings are displayed in Figure 2 for each category of drivers, as well as each testing condition (online or in lab). No statistically significant interaction between location and cognitive status was observed, F (1,52) = 2.572, p = 0.115.

Participants were found to be able to correctly assign participants to their group (DAT versus control) based on the viewing of a single short clip of their driving. DAT drivers were correctly classified in 67% (SD = 19%) of the driving scenario, on average, while older drivers were correctly classified on 56% (SD = 20%) of the driving scenario, on average. The mean differences in percent classification reached statistical significance for both DAT (t (53) = 6.52, p < 0.01) and control drivers, (t (53) = 2.27, p = 0.027), indicating that classification of driver status by participant, no matter whether they completed the survey in person or online, was better than random chance (50%).

4. Discussion

In this study, we sought to explore whether reviewing videos of driving simulator performance could provide useful information regarding an individual’s cognitive status and fitness to drive. Untrained lay participants were able to distinguish DAT drivers from controls at better-than-chance rates. Specifically, subjective safety ratings were significantly lower for DAT drivers compared to controls. Finally, no significant differences were observed between online and in-lab groups.

The results of this study provide preliminary evidence of face-validity for the use of brief driving simulator segments of maneuvers at intersections as a decision-making aid for fitness-to-drive evaluations. Previous research has shown that errors committed while navigating intersections in a driving simulator task can be used to discriminate between DAT patients and controls [30,33,35]. This study adds to these findings by demonstrating that short segments extracted from a longer assessment drive do provide relevant information regarding driving safety. Moreover, we found that lay observers who have little driving experience and have no training in cognitive assessment whatsoever were nevertheless able to classify drivers as cognitively impaired or not based on video segments that were shorter than 99 s.

There are a few notable limitations to this study that warrant discussion. First, participants were asked to simultaneously rate driver’s safety and make a determination as to their cognitive status. Thus, it is possible that the safety ratings could be influenced by the rater’s perception of cognitive fitness/impairment. While these are not necessarily related causally, we would expect some overlap between the two factors. This potential confounder should be addressed in future research.

Subjective safety ratings were relatively low for both DAT and control groups (M = 2.43 and 4.77, respectively, out of a possible 10). Although the subjective safety ratings between DAT and control groups were significantly different, safety ratings for the control group were lower than might be expected for healthy drivers. This may reflect nuances in driving simulator performance due to unfamiliarity with the platform and/or time to adapt to the simulator task. Despite training and orientation to the task, many drivers may still find the simulator to be novel and/or awkward enough that the errors that they commit is reflective of their adaptation to the driving simulator environment. Alternatively, it is possible that some of the individuals in the control category suffered from deficits in cognitive functioning that negatively affected their driving performance, despite normal scores on cognitive tests (i.e., the Mini-Mental State Exam); this is unlikely, however, as all control participants were deemed cognitively healthy at the time of assessment. Similarly, physical limitations were not assessed or controlled for. Thus, drivers with conditions such as arthritis may have had difficulty steering and/or braking, which would negatively impact their performance on the simulator task.

In the current study, participants who rated the simulator segments had a mean age of 19, indicating little driving experience, as a learner’s permit is only available at 16 years of age in Canadian jurisdictions. Further, participants had no training in assessing cognitive impairment or observing the driving reactions of cognitively impaired drivers, and, presumably, no training or experience with driving simulators in general. Participants’ lack of knowledge and training regarding simulator performance in various age groups, on top of their lack of driving experience, are postulated to be the most likely explanation for the low safety ratings overall. As this is a rather young and homogenous sample, the generalizability of these results is limited. The cultural context is also limited to a mid-sized Canadian city. Thus, it is possible that more experienced and/or more diverse raters (in both age and cultural context) would provide different results.

It is important to highlight that the video clips were not pre-selected in terms of their inherent capacity to discriminate cognitive fitness. In other words, we did not select the most unsafe reactions of a given driver. Instead, we included them all. Indeed, we employed the objective approach of randomly selecting clips from both groups. Thus, some driving errors committed by “healthy” participants are likely to have been included in the pool of clips seen by the participants and vice versa for dementia patients. Similarly, participants did not get to watch multiple clips from the same drivers. Instead, the judgements about safety and group allocation had to be established for every single segment which were each randomly selected and presented to participants.

In this preliminary validation study, our objective was to demonstrate that short video segments of simulated driving carry relevant information regarding driving safety. We also showed the relevance of these clips by demonstrating that lay observers can discriminate, with higher than chance performance, cognitively healthy drivers from drivers with dementia. It is important to reiterate that many drivers in the early stages of dementia are safe to continue driving [43]. Moreover, evaluation of driving fitness is the responsibility of trained healthcare professionals, including physicians and occupational therapists. As such, the ability of lay observers to distinguish between dementia and control participants merely illustrates qualitative differences observed in the driving simulator and offer preliminary data to support the potential of this approach for use within the driving assessment process as conducted in clinical settings. To use simulator video in its current form as a diagnostic tool for dementia would be premature and further refinement of the approach and validation amongst trained healthcare professionals responsible for assessing fitness to drive is needed.

It is reasonable to hypothesize that experienced evaluators—such as Occupational Therapists (OTs) that specialize in fitness-to-drive assessments, and/or physicians who are regularly tasked with making fitness-to-drive decisions regarding their patients—would be able to use simulator video replay to more accurately predict the safety of drivers with dementia. Further, if multiple relevant segments of the same driver were to be systematically tallied and showed to a health professional or driving assessment specialist, the determination of the driver’s safety should be even more accurate. This information could be especially valuable if used in conjunction with the results of cognitive assessments and other relevant clinical information (e.g., functional mobility) collected by the health team as part of a comprehensive assessment. That is, video replay of simulator performance, while insufficient as a decision-making tool alone, could be a useful addition to the fitness-to-drive assessment when considered alongside other important predictors of driving safety.

The long-term goal of this line of inquiry is to develop a brief, ecologically valid assessment of driving abilities, indicative of current or future driving difficulties. Results from future research should help to determine the value of using driving simulator video replay as a clinical decision-making aid. Using curated video clips of simulator performance could allow for the inclusion of important additional information regarding a patient’s driving behavior as part of their fitness-to-drive evaluation. Additionally, this information could potentially also be used to provide patients with realistic and relevant feedback regarding their driving behavior.

These preliminary findings may help guide and refine the technological development of a simulator-based driving assessment that can be integrated into the clinical assessment of older adults presenting at memory clinics. This tool could be designed to suit specific assessment contexts through customization for various clinical settings. With further validation, this approach could address an important gap in the clinical assessment of fitness to drive for dementia patients. By providing clinicians with a tool that aids the decision-making process, those tasked with making these important determinations may be able to do so with greater confidence.