This paper concentrates on revealing and explaining practical and systematic ways in which the results of different methods designed and applied in the area of intelligent transport systems (ITS) can be effectively compared against each other. The motivation behind this study resides in the claims often found in research articles (e.g., [1
]) and technical projects (e.g., [3
]) that a newly proposed method (for instance, on topics relevant to driving performance, efficient routing, traffic prediction, vehicular communication) leads to a significant improvement in terms of energy efficiency (e.g., energy savings in the order of 10–20% over the state-of-the-art). Such a claim is often dealt with skepticism from the relevant evaluators, because of the usually limited-in-size measurement samples. Moreover, the validation of such claim from the researchers’ side is often not dealt with the proper care. For this reason, the validation of a new method is frequently limited to qualitative criteria and the quantification of the results is usually avoided.
1.1. Aim and Research Questions
Considering the above, the focal point of this paper is to investigate and describe ways of validating if and to what extent a method over-performs compared to a conventional, alternative one. In this context, we employ and apply the hypothesis-testing method of the statistical inference science. In particular, to properly exemplify our approach, we discuss on ways of evaluating two different routing methods, i.e., methods of identifying the optimal route from an origin point to a destination point inside the road network. In detail, our goal is to investigate the validity of arguments such as the following:
Does Method A provide better results than Method B according to a certain routing criterion? As such criterion, the vehicle’s energy consumption is considered in our use case. Thus, in other words, we would like to study whether the average energy consumption of a vehicle following the same routing method A is statistically lower than the consumption of the same vehicle following routing method B.
Is Method A better at a γ% percentage compared to Method B, on the basis of the adopted routing criterion? In other words, we would like to examine whether the average energy savings percentage of a vehicle following routing method A is at least γ% compared to the energy consumption of the same vehicle following routing method B.
In order to provide convincing answers to the above posed questions, a proper statistical analysis of the available experimental data should be performed. In particular, we employ, establish and describe the process of hypothesis testing of the statistical inference for drawing relevant conclusions. To the best of the authors’ knowledge, it is the first time that such a systematic and practical approach is explained in the literature for transport-related use cases. It is also worth noting that, despite the fact that our study concentrates on the comparison and validation of routing methods, based on real experimental data, the same techniques can be adapted and followed in a wide range of transport-related application domains, such as driving efficiency, driver profiling, traffic prediction and optimization, logistics, vehicular communication, and others.
1.2. Related Work
To substantiate the importance of this issue, we examine papers from the literature related to energy, fuel, or emissions improvements brought by novel methods in the fields of eco-driving and green routing.
For instance, the work in [7
] evaluated, among other aspects, the CO2
benefits of eco-driving for various degrees of penetration rates (from 25% up to 100%) and three levels of traffic congestion. It was found that, under free flow traffic, the savings can reach 15% and in normal traffic 10%. In contrast, in cases of congested traffic, it was found that the presence of eco-drivers could increase the overall CO2
emissions. The work in [8
] proposed an eco-approach and departure application which uses information coming from fixed-time traffic signals to guide a driver through the intersection in an environmentally friendly way. The authors report a reduction in emissions in the range of 11–30% when the initial speed is low (up to 30 mph) and a smaller reduction in-between 3.3% and 6.2%, depending on the type of pollutants, for higher initial speeds. The study in [9
] evaluated the environmental benefits of time-dependent green routing in the greater Buffalo-Niagara region of the U.S., using a combination of two simulation models. Results show that the percent reduction was 12.76% for trucks (vs. 12.63% for passenger cars) when attempting to minimize CO emissions, and 10.22% (vs. 10.37% for passenger cars) when minimizing NOx
. The study in [10
] assessed the efficiency of eco-driving as a means for reducing the fuel consumption of freight transport. A large field test was carried out in the Chinese province of Jiangsu, showing savings of fuel up to 5.5% for high-duty vehicles, but not substantial benefits for light commercial vehicles. None of these papers reports the use of a statistical validation approach for their findings.
The work in [11
] provides a comprehensive overview of many solutions for the improvement of green-house gas (GHG) emissions of road freight transport. Specifically, the authors in [11
] review 58 relevant solutions and classify them into four classes, according to the percentage of CO2
e) savings that they can achieve (Class I: 0–6%; Class II: 6–11%; Class III: 11–16%; Class IV: >16%). In the context of the present paper, we select and examine four of these solutions in terms of the approach that they have followed to validate the percentage of CO2
e savings. The selection was random, but we made sure to cover all different classes and to select papers that spanned different years and that contained case studies conducted in various geographic regions.
In this framework, the study in [12
] (Class I) presented a methodology based on a Geographic Information System (GIS) model developed in order to improve fuel and CO2
efficiency of a Greek municipality’s waste collection and transport system, via the reallocation of waste collection bins and the optimization of vehicle routing in terms of distance and time travelled. Using a simulation model coupled with a field study, the authors show that the routing optimization results in a 5.5–12.5% reduction in the distance travelled by the waste collection vehicle, in comparison to the empirical route. The validation approach is not based on statistical inference and it is not clear whether the reported results constitute average values.
The research approach presented in [13
] (Class II) adopted a probabilistic model for vehicle routing and scheduling problems with time windows that was finetuned and applied using a test vehicle in the area of South Osaka, Japan. The authors reported average CO2
savings of 7.6%, as well as similar reductions in local pollutants, compared to the previous routing solution. Although the authors report not just the average values but also the standard deviations of their measurements, yet they do not proceed in a systematic statistical validation.
The work in [14
] (Class III) employed the vehicle specific power concept and used second-by-second vehicle dynamics to extract the emissions on various route alternatives, based on car-floating data collected from various regions in Portugal. Results show that choosing eco-friendly routes can lead to significant savings in CO2
(up to 25%) and other types of emissions. Descriptive statistics and boxplots are provided for the collected measurements in various routes under investigation, however reductions in emissions are presented in the form of percentages compared to the worst alternative route available, without elaboration on the type of statistical validation employed.
The study in [15
] (Class IV) proposed an environmentally conscious optimization model of a supply chain network, based on integer non-linear programming, with an expanded objective function that takes not only transportation costs into account, but also environmental parameters, such as GHG emissions, fuel consumption, noise, and others. The paper studies various solutions related to the planning of changes in suppliers’ or manufacturers’ capacity in a simulated supply chain network. The impact of these planning decisions is reported in terms of percentages (e.g., in the environmental, transportation, and overall costs) against the baseline scenario, without further elaboration on the statistical significance of these results.
Beyond the aforementioned four papers, there are also other interesting relevant studies that we have found and reviewed. For instance, the study in [16
] (corresponding to Class II) conducts a thorough assessment of the impact of green navigation systems in a city’s traffic flows, combining a macroscopic traffic model with a macroscopic emissions model and a GIS. Results show up to 10.4% reductions in CO2
and up to 13.8% in NOx
in congested traffic conditions for a 90% penetration of green drivers, but also that the overall population’s exposure to NOx
increases up to 20.2%. Similarly, the eco-routing study in [17
] in Lund (corresponding to Class I), Sweden, showed that in 46% of journeys (in a sample of 109 journeys), drivers do not choose the most fuel-efficient route. It also showed that green routing can lead to a mean saving of 4.0% in fuel. Despite the thoroughness of these reports, details on the approach followed for the statistical validation of the corresponding results were not provided.
The examples presented above are only a few from a range of studies related to improvements in energy, fuel, and emissions in road transport, which often neglect to present the validation of their results through statistical inference. There might be several reasons for this, e.g., sometimes authors might choose to place more emphasis on the description of their scientific or technological solution rather than on the validation of their results. In other cases, researchers might wish to keep the presentation of results as simple as possible, in order to be able to communicate them more easily to the uninitiated reader. In others, the researchers might not be familiar with the statistical testing process that they have to follow in order to validate their claim, and this reports only average and best-case results as well as standard deviations.
On the other hand, there are also studies in which the authors place adequate emphasis on validation based on statistical inference, even if the corresponding descriptions are somewhat limited. The approaches followed and presented in [18
] are the most relevant to our work presented herein.
], the authors proposed a novel eco-routing technique using vehicle-to-infrastructure communication, according to which a vehicles registers its fuel consumption when it transverses a road link and then transmits it to a traffic management center, so that next vehicles can exploit this information. Using simulation results and the analysis of variance, the authors show that the effect of the packet delay and packet loss are not statistically significant on the eco-routing system performance. In [19
], the authors evaluated the impact of car dashboards on real-world eco-driving behavior. Particularly, the study assessed the effect of numeric and symbolic eco-driving feedback against a control group, by means of field trials conducted in Switzerland. Using a series of regression analyses, results showed that only the symbolic feedback design led to significant reductions of 2–3% in fuel consumption. In [20
], the authors employ a smartphone application as a means of eco-feedback and assess its impact on fuel efficiency. Using a systematic statistical validation approach based on paired sample tests, the authors demonstrate an improvement of 3.23% in the overall fuel efficiency. In [21
], the authors proposed a dynamic eco-driving approach in an arterial corridor with traffic signals, based on velocity planning algorithms which can achieve approximately 10–15% fuel economy improvement.
In summary, the former study [18
] uses simulation results (instead of field data), the second one [19
] focuses on regression analysis and the use of a control group, the third one [20
] employs paired sample tests for validating fuel efficiency, although it does not elaborate on what approach can be followed in cases where paired sample tests are not possible, whereas the fourth one [21
] involves a statistical t-test, although it does not reveal sufficient details (such as the null hypothesis). Thus, these four papers, which seem to have a strong methodological component, all feature some limitations and/or differences with the type of analysis proposed herein.
The remainder of this paper is organized as follows: Section 2
provides the background relevant to the methodology employed for the statistical validation. Section 3
describes the experimental data and elaborates on three different approaches of comparative assessment regarding the efficiency of two competing routing methods. Also, the same section draws practical guidelines for the applicability and limitations of the three approaches. Section 4
draws additional useful conclusions.