Measuring the Change Towards More Sustainable Mobility: MUV Impact Evaluation Approach

Abstract

Urban areas can be considered the ground for the challenges related to the UN’s Sustainable Development Goals (SDGs). The objective of shaping cities as human settlement that will see a more inclusive, safe, resilient, and sustainable future is often argued in literature as an issue dependent on behavioral change of inhabitants in urban areas. In this paper, the authors question if experimental applications based on gamification can co-produce more sustainable neighborhoods through an impact evaluation method that departs from individual choices within the complex of urban mobility. This investigation is carried out within MUV (Mobility Urban Values), an EU research and innovation project, which aims to trigger more sustainable urban mobility in six pilot cities. This article describes the critical method of validation, an impact assessment of the MUV experimental gamification in the pilot cities, in order to represent a proof for future urban strategies. This methodological approach is based on an evaluation structured on indicators of both impact and process suitable for urban contexts. As based on six pilot cities, with possibilities for transferability to other contexts and scalability to other cities, the method represents a reference work for the evaluation of similar experimental applications.

1. Introduction

In recent years, urban sustainability has become a topic with special attention on policy and research. This is manifested also by the United Nations Sustainable Development Goals (SDGs) [1], which dedicate an entire objective to cities in the 2030 Agenda for Sustainable Development specifically to ‘make cities and human settlement inclusive, safe, resilient, and sustainable’ (goal 11). Urban areas are touched in one way or another by almost all the UN’s SDGs, with the need of addressing also issues, such as road traffic (e.g., goal 3.6) and air pollution in cities (e.g., goals 3.9 and 13.2). In parallel, researchers, private companies, and public bodies are nowadays focusing on sustainability approaches that will turn to participative and people-oriented solutions in future cities. Approaches of this kind include questions on how people can assume positive attitudes and routines regarding the use of non-renewable resources. Moreover, the literature on sustainable change often focuses on how people can individually be oriented and sometimes persuaded to use fewer resources; for example, raising awareness on the benefit to achieve more systemic and collective sustainable transitions. Such a movement seems to take as a basic principle what Stephen Wendel pointed out [2]: people are reluctant in changing their habits. Individuals rarely recognize their power to affect and improve the liveability of the context they inhabit as their own neighborhood/city/region/planet and as their own life. It is difficult, indeed, to believe in the ‘butterfly effect’ (i.e., our individual routines have an impact on an entire ecosystem), so that the causal link between every little change of daily routines can have an impact on a systemic dimension of the SDG as fundamental and relevant.

With the attempt to enhance this awareness, studies on behavior change highlight gamification as an effective perspective [3]. Gamification is often defined as the process of game design elements that structure playful activities [4] on non-game contexts [5]. Game design elements are generally driven by a user-perspective that leads individual human personal motivation and/or perception to provide more effective, efficient, engaging, enduring, and entertaining experiences [6,7]. Points, ranks, levels, competitions, challenges, rewards, badges, or reputations are designed to keep users, as players, in the game [8]. Gamification has been much used as a tool for user-design products or services, but also in business with the purpose to motivate individual behavior change.

Since the beginning of the XX century, psychologists, anthropologists, and philosophers have studied the function of playing for human beings. Karl Groos in his “The play of man” [9], besides acknowledging that the instinct of playing has deep physiological bases, also argued that “play” has a fundamentally social function: it offers to humans (and animals) a tool to mastery those activities that bring prosperity for their species. Later in time, Bernard Suits [10] argued that when individuals engage in gamification, they change the temporary perception of their individual experience sometimes breaking habits: “playing a game is the voluntary attempt to overcome unnecessary obstacles”. Further studies on gamification have pointed to diverse psychological motions (e.g., social motivation, intrinsic benefits, monetary, and/or personal rewards), which have proven to have the power to change people choices about their own behavior as, for example, Mihaly Csikszentmihalyi’s [11] study. Gamification can be seen as a means to facilitate change at the level of individual unsustainable habits by promoting more informed, enjoyable, environmental, and social friendly choices.

In the last decade, all over the world, physical and virtual urban games are being designed trying to demonstrate these hypotheses, but how can the gamification of urban mobility experiences become effective and valuable?

MUV (Mobility Urban Values) is a research and innovation action (2017–2020) based on an experimental approach to gamification applied in six European neighborhoods (Buitenveldert in Amsterdam, Sant Andreu in Barcelona, the historic district of the Portuguese county of Fundao, Muide-Meulestede in the harbour of Ghent, the new area of Jätkäsaari in Helsinki, and the Centro Storico in Palermo). The MUV project [12] is a concrete critical attempt to experiment with a gamification approach, with the main objective to engage people to promote a shift towards more sustainable and healthy urban mobility. MUV engagement strategy of co-creation and co-design [13] is based on urban governance and participatory design theories [14,15] that aim at triple-loop learning [16]. Public involvement of citizens and policy actors occurs progressively through thin and thick levels of participation [17] and serves in the co-creation of game communities and in the co-design of game solutions that aim to enhance interaction and to transform urban policy through a ‘conversational planning’ among a variety of actors (from communities to public authorities and vice versa).

MUV method of co-creation and co-design develops socially through a direct dialogue with the local communities and stakeholders, and technically through a mobile app, a network of air monitoring stations and dashboards, all designed through participatory and collaborative methods.

The effort to raise individual and collective awareness on car-dependency and air pollution is aimed to provide an enjoyable experience of the game for citizens with returns on an evidence-based approach to urban policies inspired by people-centered mobility data collected. The active involvement of citizens and other local stakeholders can consequently result in the long term, more efficient, and cost-effective urban planning processes, while achieving global sustainability goals.

The MUV app enables an activity-based game, previously discussed in [18], through a metaphor of sporty narrative that connects diverse kind of users: (i) citizens as MUVers play as athletes to get rewards for their sustainable mobility choices, i.e., walking, biking, and using public transportation, (ii) public authorities as trainers provide training sessions to coach athletes to improve their sustainable mobility skills, and (iii) local business communities, as sponsors, have the opportunity to promote their brand and their products through the athletes’ best achievements. The MUV app (Figure 1) connects all these users when MUVers press a start button choosing their sustainable transport mode at the beginning of their journey, and press it again when they arrive at the destination. As a result of playing MUV, spatial-temporal mobility data of inestimable value are collected and used for impact assessment purpose and for feeding mobility planners and the processes they can follow to design new mobility policies.

One of the main objectives for MUV in respect to the SDGs is also to provide a sensitive impact structure that will validate the MUV assessment method. The impact assessment method is structured through a systemic measure of the SDG goals starting from the tracking data collected. MUV impact assessment is based on the following research questions:

• to what extent can MUV change citizens’ mobility behaviors?
• to what extent can MUV lead to a reduction of urban car-dependency traffic?
• to what extent can MUV reduce the perceived gap between the use of a private car and the use of other more sustainable transport modes?
• how could different interactions among local stakeholders (i.e., citizens, local businesses, local authorities) affect the behavioral change envisaged by MUV?
• what is the added value of data collected by MUV? How can such data be used to provide insights to policymakers and mobility planners?

MUV assessment method is framed in this research paper on the systemic evaluation structure of MUV impacts, according to the diverse spheres of sustainability.

2. Materials and Methods

The impact assessment framework has been developed to validate and measure the added value of the MUV experimental gamification approach under the perspective to structure a method that will become both scalable and replicable in other contexts, thus ensuring an ease acquisition of the whole evaluation approach from other cities.

2.1. The Impact Assessment Framework

In accordance with CIVITAS SATELLITE approach [19], the MUV assessment framework covers not only impacts (impact evaluation) but also the process evaluation. The latter’s final aim is to understand the barriers to MUV implementation, the actions to overcome such barriers, and the drivers to leverage on. The process evaluation is, therefore, synergic to impact assessment and instrumental in answering the research questions set above.

MUV’s assessment approach, as based on several steps, develops a method suitable for impacts and process assessment in urban contexts, accounting for their peculiarities of places and spaces, but, at the same time, allowing comparability of the results in the six pilots (2017–2020). In order to obtain a transparent and correct understanding of the impact and the measure, it is necessary that the evaluation in each individual city/neighborhood follows the same guidelines of evaluation, especially:

• the indicators for measuring the MUV impacts have been selected to be comparable in all the pilot cities. This selection does not prevent cities having their own additional local indicators important for the impact assessment (outside the scope of the project), but only the proposed set of impact indicators guarantees consistency in all the cities;
• the methods of measurement of indicators in cities are aligned, allowing to reveal differences in results.

Special attention is paid to the identification of a set of indicators for impact evaluation (see Section 2.3 for details) according to well-grounded guiding principles. These principles are focused on (i) the transformation of MUV objectives from general indicators to comprehensive indicators, (ii) the coverage of the indicators are thought in relation to meaningful impacts derived from the MUV experimental gamification, (iii) the selection of indicators focus both on the availability of existing indicators and data, and deriving indicators from the MUV’s objectives themselves.

Since indicators are measured to orient progress toward goals, an overall guiding principle is that suitable indicators have been selected by capturing the essence of MUV objectives. This leaves aside the fact that some indicators might be available/already in use for a targeted phenomenon or not. As a matter of fact, as detailed in [20], it is essential to link the indicators to the objectives with future monitoring and evaluation activities; without clear objectives, it is not possible to monitor and evaluate whether an innovative action is on track.

With this in mind, the following indicators for impact evaluation have been selected according to the following MUV’s objectives:

• OBJ 1: Sustainable urban mobility/new mobility culture: MUV promotes a shift towards more sustainable mobility in urban contexts; individual choices are at the core of impact evaluation on behavioral change approach to reduce urban vehicle traffic;
• OBJ 2: Better health and environment: MUV raises citizens’ awareness on the quality of the urban environment and promotes healthier mobility choices, leading to a better environment;
• OBJ 3: Evidence-based and human-centered urban mobility planning: MUV promotes the integration of people and personal mobility data into urban policy-making and planning processes at the neighborhood level;
• OBJ 4: Foster local development: MUV is likely to generate positive spillover effects on the whole neighborhood and surroundings, even at the city level, involving local businesses and stimulating an innovative environment.

Moreover, the proposed assessment framework envisages the following four impact areas, that are well-grounded in the literature of impact evaluation in smart cities [19]:

• IA–1 Society-People: it refers to the effects of the measure on the citizens living in the neighborhood and in the city, in terms of acceptability, mobility habits, perceived wellbeing, and new opportunities at the community level.
• IA–2 Society-Governance: it refers to the effects of the measure on the way society is organized in terms of governance, e.g., planning and urban mobility policies.
• IA–3 Economy: it focuses on the effectiveness and/or benefits derived from the measure in relation to the costs associated with its preparation, implementation, and operation, together with the economic spillover effects deriving from MUV implementation in the local development.
• IA–4 Environment: it relates to the effects on the environment of reducing the use of private motorized transport, thanks to the measure, covering both polluting emissions and energy consumption.

Making use of the relationships between objectives and impact areas is the guiding principle to define the impact indicators that will guarantee that the resulting set of indicators will measure the effects meaningful to MUV and that they will cover different perspectives of the same result.

An overview table providing the list of impact indicators for each impact area and sub-area is shown in Table 2. The whole assessment framework is subject to a continuous review and refinement along the project lifetime in order to assess the feasibility of the baseline computation and of successive monitoring and evaluation; this flexible approach ensures a regular check and a continuous adjustment of the framework, catching new data opportunities, new trends at the societal level, and new policy and planning objectives of the six administrations.

2.2. Impact Evaluation

MUV impact evaluation is prospective, since it has been developed at the same time as MUV action has been designed, and baseline data have been collected prior to the implementation. Prospective evaluations have the best chance to generate valid counterfactuals since, at the design stage, alternative ways to estimate a valid counterfactual can be considered. The resulting impact evaluation is, thus, more likely to produce strong and credible evaluation results.

The impact evaluation follows a before-and-after comparison, as in Figure 2. The before-and-after comparison attempts to establish the impact of MUV by tracking changes in outcomes for program participants over time.

The ‘before’ situation is described by the baseline (see Section 2.4). Since the impact is the difference in outcomes for the same individual with and without participation in MUV, and since it is impossible to measure the same person in two different states at the same time (at any given moment in time, an individual either participated in MUV or did not), we encounter the so-called “counterfactual problem”: how do we measure what would have happened in the case of MUV absence? Although we can observe and measure the outcome for MUV participants, there is no data to establish what their outcomes would have been in the absence of MUV, that is the counterfactual.

Information about the counterfactual (i.e., what the outcome would have been in the six neighborhoods in case MUV action has not been yet fully implemented) is necessary in order to isolate the MUV impacts from the observed changes. Due to practical constraints, the counterfactual estimation in MUV is without a control group and envisages two different possibilities that will be alternatively selected depending on each impact indicator: A. constructing together with the involved stakeholders (e.g., pilot managers, mobility experts) a reference scenario, that, starting from the same baseline, could provide us with a likely counterfactual, or B. using the pre–MUV values at the baseline (t₀) to estimate the post-MUV outcomes (counterfeit counterfactual). A slight adjustment of the second option has been introduced as a third alternative (option C) for the indicators related to the kms traveled on frequent routes (see Section 2.3 for more details about these indicators). In this latter case, the counterfactual estimate is provided by the travel behavior of all the pilot’s players on their most frequent routes as provided during their registrations to the MUV app (thus, not necessarily at t₀). In this way, any behavioral change in their mobility patterns can be estimated by the difference between the observed behaviors and the counterfactual (i.e., the travel behavior they have declared when registering their frequent routes).

Table 1. Possible options for Mobility Urban Values (MUV) impact indicators counterfactual estimate.

Label	Type of Counterfactual Estimate
A	Constructing a reference scenario together with the involved stakeholders (e.g., pilot managers, mobility experts, local decision makers).
B	Using the pre-MUV values at the baseline (t0) to estimate the post-MUV outcomes (counterfeit counterfactual).
C	Using as reference scenario the travel behavior’s information provided by the player during the registration of each frequent route (not necessarily at t0).

summarizes the possible options for MUV impact indicators’ counterfactual estimate; the choice made for the counterfactual estimate of each impact indicator is shown in the dedicated column of Appendix A

Table A1. Mobility Urban Values (MUV) impact indicators: description, baseline, type of counterfactual estimate, and source (AMS = Amsterdam; BCN = Barcelona; FUN = Fundao; GEN = Ghent; HEL = Helsinki; PAL = Palermo).

Indicator (Code, Name, Unit of Measurement, Short Description)	Baseline Value	Counter Factual	Data Source
IA1.S1.1 Awareness level [%] % of people in the pilot with knowledge of MUV	AMS 0.13% BCN 19.31% FUN 0.00% GEN 10.14% HEL 36.43% PAL 38.44%	B	pilot managers
IA1.S1.2 Involvement level [%] % of people in the pilot involved in the co-creation activities and/or other MUV-related activities	AMS 50.37% BCN 4.47% FUN 0.00% GEN 15.17% HEL 0.41% PAL 3.60%	B	pilot managers
IA1.S1.3 Acceptance level [%] % of people in the pilot registered to MUV app	AMS 11.03% BCN 0.21% FUN 0.00% GEN 0.67% HEL 0.05% PAL 0.37%	B	MUV app
IA1.S1.4 Activeness level [%] % of active players in the pilot	0 (all the neighborhoods)	B	MUV app
IA1.S1.5 Perseverance level [%] % of players continuously active in the year	0 (all the neighborhoods)	B	MUV app
IA1.S2.1 Sustainable mobility habits [%] % of kms traveled in a sustainable way (walk/bike/public transport) on frequent routes	AMS 84.77% BCN 81.78% FUN 23.66% GEN 73.27% HEL 65.95% PAL 53.97%	C	MUV app
IA1.S2.2 Use of private car [%] % of kms traveled by private car on frequent routes	AMS 6.21% BCN 10.12% FUN 47.03% GEN 20.37% HEL 29.33% PAL 32.22%	C	MUV app
IA1.S2.3 Modal split [%] % of kms traveled using each mode (walk, bike, public transport, private car, motorbike, carpooling) on frequent routes (6–elements array)	AMS 16.90%; 43.24%; 24.63%; 6.21%; 4.51%; 4.51% BCN 24.04%; 11.20%; 46.53%; 10.12%; 7.17%; 0.94% FUN 17.50%; 2.39%; 3.77%; 47.03%; 12.32%; 16.99% GEN 9.28%; 49.39%; 14.60%; 20.37%; 4.54%; 1.81% HEL 15.30%; 17.27%; 33.38%; 29.33%; 2.36%; 2.36% PAL 27.42%; 11.23%; 15.32%; 32.22%; 11.05%; 2.76%	C	MUV app
IA1.S2.4 Travel time [minutes/day] average daily time spent traveling using each mode (walk, bike, public transport, private car, motorbike, carpooling) on frequent routes (6–elements array)	AMS 64.7; 33.1; 23.2; 4.2; 5.4; 5.4 BCN 90.6; 18.5; 41.6; 11.1; 15.5; 3.9 FUN 59.4; 20.1; 21.9; 15.3; 7.9; 9.4 GEN 50.9; 32.4; 20.9; 9.3; 4.8; 3.3 HEL 32.3; 13.1; 20.0; 16.1; 5.1; 5.1 PAL 39.8; 26.4; 20.9; 13.7; 13.6; 5.3	C	MUV app
IA1.S3.1 Community cohesion among travelers [people/car] level of contact between people living in the community, perception of being part of their community. Proxy: average carpooling vehicle occupancy	NA (the app does not have the carpooling functionality, yet)	-	MUV app
IA1.S4.1 Physical activity [cal/person*week] physical activity performed via active transport (walk and bike) on frequent routes. Proxy: average calories burned on frequent routes	AMS 1971.21 BCN 1810.68 FUN 1326.70 GEN 1518.33 HEL 877.33 PAL 849.63	C	MUV app
IA2.S1.1 Planning process [5–level Likert scale] changes in the process to develop mobility plans, thanks to MUV, in terms of strategic level vision, level of public involvement, sector integration, institutional cooperation, monitoring and evaluation, finance, implementation. Linked to the indicator “public investments” (IA3.S2.1)	1 (all the neighborhoods)	A	local decision makers
IA2.S1.2 Quality of policies, plans, and programs [5–level Likert scale] qualitative evaluation of the change in the process to develop policies, plans, and programs	1 (all the neighborhoods)	A	local decision makers
IA2.S2.1 Rules and regulations [5–level Likert scale] the extent to which MUV has contributed to, or inspired, changes in rules and regulations (i.e., if MUV is able to change the context in which it is applied, by providing a different interpretation of existing rules and regulations)	1 (all the neighborhoods)	A	local decision makers
IA2.S2.2 Policies [5–level Likert scale] the extent to which MUV has contributed to, or inspired, changes in the current urban mobility policies (e.g., update SUMP (sustainable urban mobility plan))	1 (all the neighborhoods)	A	local decision makers
IA2.S2.3 Policy making process [5–level Likert scale] the extent to which MUV has contributed to, or inspired, changes in the process to develop policies and programs, in terms of strategic level vision, level of public involvement, sector integration, institutional cooperation, monitoring and evaluation, finance, implementation	1 (all the neighborhoods)	A	local decision makers
IA2.S2.4 Finance [5–level Likert scale] the extent to which MUV has contributed to—or inspired—the development of new forms of financing of mobility solutions	1 (all the neighborhoods)	A	local decision makers
IA2.S2.5 Cooperation structures with stakeholders [5–level Likert scale] the extent to which MUV has changed the cooperation structures between public and private stakeholders to develop and implement sustainable mobility solutions	1 (all the neighborhoods)	A	local decision makers
IA2.S3.1 Quality of open data [5–level Likert scale] the extent to which MUV has changed the level of quality of mobility open data	1 (all the neighborhoods)	A	local decision makers
IA2.S3.2 Open datasets [#] # of open mobility (government) datasets born, thanks to MUV	0 (all the neighborhoods)	A	local decision makers
IA3.S1.1 Global sponsors involvement [%] involvement of global sponsors in the community (global sponsor: organization providing goods and services globally or nationally)	0 (all the neighborhoods)	B	pilot managers
IA3.S1.2 Community interaction with global sponsors [%] level of interaction of the community with the global sponsors. Proxy: # check–in (and all CTA–call–to–action, if any) at global sponsors/# active players	0 (all the neighborhoods)	B	MUV app
IA3.S1.3 Local sponsors involvement [%] involvement of local sponsors in the community (local sponsor: public or private organization providing goods and services inside the neighborhood)	0 (all the neighborhoods)	B	pilot managers
IA3.S1.4 Community interaction with local sponsors [%] level of interaction of the community with the local sponsors. Proxy: # check–in (and all CTA–call–to–action, if any) at local sponsors/# active players	0 (all the neighborhoods)	B	MUV app
IA3.S2.1 Public investments [€] amount of investments of the municipality on new mobility initiatives, thanks to MUV	0 (all the neighborhoods)	A	local decision makers
IA3.S2.2 Private investments [€] amount of investments of the sponsors on MUV and MUV-related initiatives	0 (all the neighborhoods)	A	global/local sponsors
IA3.S3.1 Innovative environment [5–level Likert scale] the extent to which MUV increases the level of innovativeness of the urban environment, in terms of exploiting new mobility-related opportunities for helping enterprises to innovate or innovate more	1 (all the neighborhoods)	A	local decision makers
IA3.S3.2 Economic activity [5–level Likert scale] the extent to which MUV impacts the economic activity of the pilot, in terms of, for instance, job creation and additional economic activity (e.g., creation of leisure-based networks)	1 (all the neighborhoods)	A	local decision makers
IA3.S3.3 Open data exploitation [#] third-party developments: number of apps/services/API calls developed by third parties from MUV open data	0 (all the neighborhoods)	A	pilot managers
IA4.S1.1 CO2 emissions from road traffic [g/km] average emissions of CO2 per km traveled on frequent routes	AMS 1971.2 BCN 1810.7 FUN 1326.7 GEN 1518.3 HEL 877.3 PAL 849.6	C	MUV app
IA4.S1.2 CO2 level [ppm] the concentration of CO2 in the neighborhood (still unsure that monitoring stations will measure this value)	NA	-	MUV monitoring stations (still unsure)
IA4.S2.1 Noise level [dB] level of noise in the neighborhood	NA	-	MUV monitoring stations
IA4.S2.2 NOx emissions from road traffic [g/km] average emission factors of NOx per km traveled on frequent routes	AMS 0.104 BCN 0.201 FUN 0.189 GEN 0.102 HEL 0.182 PAL 0.161	C	MUV app
IA4.S2.3 NO2 level [µg/m3] the average concentration of NO2 in the neighborhood	NA	-	MUV monitoring stations
IA4.S2.4 PM2.5 emissions from road traffic [mg/km] average emissions of PM2.5 per km traveled on frequent routes	AMS 0.752 BCN 2.300 FUN 1.958 GEN 1.512 HEL 4.079 PAL 6.490	C	MUV app
IA4.S2.5 PM2.5 concentration [µg/m3] the average concentration of PM2.5 in the neighborhood	NA	-	MUV monitoring stations
IA4.S2.6 PM10 concentration [µg/m3] the average concentration of PM10 in the neighborhood	NA	-	MUV monitoring stations
IA4.S2.7 CO emissions from road traffic [g/km] average emissions of CO per km traveled on frequent routes	AMS 0.500 BCN 0.462 FUN 0.764 GEN 0.748 HEL 0.363 PAL 0.893	C	MUV app
IA4.S2.8 CO level [ppm] the average concentration of CO in the neighborhood (still unsure that monitoring stations will measure this value)	NA	-	MUV monitoring stations (still unsure)
IA4.S3.1 Energy consumption from road traffic [kgoe/km] average energy consumption per km traveled on frequent routes	AMS 0.013 BCN 0.019 FUN 0.045 GEN 0.019 HEL 0.032 PAL 0.029	C	MUV app

.

During the monitoring phase, all the impact indicators defined in

Table 2. List of Mobility Urban Values (MUV) impact indicators.

Impact Area	Impact Sub-Area	Code	Indicator
IA1 Society-People	IA1.S1 Citizens participation	IA1.S1.1	Awareness level
		IA1.S1.2	Involvement level
		IA1.S1.3	Acceptance level
		IA1.S1.4	Activeness level
		IA1.S1.5	Perseverance level
	IA1.S2 Behavioral change	IA1.S2.1	Sustainable mobility habits
		IA1.S2.2	Use of private car
		IA1.S2.3	Modal split
		IA1.S2.4	Travel time
	IA1.S3 Community	IA1.S3.1	Community cohesion among travelers
	IA1.S4 Health and wellbeing	IA1.S4.1	Physical activity
IA2 Society-Governance	IA2.S1 Planning	IA2.S1.1	Planning process
	IA2.S2 Governance	IA2.S1.2	Quality of policies, plans, and programs
		IA2.S2.1	Rules and regulations
		IA2.S2.2	Policies
		IA2.S2.3	Policy making process
		IA2.S2.4	Finance
		IA2.S2.5	Cooperation structures with stakeholders
	IA2.S3 Open data	IA2.S3.1	Quality of open data
		IA2.S3.2	Open datasets
IA3 Economy	IA3.S1 Business network	IA3.S1.1	Global sponsors involvement
		IA3.S1.2	Community interaction with global sponsors
		IA3.S1.3	Local sponsors involvement
		IA3.S1.4	Community interaction with local sponsors
	IA3.S2 Investments	IA3.S2.1	Public investments
		IA3.S2.2	Private investments
	IA3.S3 Innovation	IA3.S3.1	Innovative environment
		IA3.S3.2	Economic activity
		IA3.S3.3	Open data exploitation
IA4 Environment	IA4.S1 Climate change (GHG)	IA4.S1.1	CO2 emissions from road traffic
		IA4.S1.2	CO2 level
	IA4.S2 Pollution (emissions/noise)	IA4.S2.1	Noise level
		IA4.S2.2	NOx emissions from road traffic
		IA4.S2.3	NO2 level
		IA4.S2.4	PM2.5 1 emissions from road traffic
		IA4.S2.5	PM2.5 1 concentration
		IA4.S2.6	PM10 1 concentration
		IA4.S2.7	CO emissions from road traffic
		IA4.S2.8	CO level
	IA4.S3 Energy	IA4.S3.1	Energy consumption from road traffic

¹ Particulate matter, with fine particles having a diameter of 2.5 μm or less (PM_2.5) or with coarse particles having a diameter between 2.5 and 10 μm (PM₁₀).

are being collected and analyzed building on a defined monthly monitoring plan, allowing the evaluators to adjust and refine the impact assessment framework according to data and evidence collected.

Finally, in the ‘after’ evaluation, the impacts will be evaluated at the end of MUV action, by comparing outcomes with the counterfactual.

2.3. Impact and Context Indicators

The proposed assessment framework envisages two types of indicators: impact indicators (measuring the impacts generated by the action) and context indicators (providing information describing the geographical context of interest, i.e., the neighborhood/the city). Concerning the process evaluation, such assessment does not envisage process indicators for now, but it deals with a qualitative assessment of the whole process of implementation, investigating the enabling and inhibiting factors.

We are interested in impact indicators, rather than performance indicators, since—in order to answer to the above-mentioned research questions—the focus should be on to what extent a specific initiative, i.e., MUV, has had an impact on different aspects (e.g., society, economy, environment). Since there is not an ideal target of performance to be achieved, targets will not be defined for MUV impact indicators.

As far as the temporal classification of results is concerned, i.e., the results chain, the decision is to merge results in the unique category ‘impact’. The choice of aggregating the chains of causality is due to different reasons [21]: (i) after detailing the chain’s structure, it becomes evident that some chains (considered as minor) could be deleted; (ii) to be practical, the number of categories should not be excessive; (iii) a temporal classification of the results would have added greater complexity to the assessment framework. We do know that output and outcome indicators could be affected in the project lifetime, while impact indicators are likely to be affected after the project has been implemented and is in full use, which might take a few years. Nevertheless, (long term) impacts are included among the MUV indicators since they are a fundamental measure for reaching the project objectives, making it clear how progress toward strategic objectives will be assessed. Thus, from now on, we have referred to impact indicators as indicators of short, medium, and long term effects generated by the measure.

Various institutes and authorities have developed mobility indicators. Even though consensus on meeting the ‘triple bottom line’ exists (i.e., environmental, social, and economic sustainability), yet different indicator sets have been used to evaluate mobility measures in an urban context [22,23]. MUV impact indicators come from different initiatives, such as CIVITAS [19], CITYKeys [22], and TrafficO₂ [24]; whenever necessary, tailor-made indicators have been designed. Regarding the data source of such indicators, Figure 3 summarizes the main data sources for MUV impact indicators: the MUV app, pilot managers, MUV monitoring stations, and local decision makers.

Table 2 presents the set of indicators, that is detailed in Table A1, including a special column aimed at specifying the data source of each indicator. The impact indicators have been classified in the four impact areas introduced in the impact assessment framework (see Section 2.1); moreover, some impact sub-areas have been identified to better organize the set of indicators.

Since the method aims to measure the impacts on the whole system (i.e., city/neighborhood), the impact indicators do not focus on a single individual. Obviously, the computation of some indicators need individual’s data (e.g., the indicators whose data source is the app), but their final value indicates an impact of the action on the neighborhood/city.

Many impact indicators whose data source is the MUV app relate to the kms traveled on frequent routes by the players of each pilot. The choice of computing such indicators only in relation to the frequent routes traveled relies on the fact that a real behavioral change will occur if the MUVers will change their daily mobility behaviors (i.e., the ones on their frequent routes), and not if they are only occasionally sustainable. Consider, for instance, an employee going to the workplace from Monday to Friday in his private car. Then, suppose on the weekend, he goes jogging and has an occasional bike ride, thus accumulating a lot of points on the MUV app. Not just because for this reason, his mobility habits could be considered ‘sustainable’. A real change in his mobility habits will rather be seen when he changes his mode of transport to go every day to the workplace, leaving his car at home and going to work, for example, by riding a bike.

Where possible, the choice has been to define ratio indicators, that are measurement units normalized to facilitate comparisons (e.g., per-year, per-capita, per-mile, per-trip, per-vehicle-year). Some indicators have been proxied by the app’s data, even though other indicators would have been more appropriate in case they were available. During the continuous update of indicators, some indicators can be added or modified whether new data sources become available.

Together with the impact indicators, also the set of context indicators has been developed. Context indicators provide information describing the geographical context of interest (the neighborhood/the city), and they are able to grasp the socio-demographic peculiar features of each neighborhood. They are introduced to facilitate the understanding of the neighborhood’s situation and, thus, of its impact evaluation. According to European Commission [25], context indicators usually deal with economic and financial fields (e.g., GDP, trade flows), social fields (e.g., demography, occupation, gender), and specific important sectors (e.g., education, health, environment). In the MUV case, special emphasis is put on transport and mobility.

The classification of MUV context indicators reflects this taxonomy, covering the following four areas (

Table 3. List of context indicators used in the six neighborhoods (values are presented in Table A2). PDA = Personal Digital Assistant.

Area.	Code	Context Indicator	Description/Sub-Indicators
SOCIO-ECONOMIC	C1	NUTS3 code	–
	C2	Neighborhood name	name of the neighborhood
	C3	Area dimension	area of the neighborhood [km2]
	C4	Population of the neighborhood	number of inhabitants in the neighborhood
	C5	Population of the city	number of inhabitants in the city (municipality)
	C6	Population density in the neighborhood	inhabitants of the neighborhood/km2
	C7	Age structure	age of residents in the neighborhood (share of the population in four age groups: 0–14; 15–64; 65–74; 75+)
	C8	Age structure	age of residents in the city (share of the population in four age groups: 0–14; 15–64; 65–74; 75+)
	C9	Average available income (yearly)	disposable income pro capita of private households by NUTS 2 regions expressed in PPS (Purchasing Power Standard) per inhabitant [tgs00026]
	C10	Driving age population	number of inhabitants with the minimum age for driving a car without supervision
	C11	Business density	number of registered firms per 1000 active people (those aged 15–64)
	C12	Landmarks and historic features	Historic point of interests in the target area (e.g., monuments, museums, ancient churches)
	C13	Shopping areas/commercial centers	the concentration of shopping and service centers in the area
	C14	Neighborhood assets	recreation services (e.g., community centers), recreation/culture availability, cultural facilities (e.g., libraries), schools, hospitals, etc.
	C15	Employment rate	inhabitants employed/working age population (15–64) [lfst_r_lfe2emprc]
	C16	Unemployment rate	inhabitants unemployed/labour force unemployment rate by NUTS 2 regions—from 15 to 74 years [lfst_r_lfu3rt]
	C17	Smartphone ownership	% of smartphone users in the target area
	C18	Internet use on the move (total)	% of internet users on the move (use of mobile devices via mobile or wireless connection: mobile phone (or smartphone), portable computer (e.g., laptop, tablet) or another mobile device (e.g., PDA, e-book reader) away from home or work). Base: individuals aged 16 to 74 [tin00083]
	C19	Internet use on the move (by age group)	% of internet users on the move by age group (16–24; 25–54; 55–74)
TRANSPORT	C20	Public transport	public transport lines (number of bus-tram lines available)
	C21	Public transport	public transport stops (number of bus-tram stops available)
	C22	Public transport	underground lines (number of underground lines available)
	C23	Public transport	underground stops (number of underground stops available)
	C24	Public transport	railway stations (number of railway stations available)
	C25	Public transport	waiting time: average waiting time at a bus-tram station [min]
	C26	Public transport	average number of daily users of public transport
	C27	Public transport	average bus occupancy (average number of passengers on a bus on a working day)
	C28	Public transport	bus fleet by fuel type: share of urban bus types operating in the city (% of diesel, CNG, electric buses)
	C29	Public transport	accessibility: share of the population with appropriate access to mobility services (Population living within 500 m of a bus or metro stop)
	C30	Bike	cycling paths: length of cycling path [km]
	C31	Bike	cycling paths (space devoted to cycling as a proportion of the target area, assuming a width of 1.5 m)
	C32	Bike	bike racks (# bike racks per 1000 inhabitants)
	C33	Bike	bike ownership (bikes owned per 1000 inhabitants)
	C34	Walk	pedestrian areas: – length of pedestrian streets [km] – area of pedestrian squares [km2]
	C35	Walk	pedestrian areas (space devoted to the pedestrian as the proportion of the target area). For pedestrian streets (given the street length), it is supposed a 4 m width.
	C36	Private car	car ownership (passenger cars per 1000 inhabitants)
	C37	Private car	average car occupancy
	C38	Shared services	taxi: number of taxi vehicles operating
	C39	Shared services	other on-demand ridesharing services (e.g., Uber)
	C40	Shared services	carpooling services (e.g., BlaBlaCar)
	C41	Shared services	car sharing cars (car sharing fleet size per 1000 driving age people)
	C42	Shared services	bike sharing bikes and stations (number of bike sharing bikes per 1000 inhabitants)
	C43	Transport costs	cost of a combined monthly ticket (all modes of public transport) [€]
	C44	Transport costs	gasoline prices (June 2018) [€/l]
	C45	Transport costs	average parking cost (hourly) in the neighborhood
	C46	Traffic congestion	peak hours (the peak hours during the weekday)
	C47	Traffic congestion	congestion level (increase in non-highways travel times when compared to a Free Flow situation)
	C48	Traffic congestion	morning peak (increase in morning peak travel times when compared to a Free Flow situation)
	C49	Traffic congestion	evening peak (increase in evening peak travel times when compared to a Free Flow situation)
	C50	Traffic congestion	parking: number of on-street parking spaces (on or along the curb of streets, NO parking garages)
	C51	Traffic congestion	parking: number of off-street parking garages/parking lots and the aggregate number of parking spaces (both indoor and outdoor parking facilities)
	C52	Traffic congestion	average time to travel 1 km by car on a working day [minutes]
	C53	Traffic congestion	average time to travel 1 km by bus on a working day [minutes]
	C54	Traffic congestion	average time to travel 1 km by moto on a working day [minutes]
	C55	Safety and security	people killed in road accidents per 10,000 population [urb_ctran]
	C56	Safety and security	number of car accidents (last year available) car/car
	C57	Safety and security	number of accidents involving a bike (last year available)—car/bike
	C58	Safety and security	number of accidents involving pedestrians (last year available)—car/people
	C59	Safety and security	number of accidents involving a bus (last year available)
ENV.	C60	Weather conditions	average number of days with precipitation (yearly)
	C61	Weather conditions	average annual temperature
INSTITUTIONAL	C62	Sustainable urban mobility plan (SUMP)	does a SUMP exist?
	C63	Easements	mobility subsidies (e.g., for electric vehicles, discounts on parking)
	C64	Mobility policies	for example, limited traffic zones, priority for pedestrians, etc.
	C65	Mobility initiatives/infrastructure already planned	in the neighborhood
	C66	Mobility initiatives/infrastructure already planned	in other parts of the city

): socio-economic (C1–C19), transport (C20–C59), environment (C60–C61), and institutional (C62–C66). MUV context indicators have been collected in a collaborative way, directly involving the pilot coordinators both in the choice of indicators and in the collection of the data.

Table A2. Context indicators: geographic level and their data source.

Code	Context Indicator	Geographic Level	Data Source
C1	NUTS3 code	-	-
C2	Neighborhood name	neighborhood	pilot coordinator
C3	Area dimension	neighborhood	pilot coordinator
C4	Population of the neighborhood	neighborhood	pilot coordinator
C5	Population of the city	city	UrbiStat
C6	Population density in the neighborhood	neighborhood	computation
C7	Age structure	neighborhood	pilot coordinator
C8	Age structure	city	pilot coordinator
C9	Average available income (yearly)	NUTS2	EUROSTAT (2014)
C10	Driving age population	city	UrbiStat
C11	Business density	neighborhood	pilot coordinator
C12	Landmarks and historic features	neighborhood	pilot coordinator
C13	Shopping areas/commercial centers	neighborhood	pilot coordinator
C14	Neighborhood assets	neighborhood	pilot coordinator
C15	Employment rate	NUTS2	EUROSTAT (2017)
C16	Unemployment rate	NUTS2	EUROSTAT (2017)
C17	Smartphone ownership	country	Google Consumer Barometer
C18	Internet use on the move (total)	country	Study (2017)
C19	Internet use on the move (by age group)	country	EUROSTAT (2017)
C20	Public transport	neighborhood	EUROSTAT (2017)
C21	Public transport	neighborhood	pilot coordinator
C22	Public transport	neighborhood	pilot coordinator
C23	Public transport	neighborhood	pilot coordinator
C24	Public transport	neighborhood	pilot coordinator
C25	Public transport	city	pilot coordinator (Moovit where available)
C26	Public transport	city	pilot coordinator
C27	Public transport	city	pilot coordinator
C28	Public transport	city	pilot coordinator
C29	Public transport	neighborhood	pilot coordinator
C30	Bike	neighborhood	info will be derived from the maps
C31	Bike	neighborhood	info will be derived from the maps
C32	Bike	neighborhood	Computation
C33	Bike	city	pilot coordinator
C34	Walk	neighborhood	pilot coordinator
C35	Walk	neighborhood	info will be derived from the maps
C36	Private car	country	Computation
C37	Private car	city, otherwise country	EUROSTAT (2016)
C38	Shared services	city	pilot coordinator
C39	Shared services	city	pilot coordinator
C40	Shared services	city	pilot coordinator
C41	Shared services	city	pilot coordinator
C42	Shared services	city	pilot coordinator
C43	Transport costs	city	pilot coordinator
C44	Transport costs	country	pilot coordinator
C45	Transport costs	neighborhood	www.globalpetrolprices.com (11 June 2018)
C46	Traffic congestion	neighborhood	pilot coordinator
C47	Traffic congestion	city	pilot coordinator
C48	Traffic congestion	city	TomTom traffic index where available, otherwise pilot coordinator (Fundao)
C49	Traffic congestion	city	TomTom traffic index where available, otherwise pilot coordinator (Fundao)
C50	Traffic congestion	neighborhood	TomTom traffic index where available, otherwise pilot coordinator (Fundao)
C51	Traffic congestion	neighborhood	pilot coordinator
C52	Traffic congestion	city	pilot coordinator
C53	Traffic congestion	city	pilot coordinator
C54	Traffic congestion	city	pilot coordinator
C55	Safety and security	city	pilot coordinator
C56	Safety and security	city	EUROSTAT (2016)
C57	Safety and security	city	pilot coordinator
C58	Safety and security	city	pilot coordinator
C59	Safety and security	city	pilot coordinator
C60	Weather conditions	city	pilot coordinator
C61	Weather conditions	city	WeatherBase
C62	Sustainable urban mobility plan (SUMP)	city	WeatherBase
C63	Easements	city/neighborhood	pilot coordinator
C64	Mobility policies	city	pilot coordinator
C65	Mobility initiatives/infrastructure already planned	neighborhood	pilot coordinator
C66	Mobility initiatives/infrastructure already planned	city	pilot coordinator

details the context indicators that have been collected in each MUV pilot. For each context indicator (row), its geographic level is provided (city/neighborhood/…) together with the corresponding data source. The involvement of the local stakeholders and of the pilot managers during this phase has been crucial, and we expect a continuous involvement of such actors during the evaluation so that the impact assessment results can help policy and decision makers in understanding the real impacts of the mobility measure.

Since, due to their nature, such indicators are not likely to drastically change during the project lifetime, an update of the values reported for each pilot in

Table A3. Values of context indicators in the six neighborhoods (at October 2018).

Code	Amsterdam	Barcelona	Fundao	Ghent	Helsinki	Palermo
C1	NL329 (NL326 in NUTS2013)	ES511	PT16H	BE234	FI1B1	ITG12
C2	Buitenveldert	Sant Andreu de Palomar	Old Town	Muide–Meulestede	Jätkäsaari	Centro storico (Old town)
C3	6.57	1.84	2.77	1.51	1.16	2.50
C4	20,219	57,223	9236	5729	3026	23,384
C5	833,624	1,620,809	27,912	257,029	620,715	673,735
C6	3077	31,099	3334	3794	2609	9365
C7	0–14: 12.3% 15–64: 66.0% 65–74: 11.3% 75+: 10.4%	0–14: 13.2% 15–64: 66.6% 65–74: 10.2% 75+: 10.0%	0–14: 14.91% 15–64: 66.23% 65–74: 9.85% 75+: 9.01%	0–14: 21,9% 15–64: 67,6% 65–74: 5,3% 75+: 5,2%	unknown	0–14: 17.6% 15–64: 70.0% 65–74: 6.3% 75+: 6.1%
C8	0–14: 15.6% 15–64: 72.6% 65–74: 6.9% 75+: 4.8%	0–14: 12.6% 15–64: 65.8% 65–74: 10.1% 75+: 11.5%	0–14: 11.75% 15–64: 60.55% 65–74: 13.09% 75+: 14.61%	0–14: 16.3% 15–64: 67.1% 65–74: 7.9% 75+: 8.7%	0–14: 14.0% 15–64: 69.4% 65–74: 9.7% 75+: 6.9%	0–14: 14.4% 15–64: 66.5% 65–74: 10.2% 75+: 8.9%
C9	17,200	16,800	11,600	18,900	18,400	11,800
C10	684,805	1,376,368	23,905	207,795	-	556,734
C11	177 registered firms (13.27 firms per 1000 active people)	in the CITY: 178,607 registered firms	889	unknown	unknown	unknown
C12	Hortus Botanicus VU	- Historical memory (6 buildings) - Industrial and architectural heritage: creation factories, cultural centers - Pd: app Rutes Sant Andreu	- Municipal Museum - Moagem	- old harbor site: old crane, 19th-century harbor hangars, two small churches	None	- 95 churches - 40 convents - more than 400 aristocrats palaces
C13	- shopping mall ‘Gelderlandplein’ - shopping center Rooswijk - shops in Zuidas	- commercial streets - markets	municipal market (Saturday) open-air market (Monday) supermarkets commercial streets in Old Town	- less than 10 cafes and restaurants - grocery stores (mainly run by Turkish people) - municipal market (Saturday)	- Verkkokauppa.com (home appliances and electronics outlet store)	- 4 historic markets - the high concentration of commercial activities (3 main commercial streets)
C14	- 46,700 people working in the neighborhood - business district Zuidas (78% of the people working in the neighborhood) - 3 city parks - sport clubs - low key meeting facilities churches active networks international schools	Parcs Theaters Libraries Associations Strong associative community (262 associations registered). Community centres Numerous sport clubs Recreation Education and schools Churches/Religious associations Active networks	- Fablab (social/business incubator and co–working space) - Living Lab square: concerts and cultural events - two parks - Library - Bars - Moagem, cultural center - Octogono, concert hall	2 schools, neighborhood centre, community centre, small youth centre, cultural centre (usually for small concerts/theatre plays), church might also get new possibilities for cultural events, small library mainly for kids and elderly people in local community centre	- passenger port - Kaapelitehdas event arena - Hotel Clarion (conference center, 16th floor sky bar)	- 7 theaters - nightlife (restaurants, pubs, bars)
C15	76.4%	68.4%	79.6%	70.3%	74.4%	46.1%
C16	4.5%	13.4%	7.1%	3.4%	7.7%	21.5%
C17	87%	87%	67%	71%	79%	76%
C18	87%	78%	58%	75%	79%	32%
C19	16–24: 98% 25–54: 92% 55–74: 72%	16–24: 95% 25–54: 89% 55–74: 49%	16–24: 95% 25–54: 69% 55–74: 24%	16–24: 92% 25–54: 84% 55–74: 52%	16–24: 96% 25–54: 91% 55–74: 55%	16–24: 72% 25–54: 35% 55–74: 11%
C20	5	9	0	2	3	11
C21	43	24	1	12	6	52
C22	0	1	0	0	0	0
C23	0	2	0	0	0	0
C24	0	2	1	0	0	1
C25	unknown	10 min	NA (timetabled buses).	15 min	4 min	23 min
C26	220.384	985,066 millions trips in year 2017 (total)/365 days = 2699 millions daily trips	unknown	unknown	unknown	280,000
C27	20	4.98 users/km used 166.98 users–km/1000 Average journey in Km: 2.80 (191.22 Km bus lanes in the city) (3388.12M bus places/km)	unknown	unknown	unknown	unknown
C28	100% diesel buses (2% euro IV, 72% euro V, 26% euro VI)	45.71% diesel buses (lower than Euro III) 34.84% of CNG buses 0.37% of electric buses 19.08% hybrid buses (diesel-electric/CNG/electric)	100% diesel buses	57% diesel buses (legisl. std) 43% other (hybrid buses) Remark that there are also many trams in Ghent (tram: 28% of the total amount of driven km)	unknown	unknown
C29	info will be derived from the maps	info will be derived from the maps	info will be derived from the maps	info will be derived from the maps	info will be derived from the maps	info will be derived from the maps
C30	info will be derived from the maps	info will be derived from the maps	info will be derived from the maps	9 km cycling infrastructure	2 km of cycle lanes	4.5 km of cycle lanes
C31	-	-	-	-	0.26%	0.27%
C32	unknown	in the CITY: 27,281 (26,158 park places on the street/1123 underground parking)	25 places	41 places	unknown	unknown
C33	780 (from age 12+)	106,453 biking users/6000 bikes (public) 165,499 daily trips (43,836 daily trips by biking)	unknown	823 have at least one bike (stadsmonitor)	unknown	unknown
C34	info will be derived from the maps	info will be derived from the maps	- 220 m of pedestrian streets - no pedestrian squares	info will be derived from the maps	- 1 km of pedestrian streets - no pedestrian squares	- 175,487 m2 of pedestrian squares
C35	-	-	0.03%	-	0.34%	7.03%
C36	481	492	470	503	604	625
C37	unknown	1.6	1.2 (country)	1.12 (based on MobOnderzoek 2015 - aantal autobestuurders en auto inzittenden)	unknown	unknown
C38	4000 taxis	10,522	25	220	1500	unknown
C39	Yes (Uber)	Yes (Cabify)	No	No	No	No
C40	Yes (Toogether)	No	Not structured	Yes	No	No
C41	4233 cars	399 VTC (chauffeur–driven vehicles) licenses/driving age people	No	2.04	250 vehicles	- 85 car sharing parking lots (300 parking spaces) - 140 cars
C42	- 2 bike sharing stations, 7 fast/easy bike rentals at train stations (not for tourists) - an unknown number of bikes	- 465 bike sharing stations (45 are electric ones) - 6300 bikes (300 are electric ones)	no	0.4 (excluding bike sharing system of national railway company) (Trapido)	150 stations, 2550 bicycles -> 0.004 bicycles per inhabitant	- 35 bike sharing stations - 238 bikes
C43	€ 47.00	€ 54.00	unknown	€ 53.00	€ 54.70	€ 32.00
C44	€ 1.68	€ 1.33	€ 1.60	€ 1.48	€ 1.56	€ 1.62
C45	€ 1.40	€ 2.25	€ 0.90	€ 0.80	€ 4.00	€ 1.00
C46	17–18	8–9; 17–20	no (just parking)	8–9; 16:30–18	several short ones related to ferry schedules	8–9/17–19
C47	34%	32%	unknown	30%	31%	46%
C48	35%	51%	unknown	32%	40%	61%
C49	52%	52%	unknown	34%	48%	69%
C50	13433	in the CITY: 114,105 for cars + 68,250 for motorbikes + 10,513 good distribution vehicles	unknown	2301	unknown	unknown
C51	764 (public parking space)	in the CITY: 591,612 off-street parking spaces	unknown	0	unknown	1103
C52	2.4	2.75	1	2.14	3.16	unknown
C53	3.07	4.96	2	2.14	4.5	unknown
C54	2.4	2.75	1	2.14	3	unknown
C55	0.13	0.255	1.07	0.428	0.048	0.371
C56	3474	1698 (2017)	80 (2017)	unknown	unknown	2256 (2016) The Palermo municipality provide only the aggregated data related to the number of accidents occurred.
C57	188	601 (2017)	unknown	unknown	unknown
C58	96	650 (2017)	6 (2017)	unknown	unknown
C59	621	411 (2017)	unknown	unknown	unknown
C60	117	55	112.5	221	183	74.3
C61	10.0 °C	15.5 °C	15.7 °C	11.0 °C	5.0 °C	18.5 °C
C62	Yes	Yes	Yes	Yes	Yes	Under development
C63	- subsidies for electric cars - the network of charge–stations in the public (and non–public) space - environmental zones for trucks, work-buses, taxis, and scooters - a lot of areas with max. 1 parking permit per household - park and ride facilities	- Subsidies, free surface public on the street and free tolls to buy an electric vehicle - Subsidies for electric bikes purchase (PIVE plan–AMB) - Awareness campaigns (motorcycles, Workplace Road Safety Campaign) - Incentives to renovate or sell old cars (T–Verda)	- neigh. inhabitants have a free parking pass (for one car) - national incentives for electric cars	- neigh. inhabitants have a free parking pass (for one car) - subsidies for electric car sharing, bike sharing, electric bikes, CNG	50% discount on parking for low–emission vehicles	- Carsharing vehicles have access to the restricted traffic area and can travel along taxi lanes. - The first half hour of use of the bike sharing system is always free.
C64	- policy priorities for bikes (direct and smooth bike routes, removal of unused bikes in the parking, new biking policy measure) - special attention to road safety (in particular for vulnerable users) - stimulate usage of bikes	- Reduce the number of accidents - Facilitate modal shift towards more sustainable modes. - Reduce air and noise pollution resulting from transportation - Reduce energy consumption in transportation - Ensure accessibility to the mobility system. - Increase the efficiency of transportation systems.	- parking meters to control parking spaces → little impact due to the lack of enforcement (little control of who pays and who doesn’t).	- increasing the car–parking zone outside the city center	- 5-level traffic planning guideline (pedestrians and cycling are a top priority, followed by public transport, logistics and only as of the last priority the private car traffic)	- Limited Traffic Zone (ZTL) in Centro Storico - Two free shuttles that connect the key areas of the old town with the central railway station - Pedestrianization process of streets and squares - Enhancement of bike and car sharing stations
C65	Not at a neighborhood level.	Improve the infrastructure for bus and bikes: increase of bike lanes km, increase of bus lines and of bus lanes km Remarkable works in street “Carrer Gran” (May 2018–May 2019)	construction of cycling paths (in 2018) construction of new bike parking spaces	- new bridge replacing the old one (2022) - new bridge in the south (construction starts in 2019) - the whole neighborhood will become paid parking (not clear yet when) - new cycling and pedestrian areas along the canal - new housings planned	- the former logistics port is being converted into a residential area, traffic planning & construction related to ferry terminal ongoing till 2020 - two more indoor parking facilities (by 2020) - multiple mobility service trials starting summer 2018	- Railway ring under construction with two stops inside the old town - New bicycle lanes planned
C66	More railways will be built and therefore more trains will run. Several streets will become biking streets with the car being guest. In July ‘18 a new underground line is being opened, making a fast connection from north to south.	- Organization of the city’s urban pattern in superblocks - Implementation of the new orthogonal bus network - Development of the cycling network - Promotion and positive discrimination measures of high occupancy vehicles - Review of the regulation of parking on and off road	-	- car sharing - cycling infrastructure - Low Emission Zone (from 01/01/2020)	- a western extension to metro (end of 2017) - bicycle highways (year?) - new tram lines (year?) - new legislation from June 2018 (major deregulation) will pave way for new transport services, including sharing services (e.g., Lyft, Uber)	- Restricted Traffic Zone (ZTL) also outside Centro Storico - Carsharing fleet (IoGuido) will reach 555 vehicles by 2020 - Extension of cycling paths planned by PGTU, from the current 20 km to 145 km

will be performed—if necessary—during the project lifetime.

2.4. The Baseline

The MUV baseline defines the situation before MUV is implemented in each neighborhood. Operatively, the MUV baseline consists of a set of values for each impact indicator in each neighborhood before MUV comes into force. Appropriate baseline data is always critical for impact evaluation, as it is impossible to measure changes without reliable data on the situation before the innovative action begins.

One remark should be made for the baseline of the impact indicators computed from the app data related to the kms traveled for different transport mode by MUV players on average in each neighborhood (in IA1 ‘Society–people’ and IA–4 ‘Environment’). The baseline for such indicators is provided by data collected during the registration procedure of the players in the MUV app, during which the following information is required (compulsorily):

• the modal split of each user on his/her most frequent route(s);
• the length of such route(s);
• the number of times per week he/she travels this route(s).

Furthermore, for the impact indicators in IA–4 ‘Environment’, further information is asked to the player to estimate his/her personal contribution to air emissions (i.e., CO₂, CO, NOx, PMs):

• in the case the player uses a car on the registered frequent routes, the kind of car she/he generally uses (segment: mini/small/medium/large; fuel: petrol/diesel/petrol hybrids/LPG/CNG/electric; EURO standard); then, COPERT model [26] is used to estimate the corresponding emissions in her/his urban context;
• in the case the player uses a motorbike on the frequent routes, the kind of motorbike she/he usually drives (engine: 2 stroke/4 stroke; cubic capacity: <50 cm³/51–250 cm³/251–750 cm³/>750 cm³; EURO standard); then, COPERT model [26] is used to estimate the corresponding emissions in her/his urban context.

The baseline values herewith presented (Table A1) are based on the data collected in each pilot up to 31 December 2018, considering that the app has been officially launched three months before. The missing values (NAs) in the baseline are originated by the fact that the monitoring stations envisaged by the project have not been installed yet, and thus environmental data coming from such stations have not been collected, yet.

2.5. Process Evaluation

As introduced in Section 2.1, while the impact evaluation is aimed to quantify the added value of MUV action in the six neighborhoods along specific impact dimensions, the process evaluation—performed side by side with the impact evaluation—ensures a real understanding of the role the measure can have in a sustainable mobility strategy and provides insight into which elements are crucial to observed impact. MUV process evaluation is a qualitative analysis that deals with the evaluation of the activities of planning, implementation, and operation of MUV, in order to understand whether and why MUV has succeeded or failed in each of the six neighborhoods, with the final aim of understanding the barriers to MUV implementation, the strategies to overcome such barriers, and the drivers to leverage on. Together with the results of the impact evaluation, the outcome of the process evaluation will be the basis for the recommendations for other European cities interested in joining the sustainable mobility action.

The involvement of local stakeholders and pilot managers is crucial in the process evaluation, thanks to their thorough knowledge of the local context. Accurate process evaluation tables have been defined by the evaluators and are being collaboratively filled in by the pilot managers, in order to have a comprehensive understanding of the MUV process in each pilot considering each neighborhood’s peculiarities. A monitoring plan has been defined consisting of three measurements, once every about six months, after the app introduction.

Figure 4 shows an illustrative template table that pilot managers have to fill in during the ongoing process evaluation: for each step (activation of the community and management of the community), enabling and inhibiting factors are investigated per each stakeholder in terms of drivers, barriers, and lessons learned to be turned into action at the following iteration.

Section 3 provides some preliminary results of MUV process evaluation; a complete assessment will be performed at the end of the action, so as to provide recommendations useful for other cities and/or for other sustainable mobility actions.

3. Results

In the previous section, the general approach to evaluate a sustainable mobility measure has been discussed, using the case of MUV action as an example of application. In this section, the preliminary results are discussed, in terms of a set of impact indicators, criteria for their measurement, baseline, and counterfactual estimate. Since the monitoring of the action is still ongoing, the (ex-post) impact evaluation results cannot be presented yet, and they will be the object of future works by the authors.

Regarding the context indicators, many indicators have been requested at a neighborhood level, but—due to the unavailability of data at that geographical level—in some cases, they have been collected at a higher level (e.g., city). Not all requested data was available in each pilot; thus, some missing values are present (labeled as ‘unknown’). Some remarks could be made, observing some relevant context indicators in the different pilots (Table A3):

• the six neighborhoods have a similar age structure of residents (C7, C8), with Ghent and Palermo having a slightly higher percentage of the young population than the other neighborhoods;
• the average available income, measured as the disposable per-capita income of private households, is expressed in PPS (Purchasing Power Standard) to make the figures comparable between regions (Figure 5). The differences in such indicator (C9) are quite relevant, varying from 11,600 PPS in Fundao to 18,900 PPS in Ghent;
• the social structure is different also in employment terms (C16), going from 3.4% of unemployment rate in Ghent to 21.5% of unemployment rate in Palermo (Figure 5);
• the rate of smartphone ownership (C17) will be an important context indicator to take into account during evaluation activities, since using a smartphone constitutes a prerequisite to installing the MUV app. Apart from Fundao (67%), this rate is greater than 70% in all the pilots, with a peak equal to 87% in Amsterdam and Barcelona (Figure 6);
• even though the smartphone ownership rate is quite aligned with the other pilots, Palermo constitutes an exception for the use of the internet on the move (C18, C19). Only 72% of individuals aged 16–24 use mobile devices via mobile or wireless connection in Palermo, against over 92% of the other sites (Figure 6). The difference is even more evident getting older, with 11% of individuals aged 55–74 using the internet on the move, against over 50% of the other pilots (with the exception of Fundao);
• the average waiting time at a bus station (C25) can heavily influence the city travelers to move with public transport. The six pilots show clear differences in waiting times, with users of public transport in Helsinki waiting an average of 4 min at a standstill, against a 23–minute wait for a Palermitan;
• the traffic congestion in the cities (C46–C54) is quite different among the pilots. Consider, for example, the congestion level during evening peaks (C49), i.e., the increase in evening peak travel times when compared to a free flow situation. Palermo reaches 69% of congestion level in the evening peak (i.e., during the evening peak, it takes 69% of the extra time to travel the same route compared to a free flow situation) against the 34% of Ghent;
• another important factor to take into consideration when choosing the travel mode is the rate of car ownership (C36), which is among the highest in Europe in Palermo (625 cars owned per 1000 inhabitants), while it is below the European average in Amsterdam (481 cars per 1000 inhabitants). Fundao and Palermo are, among the pilots, the sites in which the gasoline is more expensive (C44) in relation to the average available income (Figure 7);
• considering the road safety dimension (C55–C59), the numbers show a huge difference in people killed in road accidents in the six sites (Figure 8). Helsinki is the safest city among the pilots (0.048 people killed per 10,000 population), while Fundao is the least secure one (1.07 people killed per 10,000 population). Unfortunately, for many pilots, it was not possible to collect information from the pilot managers about the quantity and type of road accidents (e.g., car-car/car-bike/car-people);
• as shown in Figure 9, the six cities are very different for their weather conditions (C60, C61), that could constitute a relevant factor in the mobility choice of each individual (e.g., walk, bike). Palermo is the hottest pilot and among the least rainy cities (average annual temperature: 18 °C; the average number of days with precipitation in the year: 74.3). In Ghent, it rains two days out of three (221 days of precipitation in one year), while Helsinki is the coldest city (average annual temperature: 5 °C).

Regarding impact indicators, Table A1 provides an overview of MUV impact indicators, together with their updated baseline values and the type of counterfactual estimate established for each indicator (for the counterfactual legend, please refer to Table 1). More detailed indicator definition sheets have been developed to serve as practical information and use guidelines for each impact indicator.

A Python code has been written for the computation of impact indicators, in order to be integrated within the architecture framework. The implemented Python code is composed of three modules. The first module returns, for each most frequent route registered by a player, a) the latitude and the longitude of the origin and of the destination; b) the weekly distance traveled [km] and travel time [minutes] in each transport mode (walk, bike, public transport, carpooling, car, motorbike) on that route, that will be subsequently used as counterfactual (reference scenario), in accordance with the frequency and the modal split declared by the player when registering the frequent route. The second module computes the weekly value of the basic indicators that are related to kms traveled each week on frequent routes (e.g., km traveled by car in each pilot). This code implements an algorithm that estimates the kms traveled in each transport mode by each player on the registered frequent routes. As a matter of fact, only sustainable transport modes can be registered through the MUV app (i.e., walk, bike, public transport), and it is, thus, necessary to estimate the kms traveled by each player in private car and/or motorbike on these routes. Furthermore, this algorithm provides some adjustments to make up for forgotten tracked routes. Such estimates are based on the information provided by the user while registering each most frequent route. The last module is the code running every month (according to a defined monitoring plan) that returns the (monthly) values of impact indicators. In this code, also the counterfactual estimate is performed for the indicators related to the kms traveled on frequent routes in the pilots.

Regarding the baseline presented in Table A1, the authors are absolutely aware that the users of the MUV app are not a representative sample of the target population, but some important considerations emerge. An indicative example could be Palermo’s indicator IA1.S2.3 “Modal split”, defined as the percentage of kms traveled using each transport mode (private car, walk, bike, public transport, motorbike, carpooling) on frequent routes by Palermo’s players. Figure 10 shows the great difference between the modal split of Palermo provided by the city administration (representative of the city) and information collected by Palermo players’ mobility habits during the registration phase to MUV app (baseline).

As shown in Figure 10, people providing the baseline for MUV have very different mobility habits with respect to Palermo population, at least on frequent routes. As a matter of fact, Palermo’s MUV players exhibit high sustainable mobility habits (IA1.S2.1 “Sustainable mobility habits” = 53.97%), and the use of not-active transport is very low (car = 32.22%; moto = 11.05%; carpooling = 2.76%) compared to Palermo in general (private transport = 78%). This factor confirms that, based on the data currently available, MUV players are likely to be more sensitive to the sustainable mobility issues, and therefore constitute a sample slightly deviated from the average population of the city.

As far as the process evaluation is concerned, the first preliminary shortcomings relate to the step of activation of the neighborhood community, since the management of the community itself has not been investigated yet due to the timings of the monitoring plan. In this regard, Figure 11 summarizes the most prominent results of the first monitoring in the pilots.

As depicted in Figure 11, the success factors (i.e., the drivers) that cross almost all the pilots cover the social and political/institutional sphere, showing, on the one hand, the great relevance of effective communication activities to make people aware and involved in the sustainable mobility initiative, and, on the other, the importance of a massive involvement of the municipality to successfully engage the community. On the contrary, currently the barriers to the implementation of the initiative are mainly technological: the users declare that the app enrolment is too long and too complicated, too many bugs have been detected, and the ‘yet–another–app’ syndrome is difficult to overcome. These factors have led many users to a feeling of frustration, which potentially takes them away from the community. Thanks also to the surveys carried out within the process evaluation, during the first months after the launch of the MUV app has undergone numerous releases to make some improvements to the app enrolment and to fix some detected bugs, precisely to meet some questions raised by the users. Furthermore, each pilot has defined some priority takeaways to be turned into action in the following iteration, in accordance with the strong points and the peculiarities of each local context.

The monitoring of the MUV process and impact indicators is ongoing, and future works will deal with the results of ex-post impact and process evaluation within the framework described here.

4. Discussion

MUV evaluation approach, hereby described, effectively addresses and potentially provides answers to the questions raised at the end of Section 1 for diverse practical reasons.

The first is that the overarching framework for evaluation accounts both for impact and process evaluation, thus, leading to a both qualitative and quantitative assessment that attempts to capture the quantification of the outcomes (e.g., reduction of urban vehicle traffic), as well as the understandings related to the implementation of the solution (e.g., how supporting activities could affect MUV outcomes). Moreover, the relationships between MUV objectives and impact areas (Society-People, Society-Governance, Economy, and Environment) are used as a guiding principle to define the impact indicators. This results in a set of indicators that are meaningful for urban contexts to measure experimental gamification issues and suitable measures on the effects that are selected to be relevant. For example, in MUV, these outcomes generate data that will potentially cover different perspectives of the same result. Then, continuous involvement of the local communities during the evaluation process opens a possibility for co-monitoring with local stakeholders on the impact assessment to respond to their expectations. Thus, MUV impact evaluation results can lead to concrete and effective insights for urban policy and decision makers and urban mobility planners. Furthermore, the users’ mobility data and environmental conditions collected through MUV initiative enable more informed policy decisions for urban futures.

The MUV context indicators show how the six pilots are inherently different, and, thus, indicate a likely difference in response to MUV initiative. Such considerations about context indicators will be qualitatively used by impact evaluators to interpret and motivate outcomes that will be measured in each pilot. In this way, it would be possible to include also ongoing mobility initiatives (and/or infrastructure changes) within the neighborhood and the city (context indicators C65, C66). (context indicators C65, C66). The combination of the baseline and the context indicators provides an overall picture of the situation in each pilot before the MUV initiatives come into action. Such a picture can frame the data ‘before’ the implementation of MUV impact evaluation; at the end of monitoring, MUV outcomes can be quantified and, thus, impacts will be evaluated.

The set of 40 MUV impact indicators (Table 2) has been pursued also to be utilized during the MUV co-creation process to support both the pilot managers and the policy-makers in each pilot city towards a ‘conversational planning’ with the citizens.

As detailed in the previous sections, the evaluation approach herewith described is scalable and replicable, to allow consistency and comparability among pilot cities. Several other European cities have been already chosen to join MUV initiative. The method illustrated here is meant to be applied to other urban contexts to reflect on the real effects of these experimental gamification experiences with an aim to feed a sustainable movement for the future.