Evaluation of the ORCHESTRA project outcomes

In the recently completed deliverable D6.3, the final evaluation outcomes for the multimodal traffic management (MTM) architecture and the related models and tools proposed by the ORCHESTRA project were presented and discussed. The evaluation was performed using the assessment framework described in deliverable D6.1, and comprised qualitative evaluation, effect and impact evaluation, and process evaluation.

The evaluation was performed along 12 Key Performance Areas (KPAs), covering all important aspects of MTM, in particular:

1. Policies, governance and regulations
2. Data governance and sharing
3. Digital infrastructure
4. Technological solutions (functionality)
5. Stakeholders’ acceptance of traffic management operations and autonomy
6. Operational practices and decision making for traffic orchestrators
7. Business policy aspects
8. Organisational aspects
9. Traffic management effects
10. Economic impact (increased cost-efficiency)
11. Environmental impact
12. Transport and mobility impact

For each KPA, a set of relevant research questions was identified. For those research questions, which could be answered quantitatively, corresponding quantitative key performance indicators (KPI) were identified, which were assessed using simulated or real data.

Two living labs were set up in ORCHESTRA: Norwegian Living Lab (Herøya) and Italian Living Lab (Malpensa), which facilitated the project assessment. In the context of these living labs, multiple scenarios were defined directly related to the research questions from the assessment framework to evaluate the developed MTM architecture and the effects of its measures.

To quatify such effects, a set of dedicated KPIs was identified for each Living Lab. Examples of KPIs for the Norwegian living lab include distribution of truck travel time, truck arrival rate, truck waiting time, CAV and utilization rate. For the Italian living lab, KPIs include are passenger waiting time, passenger delay, X-ray machine utilization rate.

Qualitative evaluation

Qualitative evaluation was done by answering qualitative research questions using literature studies, input from experts (interviews, workshops, surveys), and design science research methods. The workshops with experts associated with both Living Labs greatly facilitated and provided much of useful input for answering the qualitative research questions.

The main qualitative evaluation outcomes include:

• The white paper, a roadmap, and future scenarios for a shared vision of the 2030 and 2050 MTME were defined in the ORCHESTRA including analysis of the barriers, enablers, opportunities, acceptance, and social impact of its implementation.
• Elaboration on policies, regulations, and standards needed to implement the MTM; in particular, the need for harmonization of standards across multiple transportation modes and countries is identified as essential for enabling MTM
• The Polycentric Multimodal Architecture (PMA) has been developed, which prepares the ground for further work on and realisation of MTM
• Description of roles in MTM with their responsibilities and relations between them; a particular attention is given to the exploration of flexible ways how MTM concept could be deployed in adaptation to modes and local needs
• (Minimum) dataset has been identified that needs to be shared to enable MTM, and reflection on barriers, enablers, and concerns related to data sharing
• Value Network Diagram models were proposed to assist MTM stakeholders in identifying the needs of their contemporaries and formulate partnerships and collaborations within MTME
• A description of the requirements and functionalities for digital infrastructures to enable automation and connectivity of and within MTM
• Technological solutions to enable MTM, including transport demand management, demand capacity balancing, and traffic coordination have been proposed
• Reflection on stakeholders’ acceptance of and psycho-sociological aspects related to MTM operations and autonomy
• Discussion on new skills, knowledge and training practices required for traffic orchestrators to operate MTM

Effect and impact evaluation

Effect and impact evaluation were done by answering the quantitative research questions by using computer simulations of the scenarios defined for the Living Labs and using empirical data collected from the Living Labs.

In the table below,  we provide an overview of the main quantitative outcomes expressed by the numerical values of the corresponding KPIs. For each KPI two values are provided: the first value is obtained by simulation of a baseline scenario before introducing a MTM measure, and the second value reflects the simulated effect after the MTM measure has been introduced. For Herøya, Scenario 1 is focusing on normal demand and capacity utilization, Scenario 2 is focusing on increased demand for CAV resources, while Scenario 3 simulates a day of increased demand and sporadic disruptions. For Milano, a single scenario is developed of a particularly disruptive instance. Classification of each scenario is targeted to specific research questions as previously explained in deliverable D5.2.

Process evaluation

Process evaluation of the implementation of the ORCHESTRA’s concepts and tools in the context of the Living Labs, was done by elaborating on enablers, drivers, barriers, gaps, and other experiences related to the implementation of MTM. Again, workshops with stakeholders associated with each Living Lab were used to collect feedback and discuss the experiences and outcomes. In general, the stakeholders evaluated the ORCHESTRA concepts and tools positively, and provided useful feedback for further extension and elaboration of the proposed ORCHESTRA solutions.

The main process evaluation outcomes for the Norwegian Living Lab include:

• Gaps, challenges, enablers, and other experiences related to the implementation of MTM at Herøya Industrial Park, based on a CAV service for escorting arriving trucks to their destinations in the park
• Stakeholder evaluation of some of the tools developed in ORCHESTRA used for control of CAVs and for traffic orchestration at and around Herøya Industrial Park
• Evaluation of operational aspects of CAVs and the traffic safety of the CAV service implemented in the Norwegian Living Lab are extensively discussed

The main process evaluation outcomes for the Italian Living Lab include:

• Elaboration on the perceived benefits and concerns of the living lab’s stakeholders in relation to the implementation of the MTM concept
• Reflection on the barriers, drivers and risks for the implementation of the MTM concept, in particular with respect to data sharing, trust and liability, and automation
• Description of the MTM-related tools implemented in the living lab and their evaluation by the stakeholders

More information and details concerning all types of evaluation is provided in deliverable D6.3.  Furthermore, D6.3 served as a basis for proposing recommendations in deliverable D6.5, and summarizing main outcomes of the project in deliverable D1.5.