Success message
error message
warn message
Terminal Evaluation: Transforming the global aviation sector: Emissions Reductions from International Aviation (PIMS 5254)
Commissioning Unit: Bureau for Policy and Programme Support
Evaluation Plan: 2018-2021
Evaluation Type: Project
Completion Date: 11/2018
Unit Responsible for providing Management Response: Bureau for Policy and Programme Support
Documents Related to overall Management Response:
 
1. Recommendation:

Always include an active UNDP RTA on UNDP/GEF projects as a mentor and coach for the project throughout its lifetime, in particular for multi-agency projects such as this one, and have them thoroughly review the PRF at inception. They should also provide detailed guidance on monitoring and reporting standards with specific examples for the project.

Management Response: [Added: 2018/11/12] [Last Updated: 2021/01/19]

Indeed, it’s the role of the RTA to provide mentorship and guidance during development and implementation project cycle. In this particular case, we had a RTA based in Panama supporting the project, and was in charge of it from development to inception. However, he left UNDP and the project responsibility was taken over by HQ staff in NY.  

Key Actions:

Key Action Responsible DueDate Status Comments Documents
To be taken into account for future programming
[Added: 2018/11/29]
UNDP 2018/12 Completed
2. Recommendation:

Always perform some type of independent mid-Term Review for short-term projects, even if it is only focused on monitoring and reporting issues as this is the source of the most common pitfalls in GEF projects.

Management Response: [Added: 2018/11/12] [Last Updated: 2021/01/19]

Agree. Although MTR is not mandatory for medium size projects, we are taking from this experience the importance to engage in some type of mid-term review, especially when dealing with multi-agency global projects. This exercise should be budgeted from the beginning and included in the total budget work plan.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
To be taken into account for future programming.
[Added: 2018/11/29]
UNDP 2018/12 Completed
3. Recommendation:

Always have a full-time project manager throughout the lifetime of the GEF project. It is a risky strategy to outsource so much of the project effort without a full-time management of the outcomes.

Management Response: [Added: 2018/11/12] [Last Updated: 2021/01/19]

Partially agree. During the planning phase of the ICAO-UNDP-GEF project, the work required by the project manager was discussed with UNDP and two days of project management effort was deemed necessary. Based on the project experience that followed, it is our view that the two days given to the Project Manager was adequate for the majority of the project (the first two years) and particularly during the development of the deliverables associated with guidance material, tools and documents.   As the project progressed into the delivery of the solar-at-gate deliverable, some flexibility was given (in the final year) to allow the project manager to spend three days per week on project activities.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
To give due consideration to the amount of project management time required, considering the deliverables expected, in any potential future project.
[Added: 2018/11/29]
UNDP 2018/12 Completed History
4. Recommendation:

A risk management seminar demonstrating available tools and methods for the project staff should become standard practice at the project inception, with the result that the main project risks are identified during the seminar and better monitored during project implementation.

Management Response: [Added: 2018/11/12] [Last Updated: 2021/01/19]

Agree. UNDP has been conducting internal training to project staff on the assessment of potential environmental and social risks on regular basis and during project inception.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
To be taken into account for future programming.
[Added: 2018/11/29]
UNDP 2018/12 Completed
5. Recommendation:

There should be a central UNDP standard and standard software package for “online knowledge bases” and online “technical support platforms” as these are called for in almost every UNDP/GEF project.

Management Response: [Added: 2018/11/12] [Last Updated: 2021/01/19]

Agree

Key Actions:

Key Action Responsible DueDate Status Comments Documents
This will be explored in future programming.
[Added: 2018/11/29]
UNDP 2018/12 Completed
6. Recommendation:

It is recommended to develop future guidance to assist solar-to-gate project developers and proponents on how to best account for CO2 savings resulting from both domestic and international flights operating from electrified gates. Such guidance should account for the future policy frameworks which may impact the GHG accounting.

Management Response: [Added: 2018/11/12] [Last Updated: 2021/01/19]

ICAO disagrees with the comment. The comment requests work to be carried out on accounting of domestic aviation emissions, which is outside the mandate of ICAO. Accounting for domestic emissions is specific to individual states and corresponds to national regulatory frameworks.

Having said this, it is important to note that in terms of calculating the CO2 emissions for aviation activities, ICAO has developed a methodology for calculating the time spent at the gate by an aircraft, and the associated CO2 emissions. This can be used for calculating the CO2 emissions at the gate for international flights and this globally accepted ICAO methodology was the one applied to the ICAO-UNDP-GEF project. Please note that the ICAO methodology has also been approved as a UNFCCC Clean Development Mechanism (CDM) small scale methodology for the calculation of solar power for domestic aircraft at-gate operations.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
This will be considered in future programming, where appropriate/relevant.
[Added: 2018/11/29]
UNDP 2018/12 Completed
7. Recommendation:

All UNDP-supported GEF-financed projects should always try to highlight the business case for environmental improvement measures, not just the environmental or policy case, as that will provide a better foundation for future replication of the innovation/measure.

Management Response: [Added: 2018/11/12] [Last Updated: 2021/01/19]

Agree. UNDP-supported GEF-financed projects take into account the cost-effectiveness and financial sustainability when proposing measures. A distinction has to be made between mandatory measures, like minimum energy performance standards, that will have to be complied with (and introducing these in the most cost-effective measures) and promoting the use of renewable energy options to replace a fossil-fuel based baseline. In the latter case, it is particularly important to highlight the business case, as companies will only adopt renewable energy options if they are sustainable and financially more attractive over the lifetime of the investment.

In the case of the PV demonstration at the airport in Kingston, while subsidized by the project, is financially competitive, given current cost of conventionally produced electricity. This is demonstrated by a follow-on investment of a similar solution being considered by the company that runs the Montego Bay airport, which will go ahead without this project subsidy.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
To be taken into account for future programming.
[Added: 2018/11/29]
UNDP 2018/12 Completed
8. Recommendation:

4.2. Final Recommendations and Lessons

Though it is often stated, it needs to be repeated here: Project reporting is not the same thing as project monitoring. The project produced most of the necessary reports to UNDP management but often the indicators and components are described as “on track” or “satisfactory” in the PIRs, etc. This refers more to the project activity implementation and does not necessarily reflect monitored impacts that relate back to the project Objective. The impression is that the PMU focused on implementing the activity and then assessing the impact only at the end, which is a natural tendency, but not ideal if the project data was not collected sufficiently during the activity. 

The project design had many different activity types, and in a perfect scenario each activity type would have had its own monitoring, reporting, verification (MRV) protocol that also established key risks to be monitored. This would represent a heavy load on the project initially but would have paid dividends toward the end. Such a task should have been given greater emphasis in the ProDoc and in the initial project planning to provide a proper platform for project evaluation. The ProDoc PRF is not detailed enough to have provided all the necessary indicators and MRV that the project needed. This is common for most UNDP/GEF projects but sometimes this is recognized and further developed early in the project. Otherwise, most of the main lessons bear repeating in this report to reach a wider audience. In regards to what the project did well, the project was successful in combining knowledge tools showing best practice examples of sustainable aviation practices, including two pilot projects that demonstrate grant financing and private sector financing for the renewable energy (solar PV) component. The project did so by providing good examples of :

- Adaptive management for Component 4: a re-allocation of the project funding to finance 2 separate pilot projects at NMIA and SIA. - Well-structured and precise Terms of Reference for the pilot projects at the NMIA airport facility and for Outcome 1. This allowed the selection of the main contractor (Solera and McKinsey) to proceed smoothly and for the deliverables to be easily checked and approved by all stakeholders during the implementation. - Creatively going beyond the original mandate, as demonstrated by the MAC Curve Tool development for Outcome 1. - Early engagement of the relevant local stakeholders, as shown by the early outreach efforts to secure the pilot project support and approvals by the Jamaican authorities. - An interdisciplinary team of professionals tasked with delivering the project results. Without a doubt a critical factor for the success of this project were some of the skilled and dedicated individuals at ICAO. While only working on this project on a part-time basis they managed to complete the deliverables on budget and almost on time, and with a general high level of excellence. This is a great example of leveraging co-financing resources from within the host organization that would not normally be available if the team was only funded from the project budget.

Management Response: [Added: 2021/01/12] [Last Updated: 2021/01/19]

Key Actions:

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org