Terminal Evaluation: Transforming the global aviation sector: Emissions Reductions from International Aviation (PIMS 5254)

Report Cover Image

Share

Document Type Language Size Status Downloads
Download document Evaluation Consultant ICAO_UNDP_GEF.pdf tor English 328.42 KB Posted 501
Download document ICAO UNDP Global Aviation Sector TE Final Versionv2.docx report English 6583.94 KB Posted 561
Title Terminal Evaluation: Transforming the global aviation sector: Emissions Reductions from International Aviation (PIMS 5254)
Atlas Project Number:
Evaluation Plan: 2018-2021, Bureau for Policy and Programme Support
Evaluation Type: Final Project
Status: Completed
Completion Date: 11/2018
Planned End Date: 07/2018
Management Response: Yes
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 2.5.1 Solutions developed, financed and applied at scale for energy efficiency and transformation to clean energy and zero-carbon development, for poverty eradication and structural transformation
Evaluation Budget(US $): 29,000
Source of Funding: GEF
Evaluation Expenditure(US $): 29,000
Joint Programme: No
Joint Evaluation: Yes
  • Joint with UN Agencies
  • Joint with ICAO
Evaluation Team members:
Name Title Email Nationality
Jesse Uzzell
GEF Evaluation: Yes
GEF Project Title: Transforming the global aviation sector: Emissions Reductions from International Aviation
Evaluation Type: Terminal Evaluation
Focal Area: Biodiversity
Project Type: MSP
GEF Phase: GEF-5
GEF Project ID: 5450
PIMS Number: 5254
Key Stakeholders:
Countries: GLOBAL
Lessons
Findings
1.

3. FINDINGS

3.1.Project Design and Formulation

Overall, the project design is adequate as it is based on a review of the institutional, political and the technical context. The ProDoc identified opportunities available for action at two levels, at the international level using ICAO as the platform for a top-down initiatives related to education, and bottom-up approaches at the local/national level through the pilot projects in Jamaica and the capacity building workshops for the stakeholders in the SIDs. The components of the project were well conceived and based on measures that were identified and supported by ICAO to move the aviation industry to a more sustainable path and reduce GHG emissions in the sector. As SIDs are isolated and rely more on air transportation for their economies, particularly for tourism, focusing the project on SIDs was a smart strategy. By using ICAO as the implementing partner the project also leveraged ICAO’s networks with the SIDs and developing States which provides a much larger impact with the funding and project staffing available. As mentioned previously, the project was also designed to complement the ICAO-EU project and there are synergies and overlaps that contributed to both projects meeting their goals, e.g. SAF feasibility studies, future implementation of the MAC curve tool, solar-to-gate projects, etc. From the technical side, the project design provides a clear framework for the incremental measures and how these relate to the current (baseline) activities at ICAO as shown in Figure 1. The project design is also very prescriptive in how to move forward with pilot activities and even includes a letter of support from the Government of Jamaica which provides access to land (land grant) for a future project. Overall it can be said that the Project Design, as detailed in the ProDoc, was well thought out and structured, and this is evidenced by the PMUs ability to implement the project activities on time and on budget with only modest challenges which are typical of most projects.


Tag: Coherence Relevance Policies & Procedures Programme/Project Design Theory of Change Country Government

2.

3.2.The Project Results Framework (PRF) (continuation) 

A review of the project risk matrix shown in Table 3 shows that many major risks where properly identified during the project screening and development phases. The PMU did not follow-up adequately regarding Risk 5 (GHG ER verification) and there is little evidence that a systematic effort was put in place in regards to collecting information to justify the indirect emissions reductions from the project activities. However, the monitoring system put in place for the real-time monitoring of the electricity production from the solar to gate project (and thus the direct ER’s) is state-of-the-art and allows remote monitoring from ICAO headquarters.In hindsight, the risk matrix mainly focused on macro factors and missed institutional risks related to differences in e.g. procurement procedures between UNDP and ICAO that went unnoticed until they ultimately created some unexpected delays for the project. Otherwise generally speaking the risk matrix is logical and robust. These issues related to the PRF and monitoring are described in more detail in the next sections. In general it is the Evaluator’s opinion that the original PRF was not as well designed as the project itself and would be rated as “Moderately Unsatisfactory” if such a rating was done as part of the TE. However, it is the responsibility of the PMU to notice and rectify this as much as possible during the project implementation. 


Tag: Efficiency Human and Financial resources Results-Based Management Risk Management Theory of Change

3.

3.3.Project Implementation & Management Arrangements (continuation)

It is very unusual to not have any full-time dedicated staff for a GEF project. During the interviews this was consistently raised as one of the challenges this project faced, and the decision to hire a part-time PM might have seemed like a good idea at the time of the project formulation. The rationale for this was that the bulk of the work would be done by sub-contractors (consultants) and that the PM role was more about oversight and management of the deliverables, which would not justify a full-time position at ICAO. Even though a suitable candidate was found for the PM role and one of the reasons she applied was that it was part-time, it was by all accounts a situation where she often worked on the project outside of the 2 days a week and often from home. This is also evidenced by the increase in man-days for the PM from 2 days a week to 3 days a week in 2018.Normally having a project with so many moving parts without any dedicated full-time staff would be a recipe for disaster. It is a testament to the professionalism and conscientiousness of the project team at ICAO that the project performed as well as it did in delivering so many results, considering that the list of staff in Table 3 (except the PM) all had other full-time duties within the Environmental Department. Some interviewees also attributed the success to the strong team feeling within the department, the strong leadership from the Department Deputy Director, and the PM’s personality and ability to get others to “buy-in” to the project. As one staff member described it, “You took pride in working on the project, and wanted to be a part of it because it had very positive and tangible results (the pilot solar-to-gate project) compared to the typical development of reports or guidance documents etc. So, it was an exciting project for the department and you wanted to be part of that team.” It should also be pointed out that the primary contractors/consultants for the project were mostly highly experienced professionals who delivered good results for the project within the budget limitations. The feedback from the ICAO staff was overwhelmingly positive in this regard, though there were typical issues related to revisions, reference to ToRs, etc. before the tasks could be considered completed. That is a positive consequence of the review and oversight role of the PMU and is normal for any project.One surprising issue that was raised during the staff interviews, and also in the official project documents (PIR, PSC minutes, etc.), was the difficulty the PMU faced with aligning the procurement procedures between UNDP and ICAO. This was mentioned by everyone as a major challenge the project faced during implementation and it caused major delays. Essentially the problem boils down to two institutions with very well-developed guidelines and procedures having a policy conflict that created a deadlock: UNDP could not disburse funds until an assignment was completed, and ICAO could not issue a tender for an assignment without the necessary funds being in the ICAO accounts to back the tender. This procurement issue effectively froze the project progress in 2016 until the right people of the appropriate high-level in the two bureaucracies took the decisions to use common sense to break through the resulting deadlock and granted an exception to the procedures.


Tag: Effectiveness Efficiency Human and Financial resources Project and Programme management

4.

3.5.Monitoring and Evaluation: design at entry and implementation

As discussed earlier the monitoring indicators in the ProDoc, particularly for Objective 1 and Outcome 2, did not provide a good relationship between the project’s environmental impact and progress with the project’s activities. Prime examples are - Objective 1.3 Target: Indirect Emissions: Additional 1,000,000 tons of CO2 reduced in developing countries and SIDS over 20 year - Outcome 2.1 Target: Policies and regulations guidelines are adopted and enforced to stimulate low emission aviation investments in developing States and SIDS.

Within the ProDoc the onus of the M&E design is placed on the future PMU and the project participants. In Annex 5 Monitoring Framework and Evaluation it clearly states: The Inception Workshop should address a number of key issues including: a) Assist all partners to fully understand and take ownership of the project. Detail the roles, support services and complementary responsibilities of UNDP GEF staff vis à vis the project team. Discuss the roles, functions, and responsibilities within the project's decision-making structures, including reporting and communication lines, and conflict resolution mechanisms. The Terms of Reference for project staff will be discussed again, as needed. b) Based on the project results framework and the relevant GEF Tracking Tool if appropriate, finalize the first annual work plan. Review and agree on the indicators, targets and their means of verification, and recheck assumptions and risks. c) Provide a detailed overview of reporting, monitoring and evaluation requirements. The M&E work plan and budget should be agreed and scheduled. 


Tag: Effectiveness Efficiency Monitoring and Evaluation Programme/Project Design Project and Programme management Results-Based Management

5.

3.6.Partnership Arrangements

The engagement of key stakeholders was primarily done through participation as full members or observers at the PSC and regular communication by email and phone calls during the implementation of the projects in Jamaica. The primary partners in the project besides UNDP were - Key government stakeholders from the Jamaican federal government, and from the local administrations in the pilot cities of Kingston and Montego Bay. - Technical experts and contractors in the USA, Canada, and Jamaica - A few private stakeholders involved with the airports. There are few records of project meetings outside of the 1 PSC meeting in 2018 and the annual PIR. The project produced excellent minutes from the mission to Jamaica that took place in early 2016. The level of engagement was with partners was generally appropriate though the project might have benefited from a more active Steering Committee that met more often in 2017 or more regular input from a UNDP RTA. 


Tag: Effectiveness Efficiency Business Model Implementation Modality Partnership Strategic Positioning

6.

3.8.Finance and co-finance

The project finances were reviewed during the TE and the information made available to the evaluator only covers the period through June 30, 2018, as ICAO has, per agreement, until December 31, 2018 to finalize the project payments and disbursements. Table 5 and Figure 4 show the budget disbursements during the project life. The project activities effectively started in 2016 and with the extension ended in July 31, 2018. This shift in start date and also the shift of the payments for the deliverables in Components 1 and 4 to 2018 explains the large difference between the ProDoc budget and the real disbursements by the project.The blue line in Figure 4 has a larger amount than the project budget. According to UNDP staff, the “… blue line line indicates the yearly budget entered into Atlas, which is not limited to the total grant amount. That line is more is more informative, the black and green are showing the real spending against the grant amount.”. The distribution distribution of resources among the four components of the project is presented in Table 6.

Within the components the largest budget change occurred within Component 3. The cost of the SIDS seminars was under-budgeted and approx. $40,000 was transferred from the development budget of the Knowledge Platform to travel costs and other costs associated with the 2 SIDS seminars. These budget changes were approved by UNDP in consultation with ICAO.The planned project co-financing was confirmed by the PMU and evidence was provided during the TE. The bulk of the co-financing was provided by the ICAO-EU project that collaborated actively with this one. 


Tag: Efficiency Global Environment Facility fund Human and Financial resources

7.

3.10. Project Results

Each Component/Outcome and the main Objective are discussed in detail in the following sections and the justification for each rating is provided. Overall the project is rated as Satisfactory in accordance with the UNDP/GEF definition that “the level of outcomes achieved more or less as expected and/or there were moderate short comings” in the project. These outcomes and any shortcomings are discussed in more detail in the following sections. 

3.11. Outcome 1: More developing States and SIDS identify low emissions measures. Table 10 below shows the PRF indicators agreed for the project for Outcome 1. The last column shows the status at project end. Component 1 had 2 major main activities in mind to development guidance on different measures to reduce emissions and to disseminate the results. The project team took this task to an entirely higher level by deciding to develop an online platform that estimates for users the Marginal Abatement Cost curves (MAC) for different reduction options and even a basket of reduction options. The team hired McKinsey, famous for developing MAC curves for GHG reduction actions globally, and had them apply their expertise to the aviation sector. The result is a comprehensive and unique analysis of the economic costs and environmental benefits of a basket of 20 distinct measures.


Tag: Emission Reduction Environment Policy Green Climate Green Economy Effectiveness Donor relations Partnership

8.

3.12. Outcome 2: Instruments are available to support the development of a legal and regulatory environment that facilitates the financing of feasible low emissions aviation measures in States. The primary deliverables for Outcome 2 were 4 reports developed by two experts contracted to the project. These reports were developed in collaboration with the project team and consist of the following: 1. Regulatory and Organizational Framework to Address Aviation Emissions 2. Renewable Energy for Aviation: Practical Applications to Achieve Carbon Reductions and Cost Savings 3. Sustainable Aviation Fuels Guide 4. Financing Aviation Emission Reductions

The primary audience for these reports are policy-makers and managers in Government Agencies and as such the reports are meant to be more of a first induction to the topics whereby the interested party may seek out more specific knowledge elsewhere. All the reports strike a good balance between teaching the basic knowledge and giving practical examples of cases in the aviation sector around the world. It is fair to say that the reports are useful and of high quality. The Sustainable Aviation Fuels Guide is a particularly useful reference which presents the different SAF options, processes and feedstocks in a nicely organized fashion for the reader (Figure 5 gives an example). The Financing Aviation Emission Reductions report was also well put together and follows a pedagogic path in its approach to taking the reader from basic project finance concepts to an overview of multi-lateral and bi-lateral financing initiatives. The main critique of the reports, and the improvement that this Evaluator would propose, would be to provide a stronger emphasis on how to build “the business case” for e.g. renewable energy projects. All of the different elements are touched upon but none of the case studies delve into the economics of the projects and how they were funded. It would be useful to show examples that could discuss (even theoretically) important financial metrics such as pay-back period, internal rate of return (IRR), net present value (NPV), and return on investment (ROI). In addition, including a basic Project Idea Note (PIN) template1 (or something similar) that is used by several multi-lateral banks would highlight the main items that investors or financing agencies want to analyse. The ProDoc actually provides an example on page 21 of framing the basic business case for a solar panel project at airports. The Financing Aviation Emission Reductions and/or the Renewable Energy for Aviation reports would have benefited from such a section, or project case study examples, on how to prepare the business case for investors, banks, and other stakeholders.


Tag: Emission Reduction Effectiveness Policies & Procedures

9.

3.13. Outcome 3: The cost and resources expended by developing States and SIDS to have access to updated technical information on low emission mitigation measures is greatly minimized. Outcome 3 is described in ambitious terms in the ProDoc and never uses the term website or webpage but rather “technical platform” and “portal”. It is worth repeating the description here: The technical platform will bring together all technical information essential to implement emissions reduction measures such as standards, tools, guidance documents, guidelines, information on indicative costs, benefits, training material, in a SharePoint-like portal accessible from anywhere by States. The information will be public domain and will be collected from different sources including: ICAO, national governments, academia, vendors, business associations, etc. The availability of such a platform will also significantly reduce the time spent, costs and other overheads of developing States to collate information, leading to incremental implementation of emissions reduction measures.

In the last PIR from 2017 the primary description of the platform is that it will be part of the larger ICAO website and that it would be ready by the end of 2017: Development of the platform is implemented in close cooperation with Business Technology and Services Section (BTS) of ICAO. The platform will be ready by the end of 2017 to serve as a stage for dissemination of the guidance documents developed within the project, as well as technical materials developed by other environmental aviation organizations to support environmental protection in aviation. The technical platform will also showcase in detail the best practices of the pilot project in Jamaica to encourage its replicability in other SIDs. The advantage of the integrated environmental portal is that it will share a wide-range of information on low emission mitigation measures (e.g. tools, standards, guidance document, guidelines, recommendations) from various sources, including ICAO, national governments, academia, business association and private sector. Brought together on one single technical platform, more complete and comparable information will be easily accessible to relevant aviation stakeholders, including aviation authorities, airports, aviation industry, civil society of Member States, particularly developing States and SIDS. 


Tag: Emission Reduction Effectiveness Efficiency Human and Financial resources

10.

3.13. Outcome 3: The cost and resources expended by developing States and SIDS to have access to updated technical information on low emission mitigation measures is greatly minimized.(continuation)

This was not a good result and prompted the discussion about future maintenance needs of the KSP. Since little content is saved locally in the KSP the success or failure of the KSP information depends on the external links being maintained by other parties (and the Boeing database) and presumably upon a regular updating of the hyperlinks maintained in the KSP database. It should also be noted that this concern about the challenge for ICAO of maintaining the KSP was raised by the UNDP Representative during the PSC meeting in March, 2018. As part of the Action Plan for improvement ICAO has updated the KSP and implemented an internal process for updates and quality control of the KSP information. In addition the Questions and Answers section is moderated and easier to access. The updated version is an improvement and further development will provide a resource for the target audience, namely air transport managers and administrators in SIDS and LDCs.There are a total of eight (8) videos collected on the KSP landing page/platform. ICAO plans to add a greater amount of videos in the future and some that more focused on the project results in Jamaica. ICAO has not provided statistics on the KSP usage at this time as called for in the PRF target indicator 3.3. The “Ask our Expert Community” section is currently not being actively used but this was partially due to some of aforementioned access issues


Tag: Effectiveness Efficiency Impact Technology Capacity Building

11.

3.14. Component 4: The feasibility of implementing low emissions aviation measures is demonstrated through the pilot project and the lessons learnt are made available to facilitate the replication in developing States and SIDS. The Government of Jamaica supported the project at the ProDoc stage in 2014 as evidenced by the Endorsement letter. When the project started one of the first priorities of the ICAO staff was to re-engage the Jamaican Government and Agencies in the project and the ICAO Project Manager and Environmental Department Head made a mission in late September, 2016 to Kingston and Montego Bay to discuss a solar PV project at one of the airports. 

After some analysis and discussions over the following year it was decided that the Norman Manley International Airport (NMIA) in Kingston, which is publicly owned, would be the recipient of the pilot solarto-gate project. One of the early changes in the project design was the reduction of the size of the solar PV system from 398KW to 100KW to accommodate the new PV location in the parking lot. The original planned location was rejected due to glare concerns once a new air traffic control tower was erected. Another deviation from the original planning and budget relates to the gate electrification equipment. It was assumed in the ProDoc stage that the PCA and GPU would be at the airport gate and that the project budget would only be used for the PV equipment and installation. Subsequently, in a good example of adaptive management the PMU broadened the scope and ambitions of Component 4 by shifting budget resources and also engaging the private airport in Montego Bay, Sangster International Airport. Due to the smaller than planned PV installation at NMIA the project had enough resources to purchase one set of PCAs and GPUs at both airports to electrify Gate #1 (NMIA) and Gate #(SIA). Gate electrification and air conditioning/air handling is necessary for the aircraft to not run its own APU at the gate or utilize a mobile GPU using diesel, thus potentially lowering CO2 emissions.


Tag: Emission Reduction Effectiveness Impact Communication Knowledge management

12.

3.14. Component 4: The feasibility of implementing low emissions aviation measures is demonstrated through the pilot project and the lessons learnt are made available to facilitate the replication in developing States and SIDS.(continuation)

The Solar-to-Gate system is well designed and constructed. Some design limitations relate to the positioning and angle of the solar canopy. Normally solar arrays are often positioned to face South or oriented along the East-West direction to capture maximum sunlight. Due to glare issues this solar array is oriented along the North-South direction and angled slightly to the East in order to not cause glare on the main air traffic control tower. This placement reduces the daily power output but was a necessary compromise. The system does not include batteries for energy storage and the electricity from the PV panels is connected to the main terminal power supply, not directly to Gate 1. This a better solution as it allows the continuous use of the solar PV array. According to the staff it generates approximately 425KWh electricity per day which should more than meet the future needs of Gate 1.

The project has also erected signs at the parking lot (Figure 14) so people can read more about the project and realize they are parked under a solar array; not everyone realizes it when they park their car under the “roof”. In the main terminal there is also display in which has more real-time information about the project and local weather conditions (Figure 15). Even though Gate 1 was not currently accepting flights, the solar PV array was still producing electricity for the airport which would offset electricity consumption from the grid, resulting in CO2 emission savings. In regards to Outcome 4.2, electricity monitoring data at NMIA has been reported by the project from May until mid-September, 2018. Currently 57,771 kwh of electricity has been generated by the solar at gate project. This results in an estimated CO2 savings of 51.44 tCO2 which is lower than the target indicator of 3000 tCO2 by project end. The monthly theoretical production would be around 15,000 kwh of electricity and the reported amount is in that range. In the 2017 PIR it was stated that the project estimated the savings to be 1455 tCO2 by project end. The same PIR states that the NMIA project in Kingston should reduce 1000 tons of CO2 per year. However, this number is too large and appears to be from revised ProDoc estimates. There was not clarity during the TE on how that estimate in the PIR was chosen, as estimates provided by Dr. Dickson during the TE for the annual CO2 savings from NMIA are more accurate at 176 tCO2 per year.


Tag: Emission Reduction Effectiveness Communication Knowledge management Capacity Building South-South Cooperation

13.

3.15. Project Objective

The Project Objective is the penultimate target of the GEF project: To support capacity building in developing States for implementing technical and operational measures for reducing CO2 emissions from international aviation; as measured by the project indicators. Table 1Table 14 lists the three indicators for the Project Objective that the Project should have monitored and reported during its lifetime. The Evaluator has listed the PRF target values and the reported values achieved by the project.The Project exceeded the key Objective targets in relation to capacity building and the implementation of pilot solar-to-gate demonstration projects. In particular, the project managed to catalyze the development of two solar-to-gate projects in Jamaica, going beyond its original mandate and deservedly should receive a Highly Satisfactory rating for Objective Indicator 2.

Despite these successes the project had some difficulties in justifying its key objectives of CO2 emission reductions at the time of the TE. In regards to the Direct Emission Reductions the issue relates to the key ProDoc assumption that the project would not have to fund the gate electrification equipment, and would therefore have a larger budget for the solar PV panels at the NMIA. This turned out to be an invalid assumption and the gate electrification equipment was also financed by the project at both NMIA and SIA. Due to budget and space restrictions at NMIA (the airport is located on a thin peninsula) the solar PV capacity was reduced to 100KW installed, which was about a quarter of the planned capacity in the ProDoc (398KW). Consequently, the Direct Emission Reductions that are estimated to occur from NMIA over its lifetime of 25 years are reduced from 17,500 tons of CO2 to 4,400 tons of CO2, which is significantly below the target value. Currently the project has reduced of 51.44 tCO2 through September 18, 2018. 


Tag: Effectiveness Efficiency Impact Global Environment Facility fund Institutional Strengthening Technical Support

14.

3.15. Project Objective (continuation)

The source of verification is stated to be the project monitoring reports, but beyond the PIR such reports were not developed by the PMU. The ProDoc itself gives very little guidance on what data to collect to estimate the Indirect Emissions except for the above comparison of State Action Plans and “relevant information” from industry associations.

The end result is that the only basis for justifying the Indirect Emissions is the statistic that 20 out of the 107 ICAO Member States have indicated plans in their voluntary Action Plans to implement solar energy projects at airports in the future (approximately 19% of the submitted State Action Plans). This is a laudable number but it is very unclear as to how much installed PV capacity this could result in, and it says nothingabout the other actions that could be attributed to low-emission aviation measures (Components 1 through 3). In that regard the PMU can only offer the same analysis that is done in the ProDoc without justifying the analysis with any new data derived from, e.g. a stakeholder survey or industry statistics developed during the project. This is a lost opportunity and makes the Indirect Emissions estimation impractical to verify. The indirect ER numbers proposed by the project are not unreasonable, but the causal link needs to be better supported. For example, some low-emission aviation measures could just as easily happen due to macroeconomic factors such as higher aviation fuel prices and have little to do with the project efforts.


Tag: Clean Energy Environment Policy Green Economy Effectiveness Knowledge management Project and Programme management Results-Based Management

15.

3.16. Project Relevance

Project Relevance is defined by the GEF as “The extent to which the objectives of a development intervention are consistent with beneficiaries’ requirements, country needs, global priorities and partners’ and donors’ policies.”. This project is Very Relevant to the future needs of SIDs in particular as well as developing States. It is also very relevant to the ongoing work at the international level in ICAO in regards to CORSIA and the introduction of SAFs and other measures to reduce GHG emissions in the aviation sector. 


Tag: Coherence Effectiveness Efficiency Relevance Global Environment Facility fund Operational Efficiency Partnership Policies & Procedures

16.

3.18. Sustainability

As highlighted in the ProDoc, the project’s sustainability relies in the successful implementation of all four project Outcomes: 1. The project aims to assist States to contribute to the sustainable development of international aviation. It promotes the strengthening of national capacities through the provision of appropriate material and guidance to developing States and SIDS in order to facilitate the implementation of cost effective mitigation measures that can generate a positive cash flow during their lifecycle. 2. The Outcomes of Components 1 to 3 promote a better understanding among developing States and SIDS of the feasibility of these measures, as well as their environmental and financial benefits. They also enhance awareness among States on the various financing mechanisms that are available to implement these measures, without the need for additional grant support. 3. Furthermore, the lessons learnt from the implementation of the pilot project (Component 4, including installation of PV panels at an airport), along with the associated guidelines and policy recommendations, have the potential to become a model for many developing States and SIDS. The importance of establishing the economic incentives for future measures and the financing mechanisms for them is the main message for the future sustainability of these mitigation measures in the aviation sector. The main carriers of sustainability for the project can be summarized as: o Educational (Outcomes 1 – 3) – The establishment of the Knowledge Base and online tools such as the Marginal Abatement Cost Curve model, as well as the guidance documents. o Legislative (Outcomes 2 & 3) – the project introduced legislative case studies and examples of legislative measures in the Knowledge Base. o Demonstration Projects (Outcome 4) – The pilot solar-to-gate projects in Jamaica.


Tag: Environment Policy Sustainability Communication Knowledge management Institutional Strengthening

17.

3.18. Sustainability (continuation)

3.1.Financial risks to sustainability The main financial risks relate to Outcomes 3 and 4. There is the risk that appropriate financing for the SIA solar PV installation will not be found and that the project only supports an electrified bridge system at SIA. Another long-term risk is in relation to resources within ICAO dedicated to maintaining the Knowledge Base once the project ends. Generally speaking these risks are viewed as being low to moderate. 

3.2.Socio-economic risks to sustainability The socio-economic risks to sustainability relate to macro-economic issues in the aviation and renewable energy sector. Currently there is little public opposition within the aviation sector against moving toward a more fuel efficient and sustainable path to reduce GHG emissions. ICAO, its member states and aviation stakeholders have developed a basket of measures to reduce emissions including technology and standards, sustainable alternative fuels, operational improvements and CORSIA. Aviation fuel prices are also increasing with the higher price of oil (around $70 a barrel) which gives more incentives to reduce consumption while also making SAFs more price competitive. In regards to SIDS in particular, electricity prices are typically much higher than in other countries as all the fuel needs to be imported, while prices for solar PV is continually declining due to strong competition. Today there are only regulatory, policy, and finance barriers to greater use of solar PV in SIDS and at SIDS airports where a huge fixed cost is energy. The general economics of solar PV make total sense if one can arrange the financing, and if policy makerssupport the opening up of often monopolistic energy markets in SIDS. Therefore socio-economic risks to the project long-term goals are low and the socio-economic sustainability is Likely.


Tag: Disaster Risk Reduction Environmental impact assessment Sustainability Global Environment Facility fund Risk Management Country Government Inclusive economic growth Technology

18.

3.5.Catalytic Role and Impact

Figure 16 explains the different types of catalytic role the project is evaluated against. The primary catalytic role of this project is through the demonstration projects in Jamaica and a capturing of the lessons learned from these projects for dissemination to a wider audience in the SIDs and developing States. The project also supported parallel activities (and vice versa) within the EU sponsored project “Capacity Building for CO2 Mitigation from International Aviation” which also plans to implement two solar-to-gate projects in Cameroon and Kenya; and has produced feasibility case studies for SAFs in the Dominican Republic and Trinidad and Tobago, including requests for funding. The synergies between these two projects are high by design and even the same consultant who developed the reports in Outcome 2 is working with the ICAO-EU project. With the exception of the ICAO-EU funded projects, it is not known if there are any more planned solar-to-gate projects in the SIDS or developing States, but a foundation for knowledge transfer and replication was laid by the project and the online knowledge base of reports and the TSP. The business case for solar projects in SIDS is typically strong due to the high electricity prices and the low cost of solar per installed KWh. However, sometimes other institutional or market barriers need to be overcome, and one of the goals of this project was provide knowledge on how to overcome such barriers. It is now up to people in the member states to use this knowledge and apply it for future legislation and projects. Future environmental initiatives after the project, whether public or private, should be able to obtain valuable information about solar-to-gate opportunities along with other CO2 mitigation measures which should assist them in developing the feasibility study and value case for potential projects in the aviation sector. The project ratings for Catalytic Role are given in Table 16 and are satisfactory for the four indicators. The project carried out specific training and certification for the workers at NMIA as well as the two well attended workshops in Jamaica and Fiji as discussed for Outcome 3. 


Tag: Impact Sustainability Gender Equality Gender Mainstreaming Women's Empowerment Communication Knowledge management

Recommendations
1

Always include an active UNDP RTA on UNDP/GEF projects as a mentor and coach for the project throughout its lifetime, in particular for multi-agency projects such as this one, and have them thoroughly review the PRF at inception. They should also provide detailed guidance on monitoring and reporting standards with specific examples for the project.

2

Always perform some type of independent mid-Term Review for short-term projects, even if it is only focused on monitoring and reporting issues as this is the source of the most common pitfalls in GEF projects.

3

Always have a full-time project manager throughout the lifetime of the GEF project. It is a risky strategy to outsource so much of the project effort without a full-time management of the outcomes.

4

A risk management seminar demonstrating available tools and methods for the project staff should become standard practice at the project inception, with the result that the main project risks are identified during the seminar and better monitored during project implementation.

5

There should be a central UNDP standard and standard software package for “online knowledge bases” and online “technical support platforms” as these are called for in almost every UNDP/GEF project.

6

It is recommended to develop future guidance to assist solar-to-gate project developers and proponents on how to best account for CO2 savings resulting from both domestic and international flights operating from electrified gates. Such guidance should account for the future policy frameworks which may impact the GHG accounting.

7

All UNDP-supported GEF-financed projects should always try to highlight the business case for environmental improvement measures, not just the environmental or policy case, as that will provide a better foundation for future replication of the innovation/measure.

8

4.2. Final Recommendations and Lessons

Though it is often stated, it needs to be repeated here: Project reporting is not the same thing as project monitoring. The project produced most of the necessary reports to UNDP management but often the indicators and components are described as “on track” or “satisfactory” in the PIRs, etc. This refers more to the project activity implementation and does not necessarily reflect monitored impacts that relate back to the project Objective. The impression is that the PMU focused on implementing the activity and then assessing the impact only at the end, which is a natural tendency, but not ideal if the project data was not collected sufficiently during the activity. 

The project design had many different activity types, and in a perfect scenario each activity type would have had its own monitoring, reporting, verification (MRV) protocol that also established key risks to be monitored. This would represent a heavy load on the project initially but would have paid dividends toward the end. Such a task should have been given greater emphasis in the ProDoc and in the initial project planning to provide a proper platform for project evaluation. The ProDoc PRF is not detailed enough to have provided all the necessary indicators and MRV that the project needed. This is common for most UNDP/GEF projects but sometimes this is recognized and further developed early in the project. Otherwise, most of the main lessons bear repeating in this report to reach a wider audience. In regards to what the project did well, the project was successful in combining knowledge tools showing best practice examples of sustainable aviation practices, including two pilot projects that demonstrate grant financing and private sector financing for the renewable energy (solar PV) component. The project did so by providing good examples of :

- Adaptive management for Component 4: a re-allocation of the project funding to finance 2 separate pilot projects at NMIA and SIA. - Well-structured and precise Terms of Reference for the pilot projects at the NMIA airport facility and for Outcome 1. This allowed the selection of the main contractor (Solera and McKinsey) to proceed smoothly and for the deliverables to be easily checked and approved by all stakeholders during the implementation. - Creatively going beyond the original mandate, as demonstrated by the MAC Curve Tool development for Outcome 1. - Early engagement of the relevant local stakeholders, as shown by the early outreach efforts to secure the pilot project support and approvals by the Jamaican authorities. - An interdisciplinary team of professionals tasked with delivering the project results. Without a doubt a critical factor for the success of this project were some of the skilled and dedicated individuals at ICAO. While only working on this project on a part-time basis they managed to complete the deliverables on budget and almost on time, and with a general high level of excellence. This is a great example of leveraging co-financing resources from within the host organization that would not normally be available if the team was only funded from the project budget.

1. Recommendation:

Always include an active UNDP RTA on UNDP/GEF projects as a mentor and coach for the project throughout its lifetime, in particular for multi-agency projects such as this one, and have them thoroughly review the PRF at inception. They should also provide detailed guidance on monitoring and reporting standards with specific examples for the project.

Management Response: [Added: 2018/11/12] [Last Updated: 2021/01/19]

Indeed, it’s the role of the RTA to provide mentorship and guidance during development and implementation project cycle. In this particular case, we had a RTA based in Panama supporting the project, and was in charge of it from development to inception. However, he left UNDP and the project responsibility was taken over by HQ staff in NY.  

Key Actions:

Key Action Responsible DueDate Status Comments Documents
To be taken into account for future programming
[Added: 2018/11/29]
UNDP 2018/12 Completed
2. Recommendation:

Always perform some type of independent mid-Term Review for short-term projects, even if it is only focused on monitoring and reporting issues as this is the source of the most common pitfalls in GEF projects.

Management Response: [Added: 2018/11/12] [Last Updated: 2021/01/19]

Agree. Although MTR is not mandatory for medium size projects, we are taking from this experience the importance to engage in some type of mid-term review, especially when dealing with multi-agency global projects. This exercise should be budgeted from the beginning and included in the total budget work plan.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
To be taken into account for future programming.
[Added: 2018/11/29]
UNDP 2018/12 Completed
3. Recommendation:

Always have a full-time project manager throughout the lifetime of the GEF project. It is a risky strategy to outsource so much of the project effort without a full-time management of the outcomes.

Management Response: [Added: 2018/11/12] [Last Updated: 2021/01/19]

Partially agree. During the planning phase of the ICAO-UNDP-GEF project, the work required by the project manager was discussed with UNDP and two days of project management effort was deemed necessary. Based on the project experience that followed, it is our view that the two days given to the Project Manager was adequate for the majority of the project (the first two years) and particularly during the development of the deliverables associated with guidance material, tools and documents.   As the project progressed into the delivery of the solar-at-gate deliverable, some flexibility was given (in the final year) to allow the project manager to spend three days per week on project activities.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
To give due consideration to the amount of project management time required, considering the deliverables expected, in any potential future project.
[Added: 2018/11/29]
UNDP 2018/12 Completed History
4. Recommendation:

A risk management seminar demonstrating available tools and methods for the project staff should become standard practice at the project inception, with the result that the main project risks are identified during the seminar and better monitored during project implementation.

Management Response: [Added: 2018/11/12] [Last Updated: 2021/01/19]

Agree. UNDP has been conducting internal training to project staff on the assessment of potential environmental and social risks on regular basis and during project inception.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
To be taken into account for future programming.
[Added: 2018/11/29]
UNDP 2018/12 Completed
5. Recommendation:

There should be a central UNDP standard and standard software package for “online knowledge bases” and online “technical support platforms” as these are called for in almost every UNDP/GEF project.

Management Response: [Added: 2018/11/12] [Last Updated: 2021/01/19]

Agree

Key Actions:

Key Action Responsible DueDate Status Comments Documents
This will be explored in future programming.
[Added: 2018/11/29]
UNDP 2018/12 Completed
6. Recommendation:

It is recommended to develop future guidance to assist solar-to-gate project developers and proponents on how to best account for CO2 savings resulting from both domestic and international flights operating from electrified gates. Such guidance should account for the future policy frameworks which may impact the GHG accounting.

Management Response: [Added: 2018/11/12] [Last Updated: 2021/01/19]

ICAO disagrees with the comment. The comment requests work to be carried out on accounting of domestic aviation emissions, which is outside the mandate of ICAO. Accounting for domestic emissions is specific to individual states and corresponds to national regulatory frameworks.

Having said this, it is important to note that in terms of calculating the CO2 emissions for aviation activities, ICAO has developed a methodology for calculating the time spent at the gate by an aircraft, and the associated CO2 emissions. This can be used for calculating the CO2 emissions at the gate for international flights and this globally accepted ICAO methodology was the one applied to the ICAO-UNDP-GEF project. Please note that the ICAO methodology has also been approved as a UNFCCC Clean Development Mechanism (CDM) small scale methodology for the calculation of solar power for domestic aircraft at-gate operations.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
This will be considered in future programming, where appropriate/relevant.
[Added: 2018/11/29]
UNDP 2018/12 Completed
7. Recommendation:

All UNDP-supported GEF-financed projects should always try to highlight the business case for environmental improvement measures, not just the environmental or policy case, as that will provide a better foundation for future replication of the innovation/measure.

Management Response: [Added: 2018/11/12] [Last Updated: 2021/01/19]

Agree. UNDP-supported GEF-financed projects take into account the cost-effectiveness and financial sustainability when proposing measures. A distinction has to be made between mandatory measures, like minimum energy performance standards, that will have to be complied with (and introducing these in the most cost-effective measures) and promoting the use of renewable energy options to replace a fossil-fuel based baseline. In the latter case, it is particularly important to highlight the business case, as companies will only adopt renewable energy options if they are sustainable and financially more attractive over the lifetime of the investment.

In the case of the PV demonstration at the airport in Kingston, while subsidized by the project, is financially competitive, given current cost of conventionally produced electricity. This is demonstrated by a follow-on investment of a similar solution being considered by the company that runs the Montego Bay airport, which will go ahead without this project subsidy.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
To be taken into account for future programming.
[Added: 2018/11/29]
UNDP 2018/12 Completed
8. Recommendation:

4.2. Final Recommendations and Lessons

Though it is often stated, it needs to be repeated here: Project reporting is not the same thing as project monitoring. The project produced most of the necessary reports to UNDP management but often the indicators and components are described as “on track” or “satisfactory” in the PIRs, etc. This refers more to the project activity implementation and does not necessarily reflect monitored impacts that relate back to the project Objective. The impression is that the PMU focused on implementing the activity and then assessing the impact only at the end, which is a natural tendency, but not ideal if the project data was not collected sufficiently during the activity. 

The project design had many different activity types, and in a perfect scenario each activity type would have had its own monitoring, reporting, verification (MRV) protocol that also established key risks to be monitored. This would represent a heavy load on the project initially but would have paid dividends toward the end. Such a task should have been given greater emphasis in the ProDoc and in the initial project planning to provide a proper platform for project evaluation. The ProDoc PRF is not detailed enough to have provided all the necessary indicators and MRV that the project needed. This is common for most UNDP/GEF projects but sometimes this is recognized and further developed early in the project. Otherwise, most of the main lessons bear repeating in this report to reach a wider audience. In regards to what the project did well, the project was successful in combining knowledge tools showing best practice examples of sustainable aviation practices, including two pilot projects that demonstrate grant financing and private sector financing for the renewable energy (solar PV) component. The project did so by providing good examples of :

- Adaptive management for Component 4: a re-allocation of the project funding to finance 2 separate pilot projects at NMIA and SIA. - Well-structured and precise Terms of Reference for the pilot projects at the NMIA airport facility and for Outcome 1. This allowed the selection of the main contractor (Solera and McKinsey) to proceed smoothly and for the deliverables to be easily checked and approved by all stakeholders during the implementation. - Creatively going beyond the original mandate, as demonstrated by the MAC Curve Tool development for Outcome 1. - Early engagement of the relevant local stakeholders, as shown by the early outreach efforts to secure the pilot project support and approvals by the Jamaican authorities. - An interdisciplinary team of professionals tasked with delivering the project results. Without a doubt a critical factor for the success of this project were some of the skilled and dedicated individuals at ICAO. While only working on this project on a part-time basis they managed to complete the deliverables on budget and almost on time, and with a general high level of excellence. This is a great example of leveraging co-financing resources from within the host organization that would not normally be available if the team was only funded from the project budget.

Management Response: [Added: 2021/01/12] [Last Updated: 2021/01/19]

Key Actions:

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org