Terminal Evaluation of Pacific Adaptation to Climate Change Projects (PACC and PACC+)

Report Cover Image
Evaluation Plan:
2013-2017, Samoa
Evaluation Type:
Final Project
Planned End Date:
05/2015
Completion Date:
05/2015
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
57,130

Share

Document Type Language Size Status Downloads
Download document Final PACC and PACC+ TE.pdf report English 1659.59 KB Posted 2002
Download document TE Report Audit Trail for PACC and PACC+.pdf related-document English 283.58 KB Posted 1443
Download document PACC and PACC+ Management Response.pdf related-document English 1062.36 KB Posted 1345
Download document PACC TE Report_FINAL.rar tor English 2007.27 KB Posted 1265
Title Terminal Evaluation of Pacific Adaptation to Climate Change Projects (PACC and PACC+)
Atlas Project Number: 00063283
Evaluation Plan: 2013-2017, Samoa
Evaluation Type: Final Project
Status: Completed
Completion Date: 05/2015
Planned End Date: 05/2015
Management Response: Yes
Focus Area:
  • 1. Others
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 1.4. Scaled up action on climate change adaptation and mitigation across sectors which is funded and implemented
Evaluation Budget(US $): 57,130
Source of Funding:
Joint Programme: No
Joint Evaluation: Yes
Evaluation Team members:
Name Title Nationality
Shm Fakhruddin Mr
Shin Furuno Mr
Vincent Lefebvre Mr
GEF Evaluation: Yes
GEF Project Title:
Evaluation Type: Terminal Evaluation
Focal Area: Climate Change
Project Type: FSP
GEF Phase: GEF-4
GEF Project ID: 3101
PIMS Number: 2162
Key Stakeholders: SPREP, Gov. 14 PICTs, Gov. Australia
Countries: SAMOA
Lessons
Findings
Recommendations
1 Robust Vulnerability and Adaption (V&A) and Cost Benefit Analysis (CBA) assessments are essential before design, prioritization and selection of adaptation measures.
2 There is a need to look beyond the specific adaptation measure and consider the non-climate drivers behind exposure, vulnerability and risk; such as, alternative water sources and efficient water management, which are essential for each of the islands. The project could also learn from the ?Ridge-to-Reef? concept, which is an integrated approach for coastal, water and food security management. Overall, a more holistic approach is needed and while the rationale behind the 3 priority sectors (food security, costal management and water) was valid for simplicity reasons, a wider approach is needed. ?In Federated States of Micronesia and Samoa ? inappropriate coastal development /protection / drainage can increase vulnerability to inland flooding and exacerbate coastal erosion ?In Fiji, increasing and more intense inundation events were approached with drainage solution and there was no analysis upstream ?In Cook Islands and Vanuatu, wharf designs need to take into consideration local currents, user needs and conditions or else they can increase risks ?In Tonga, the second phase of the project protected the coastal areas by constructing 1-meter high dykes. These dykes have insufficient drainage facilities and could lead to flooding in the community due to heavy rainfall
3 The documentation of technical design modifications, farming methods and guidelines should be improved to inform better sector planning and implementation; adequate documentation should be completed for each country.
4 Local on-the-job training and employment should be prioritized where appropriate ? increasing community involvement and ownership (water resources and coastal management).
5 Communities need more time and direction to incorporate the benefits of new technologies. Such technologies require continuous support through institutions or access to trained experts. ?In Nauru, most Solar Water Purifiers (SWP) are not operational because of the lack of in-country expertise. Differences in success of the SWP in Nauru and in the Marshall Islands, in which the initiative was a success, indicated the importance of targeted technical capacity building for installation and maintenance professionals.
6 Analysis of the local policy and institutional environment must be conducted prior to implementation to identify opportunities and build high-level support for policy mainstreaming.
7 Institutional coordination should be supported for integrated adaptation actions. This will also ensure active cross-agency oversight of implementation both locally and at national level. Comparative advantages of regional organizations and technical capacity gaps should be identified at the outset to enhance partnerships for delivery (regional/national).
8 National accountability of regional projects needs to be improved and the financial processes need to be more simple and aligned with national systems. More attention should be paid to detailed project planning, tracking and monitoring to allow for more timely and responsive changes for delivery arrangements.
9 Communications/knowledge management should be adequately resourced at regional and national levels to ensure effective delivery of communications strategies.
10 SPREP should focus on improving (1) the timeliness of delivery and adequate sourcing of technical expertise to provide continuous support of technical guidelines (2) prioritizing activities, and (3) sensitizing the government to the importance of climate change adaptation.
11 Close engagement is required with state and national policy processes and local councils so that legislative changes incorporate CCA and DRR
12 Demonstration designs based on cost benefit analysis and future climate projections should be integrated into adaptation measures (Federated States of Micronesia). Combination of different demonstration measures should be considered for for added impact (Fiji and Solomon Islands)
13 Future interventions that support policy drafting must have a strong component (mainly technical assistance) at policy level to enable the Ministries of Planning / Finance to enforce such policies through advice to and mentoring of Ministries and overall budget allocation. One endeavor of PACC in Fiji was the drafting of the Climate Change policy that was later on endorsed by Parliament; it is however unclear whether the policy is actually being implemented by all beneficiary ministries although a Climate Change Division is supervising the process;
14 Future interventions should adopt an inclusive consultation process: the beneficiaries must be an integral part of the formulation stage taking into account their specificities and viewpoints on the best approach to adopt when responding to an issue. Under PACC Fiji for example, there is a need for better communication between Government and farmers? representatives on what can be realistically achieved; a final workshop / community meeting should be organized. The drainage was initially the only solution proposed by Government to respond to community flooding as a strategy to enhance climate change adaptation; however over time, Government came up with an alternative less costly and more relevant: water resilient planting material; other adaptive solutions to enhance their livelihoods could also have been considered (e.g. marine resources use).
1. Recommendation: Robust Vulnerability and Adaption (V&A) and Cost Benefit Analysis (CBA) assessments are essential before design, prioritization and selection of adaptation measures.
Management Response: [Added: 2015/06/14]

We agree with the recommendations. The recommendation is taken on board as a lesson for future projects. It is important to note that the V&A assessment has been an essential tool of the project that has been applied at the situation, problem and solution analysis phases of the project, following inception. Lessons from the application of the CBA tool half way through the project (during design and implementation phases) has also suggested the same. This is captured in PACC Technical Reports 1 and 2. Capacity building (including trainings on the use and application) and active participation of partners in all stages of the project is further recommended.

Key Actions:

2. Recommendation: There is a need to look beyond the specific adaptation measure and consider the non-climate drivers behind exposure, vulnerability and risk; such as, alternative water sources and efficient water management, which are essential for each of the islands. The project could also learn from the ?Ridge-to-Reef? concept, which is an integrated approach for coastal, water and food security management. Overall, a more holistic approach is needed and while the rationale behind the 3 priority sectors (food security, costal management and water) was valid for simplicity reasons, a wider approach is needed. ?In Federated States of Micronesia and Samoa ? inappropriate coastal development /protection / drainage can increase vulnerability to inland flooding and exacerbate coastal erosion ?In Fiji, increasing and more intense inundation events were approached with drainage solution and there was no analysis upstream ?In Cook Islands and Vanuatu, wharf designs need to take into consideration local currents, user needs and conditions or else they can increase risks ?In Tonga, the second phase of the project protected the coastal areas by constructing 1-meter high dykes. These dykes have insufficient drainage facilities and could lead to flooding in the community due to heavy rainfall
Management Response: [Added: 2015/06/14]

We accept the recommendation. The actions from the recommendation are taken on board as lessons for future projects. The PACC project was the first regional project that was designed based on few lessons and practices of individual and isolated country adaptation projects. Concepts such as ridge-to-reef, and community-to-cabinet had just come to light at its inception phase. While the objective and outcomes were impossible to be reconsidered given that the project had commenced, the outputs and activities were revised to take into consideration partnership projects, with the view that when carried out collaboratively or as co-financing partners, the overall result will be integrated and holistic. For example, the partnership efforts of PACC and the EU and GEF Integrated Water Resource Management (IWRM) projects in all 5 water focused countries except Tokelau. While PACC addressed quantity, storage and supply, IWRM addressed quality and conservation of water. In Fiji, where coastal inundation was addressed by PACC downstream, projects funded by GIZ helped promote reforestation and agroforestry practices and discouraged deforestation upstream. ? In Federated States of Micronesia ? installation of large culverts to flush out inland flooding along the strategic positions of the road sections 3 and 4 are aimed at reducing vulnerability of the coast from inland flooding. The avoidance of cutting through the mangrove forest, but going around in the circumferential direction aims to increase resilience against coastal erosion, naturally, thus maintaining the ecosystem services of the mangrove species in the area, and promoting ecosystem based adaptation. ?In Samoa - it was important to learn that not all risks can be managed and controlled by a project. ?In Fiji, the project focused on low-lying farming communities as a pilot. The scope of the project (time and resources) only catered for this area. GIZ projects were identified recently by the project to have addressed upstream watershed areas of this pilot area of Namosi Province. This is important to note for upstream changes in terms of conservation and reforestation efforts by that project ?In Cook Islands and Vanuatu, the monitoring by the government was identified by the project as an activity that needed to follow after the life of the project, to be able to collect data that suggests any risks or opportunities to improve the user needs and conditions of these infrastructures; ?In Tonga, the third phase will be a government and community activity that requires monitoring and collection of data over time. Again to identify potential risks and also opportunities to improve on the interventions. The Ministry of Environment initiated discussions for the next project to further the coastal adaptation and improve on monitoring and evaluation.

Key Actions:

3. Recommendation: The documentation of technical design modifications, farming methods and guidelines should be improved to inform better sector planning and implementation; adequate documentation should be completed for each country.
Management Response: [Added: 2015/06/14]

The knowledge management component of the project (outcome 3) ensures that products generated by the project are targeted at the right audience. These are made explicitly clear in TORs that are tendered out. The PACC Technical Report series addresses this recommendation clearly. The products we generated here captures and documents actual, suggested and proposed technical information of the project for each country. These can be viewed online at www.sprep.org/pacc/pubications. The demonstration guides published for each country is designed to inform those relevant sectors on how and why the PACC carried out the selected demonstrations; and what the lessons and practices (experiences) are relevant to inform future planning and implementation. Documentation is an ongoing process at the country level, but all information collected by the project for each country is captured and made available online via the portal and the pacific environment information network. Information on the design of projects are further captured in country logframe matrices and annual work plan which are also annexed in the country demonstration guides. This is part of knowledge management. The technical design specifications and drawings, while specific to the country demonstrations are and will continue to be shared and included where relevant. All printed publications have since been mass distributed to all partners, focal points, government ministries, and universities around the Pacific.

Key Actions:

4. Recommendation: Local on-the-job training and employment should be prioritized where appropriate ? increasing community involvement and ownership (water resources and coastal management).
Management Response: [Added: 2015/06/14]

The PACC project carried out a slew of on the job trainings, in particular , building the capacity of project coordinators (all 14 coordinators, on V&A - Socio Economic Assessments (SEA), climate science information, gender mainstreaming, gender equality, cost benefits analysis, project management, etc); technical committee members for site-specific demonstration trainings (e.g., maintenance of water supply system, and user pay tariff systems for Hihifo communities, Tonga; use and maintenance of solar water purifier technology, Nauru); the local government council members (Mangaia Islands, Cook Islands on gender and climate change; Epi Island chiefs, Vanuatu on participatory 3-Dimension model community participation; Ontong Java & Sikaiana, Solomon Islands and men and women representatives of 4 Kivori communities in Papua New Guinea on Socio Economic Assessment surveys), and all community members (all 14 PICTS on climate change, climate science awareness). The employment of community members, which differ from employing national project management employees to make up the national project management unit; differs in this context and could not be entertained or prioritized by the PACC project due to the fact that there is no provision under the project to do so.

Key Actions:

5. Recommendation: Communities need more time and direction to incorporate the benefits of new technologies. Such technologies require continuous support through institutions or access to trained experts. ?In Nauru, most Solar Water Purifiers (SWP) are not operational because of the lack of in-country expertise. Differences in success of the SWP in Nauru and in the Marshall Islands, in which the initiative was a success, indicated the importance of targeted technical capacity building for installation and maintenance professionals.
Management Response: [Added: 2015/06/14]

We acknowledge the recommendation. Lessons from Nauru shows that the technology is more applicable at the community level versus the household level. Climate and environment factors such as the rainy season or dry season and isolated or urban island environments, existing overriding technology or none does influence the preference and usability of the technologies introduced. In turn it influences the sustainability of that technology. The SWP applies very well and very much needed by remote locations that have no industrial desalination plants such as the Marshall Islands. Further, that communities will need to be engaged throughout the life of the project, using different tools (provision of information, community champions, on the job training using technology manuals - for operations and maintenance); practical knowledge and skills building to encourage ownership and provide confidence in new technology. The introduced technologies will also be sustaining in its applications by training government staff (including government utility corporations) first and foremost before heading out into communities.

Key Actions:

6. Recommendation: Analysis of the local policy and institutional environment must be conducted prior to implementation to identify opportunities and build high-level support for policy mainstreaming.
Management Response: [Added: 2015/06/14]

We acknowledge the recommendation. The analysis of local policy and institutional environment was in fact conducted during the preparatory phase in 2006-2008. We find that an analysis of the mainstreaming landscape should be an ongoing activity to engage those involved at the policy and decision making level. It is important to note that both outcome 1 and 2 (mainstreaming and demonstration) were carried out in parallel and not necessarily in a preceding fashion. The lessons from one stream (mainstreaming, policy changes) could be used to inform the other (demonstration) and vice versa. Lessons learned from the mid-term evaluation show that the analysis should be done prior to initiating the project, at its mid-term, and at the end. This will help strengthen and maintain the high-level support for mainstreaming.

Key Actions:

7. Recommendation: Institutional coordination should be supported for integrated adaptation actions. This will also ensure active cross-agency oversight of implementation both locally and at national level. Comparative advantages of regional organizations and technical capacity gaps should be identified at the outset to enhance partnerships for delivery (regional/national).
Management Response: [Added: 2015/06/14]

We acknowledge the recommendation. The capacity building goal of the project was to build national management unit that could oversee day-to day implementation. The challenges at the first 2 ½ years of the project, with too low capacity at the regional and national level, led to resources being concentrated less on utilizing the comparative advantages of regional organizations and partners. It was only later on half way through the project, when all units were in place, that the assistance and gaps in regional partners and countries were assessed. The solution was to tenders technical experts in coastal, water and agriculture fields, increase technical capacity of project units by active involvement of the Climate Change Division of SPREP, and hiring team leaders for communications and knowledge management, cost benefit analysis program, and monitoring and evaluation; and lastly support of the Regional Technical Support Mechanisms for future up scaling projects.

Key Actions:

8. Recommendation: National accountability of regional projects needs to be improved and the financial processes need to be more simple and aligned with national systems. More attention should be paid to detailed project planning, tracking and monitoring to allow for more timely and responsive changes for delivery arrangements.
Management Response: [Added: 2015/06/14]

On Financials: In accordance to the project document, PACC is described as a NIM modality, where national financial rules and regulation are followed. Therefore, the project did allow all 14 countries to follow the national financial policies and processes. The difficulty faced by countries was on ?Financial Reporting complying with UNDP NIM guideline?. In particular, the modality for delivering advances only when 80% expenditure is reached, performing countries would be at a disadvantage due to non-performing countries whose delivery were below 80%. The 80% threshold rule has not been tested against a Regional Project of this magnitude with number of countries involved. As such, for PACC, this rule became a bottleneck and a risk factor on the pace of delivery during implementation stages. The second difficulty was consolidating all countries report and for submission to UNDP within the quarterly deadline. The option of direct payment from UNDP does provide some solution, there is a need to plan better in order to collect all quarterly reporting on time. It is further recommended that during the inception stage, Management Arrangement and Operational Rules ad Regulation should be clearly discussed with noted recommendations practiced. On Monitoring & Reporting: Project management training should be provided to project coordinators at the start of project. Capacity in terms of project management in the Pacific is improving, future projects may be able to hire coordinators that have demonstrated project management experience and M&E experience. It is also key that the regional project team allocate time and budget for regular fortnightly or monthly calls with national project teams to encourage regular project monitoring and consideration of adaptive action.

Key Actions:

9. Recommendation: Communications/knowledge management should be adequately resourced at regional and national levels to ensure effective delivery of communications strategies.
Management Response: [Added: 2015/06/14]

We agree with the recommendation. Following the inception phase, SPREP secured funds from UNITAR to assist with communication activities at the regional level. This included funding a communications officer position for 2 years, and rolling out a regional write shop. The officer's role was effective in that all country communications plans were developed for the project. The project, however, underestimated the amount of resources required to adequately implement the plan and effectively carry out the activities. The challenges therefore lied in national level capacities. The regional communication strategy challenge was pushing countries to move from visibility materials towards development of knowledge management products to address sustainability of outcomes of their projects. The KM activity of the regional although proved very successful in capturing all information, was delayed into the last stages of the project, and required significant time and resources to complete it.

Key Actions:

10. Recommendation: SPREP should focus on improving (1) the timeliness of delivery and adequate sourcing of technical expertise to provide continuous support of technical guidelines (2) prioritizing activities, and (3) sensitizing the government to the importance of climate change adaptation.
Management Response: [Added: 2015/06/14]

(1) We have discussed at every MPR meeting the important issue of timeliness of delivery of technical expertise and we believe we have addressed this well and truly. SPREP developed and called for retainer consultants under its External PACC Thematic Experts Team (EPTET) halfway through the project, with the view to assist SPREP and the RPMU. It channeled the efforts of the Climate Change Division team to provide targeted technical and operational support to the project with each member of the Division focusing on their assigned countries. It provided technical and advisory support, reviewed technical and operational progress reports, and assisted countries with implementation of country activities. This temporary measure by the CCD team proved useful as the RPMU moved to secure a team of 5 specialists to ensure solid support right throughout the remaining time of the project. This resulted in timely delivery of technical support in four key areas of management being finance and operations, cost benefit analysis program, knowledge management and communications, and monitoring and evaluation. These activities were a response to the Mid Term Evaluation recommendations to improve on the technical and advisory support to the countries from the regional unit. (2.) The RPMU encouraged more frequent contact with countries as a result with the team providing oversight, follow up and follow on activities with coordinators in country, ensuring simple, measurable, attainable, realistic and timely plans and activities against resources available. It included specialists flying into countries to assist in implementation, at the same time providing ongoing communications remotely (skype, phone calls, emails, etc.). The project as a result re affirmed the priority activities by mid 2014 during its last MPR meeting, with UNDP, SPREP and countries discussing on what were realistic activities that could be delivered within funds and time restrictions. As such activities were effectively prioritized (3) We feel that all governments consider climate change as a priority in their development agendas; assessment results from the mainstreaming work of the project provides proof to this. As such, there is no need to invest any further work and activity to sensitize an issue that is already identified, considered, reviewed and endorsed by government as a key and important development priority.

Key Actions:

11. Recommendation: Close engagement is required with state and national policy processes and local councils so that legislative changes incorporate CCA and DRR
Management Response: [Added: 2015/06/14]

This is what the project FSM carried out. All other countries proposed in their projects under mainstreaming that they focus on policies - sectoral or national, or strategies. FSM was the only member country to have focused on legislative changes. This is an issue where countries will need to suggest they seek to achieve changes at the legislation and regulation levels. The project then agrees to proceed along those outcomes.

Key Actions:

12. Recommendation: Demonstration designs based on cost benefit analysis and future climate projections should be integrated into adaptation measures (Federated States of Micronesia). Combination of different demonstration measures should be considered for for added impact (Fiji and Solomon Islands)
Management Response: [Added: 2015/06/14]

We acknowledge this recommendation. The lessons and experiences from FSM, Fiji and Solomon Islands and other countries have contributed to the development of the Mainstreaming Guide of the PACC. The Guide clearly outlines where in the project phase the CBA, SEA, V&A, climate science information including projections, and other tools and methodologies are applied. The promotion of ridge-to-reef approaches for example, was promoted highly by Samoa towards the last stages of its project, learning from lessons of its demonstration following Tropical Cyclone Evan in 2012. This has meant a number of strategically planned combination of different demonstration measures (e.g., downstream: re plantation of native trees along coastline, protection of water holes / sources; and upstream - re planting of native forest trees by other projects) are considered for added impact.

Key Actions:

13. Recommendation: Future interventions that support policy drafting must have a strong component (mainly technical assistance) at policy level to enable the Ministries of Planning / Finance to enforce such policies through advice to and mentoring of Ministries and overall budget allocation. One endeavor of PACC in Fiji was the drafting of the Climate Change policy that was later on endorsed by Parliament; it is however unclear whether the policy is actually being implemented by all beneficiary ministries although a Climate Change Division is supervising the process;
Management Response: [Added: 2015/06/14]

For Fiji: we acknowledge the recommendation, noting a stronger component to enforce policies can be helpful. It is useful for future projects. It must be noted, however, that the PACC Fiji's objective under its mainstreaming outcome was to support the development of the climate change policy as none existed before. The activity was planned to be a major activity and that resources were inadequate any more beyond the development and seeing to it that it be endorsed by cabinet. The political risk here was medium to high. In that the chances of getting the climate change policy endorsed in time (i.e., within time scope of the project) was unknown. It is important to note here as well - that endorsement of policies by government cabinets - as experience shows - vary greatly from immediate to a time that is outside of the project period. Hence the classification of medium to high risk. There is also the point of significant costs and time taken to enforce the policy throughout the country. As such, project planning by the stakeholders agreed to leave it to the development of a policy stage only where this stage includes endorsement by cabinet. The project therefore did not plan to extend the activities to implementing and enforcing what has been stated under the policy. The project team agreed as well to concentrate the budget on the demonstration activities. It weighed these decisions based on stakeholder involvement in different products of the project as well. i.e., the Climate Change Division (then under Ministry of Environment, now under the Ministry of Foreign Affairs) had more influence of the policy. As such, through a holistic fashion approach, it moved the implementation of or enforcement of the climate change policy to that Division, while PACC focused its resources to on the ground adaptation measures with the communities where the PACC implementing partner - MPI - had more influence and control. This was found to be (in the immediacy) more relevant, effective, efficient and potential impact to beneficiaries on the ground.

Key Actions:

14. Recommendation: Future interventions should adopt an inclusive consultation process: the beneficiaries must be an integral part of the formulation stage taking into account their specificities and viewpoints on the best approach to adopt when responding to an issue. Under PACC Fiji for example, there is a need for better communication between Government and farmers? representatives on what can be realistically achieved; a final workshop / community meeting should be organized. The drainage was initially the only solution proposed by Government to respond to community flooding as a strategy to enhance climate change adaptation; however over time, Government came up with an alternative less costly and more relevant: water resilient planting material; other adaptive solutions to enhance their livelihoods could also have been considered (e.g. marine resources use).
Management Response: [Added: 2015/06/14]

We acknowledge the recommendation. The project did carry out extensive consultations during its preparatory phase (in 2006-2008) and further into the initiation or inception stages. Experiences gathered during implementation stage shows, however, that beneficiaries (farmers, women in business, markets, students, elderly) will have contributed more towards an effective implementation and monitoring and data collection if frequent and constructive engagements (e.g., P3DM) and consultations were also carried out over the duration of the project. i.e., not only at the outset, but frequently during the project period. Some of the methodologies, such as those that considered gender perspectives, were not utilized during the inception phases as well. This will have contributed further to an effective, sustainable responses. This was a lesson captured by the project halfway through when it assessed its ability in addressing gender. It fully promotes this clear lesson from PACC for future projects.

Key Actions:

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org