Final Project Evaluation - “Enhancing adaptive capacity of communities to climate change-related floods in the North Coast and Islands Region of Papua New Guinea

Report Cover Image
Evaluation Plan:
2018-2022, Papua New Guinea
Evaluation Type:
Final Project
Planned End Date:
12/2018
Completion Date:
05/2019
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
50,000

Share

Document Type Language Size Status Downloads
Download document TOR_International Consultant_PNG AF_Final Project Evaluation_ 31102017 Final.docx tor English 87.32 KB Posted 455
Download document Final Report_PNG AF_Final Project Evaluation_201218.pdf report English 3402.40 KB Posted 597
Title Final Project Evaluation - “Enhancing adaptive capacity of communities to climate change-related floods in the North Coast and Islands Region of Papua New Guinea
Atlas Project Number: 00059799
Evaluation Plan: 2018-2022, Papua New Guinea
Evaluation Type: Final Project
Status: Completed
Completion Date: 05/2019
Planned End Date: 12/2018
Management Response: Yes
UNDP Signature Solution:
  • 1. Resilience
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 2.1.1 Low emission and climate resilient objectives addressed in national, sub-national and sectoral development plans and policies to promote economic diversification and green growth
  • 2. Output 2.3.1 Data and risk-informed development policies, plans, systems and financing incorporate integrated and gender-responsive solutions to reduce disaster risks, enable climate change adaptation and mitigation, and prevent risk of conflict
SDG Goal
  • Goal 1. End poverty in all its forms everywhere
  • Goal 13. Take urgent action to combat climate change and its impacts
SDG Target
  • 1.5 By 2030, build the resilience of the poor and those in vulnerable situations and reduce their exposure and vulnerability to climate-related extreme events and other economic, social and environmental shocks and disasters
  • 13.1 Strengthen resilience and adaptive capacity to climate-related hazards and natural disasters in all countries
  • 13.2 Integrate climate change measures into national policies, strategies and planning
Evaluation Budget(US $): 50,000
Source of Funding: Adaptation Fund
Evaluation Expenditure(US $): 50,000
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
Eduardo Quiroga International Consultant sylvagro@aol.com
GEF Evaluation: No
Key Stakeholders: Climate Change Development Authority, Conservation and Environment Protection Authority, National Disaster Centre, National Weather Service and 5 Provinces of East Sepik, Madang, Morobe, New Ireland and Northern
Countries: PAPUA NEW GUINEA
Comments:

This is an AF funded project and not GEF.

Lessons
1.

The lessons learnt were contained in the recommendations as presented in evaluation report on page 59. Refer to section 4 (Conclusions, Recommendations and Lessons).


Findings
1.

3- FINDINGS AND CONCLUSIONS 3.1- Project Design / Formulation 3.1.1- Analysis of Logical Results Framework
The Project was designed around four outcomes [Prodoc, page 20]. For analytical purposes, these can be subsumed into two larger outcomes: ? One outcome dealt with adaptation to flood-related risks and hazards in two ecosystems: [1] coastal ecosystems in the North Coast and Island region communities and [2] inland riverine ecosystems in the East Sepik, Oro, Morobe and Madang Provinces.


Tag: Climate Change Adaptation Disaster Recovery Disaster Risk assessments Natural Disaster Natural Resouce management Site Conservation / Preservation Policies & Procedures

2.

3.1.2- Assumptions and Risks
The essence of the risks assessed in the Prodoc were realized during project implementation but mitigation measures were not easily amenable to implementation. This is reflected in Table 2 where a comparison of risk levels between the conditions during the preparation of the Prodoc and those prevailing during the implementation of the TE shows slight differences.
The assumptions for the political stability and commitment to climate compatible development are maintained, as are the assurances of the PNGNWS to expand and manage their weather monitoring and forecasting activities. Likewise the relative support from district and provincial level officials continues to enable cooperation among communities, districts and provinces. Provincial governments encourage expanding the role and resources for the climate change officers/focal points, and in particular, the Department of Education remains committed to the integration of climate change in school curricula.


Tag: Monitoring and Evaluation Risk Management Coordination

3.

3.1.3-Lessons from other Relevant Projects
There were several other GEF funded interventions [see Table 3] in the context of community-based climate change adaptation, including regional climate change adaptation in a pilot site in PNG. These climate-related projects were well analyzed. Other projects dealt with awareness raising and institutional strengthening that were complementary to the current Project—where the focus was on the integration of climate change related issues into established awareness raising and institutional training platforms. These project experiences led to a solid understanding of local and national institutional frameworks, and working relationships were established with several officers within ministerial and agency organizations.


Tag: Climate Change Adaptation Knowledge management

4.

3.1.4-Planned Stakeholder Participation
The information available indicates that during the project preparation, stakeholders’ involvement and consultation were strong in addressing climate change impacts, adaptation and mitigation risks. The specific topics focused were mangrove rehabilitation, perceptions and behavior patterns in a specific geographical area concerning climate issues. Undoubtedly these consultations provided sound insights to ensure that the project responded to national priorities.


Tag: Climate Change Adaptation Disaster Recovery Disaster risk management Natural Resouce management Communication Civil Societies and NGOs Capacity Building Coordination

5.

3.1.5 Replication Approach Replication is defined as lessons and experiences coming out of the project that are replicated or scaled up in the design and implementation of other projects. The realities on the ground are that this Project has a long maturation period. The Project results currently available include only outputs [FPRR pp 7-17]. In GEF terms, the main focus is at the outcome level. Most UNDP supported GEF financed projects are expected to achieve anticipated outcomes by project closing, and recognizing that global environmental benefit impacts are difficult to discern and measuring outputs is insufficient to capture project effectiveness. Therefore, it would be premature to submit experiences from this Project to be replicated or scaled in the design and implementation of other projects.


Tag: Climate Change Adaptation Disaster Recovery Disaster risk management Natural Disaster Natural Resouce management Innovation Programme/Project Design Capacity Building

6.

Summation
Consider coordinating with relevant stakeholders on the provincial development plan content of the planning process so it encompasses the key economic sectors; and update the skills needed [i.e. elements of geomatics] to incorporate the geo spatial DRM/DRR data into the planning process in such a way that interoperability is ensured. interoperability facilitates institutional work given that by sharing information in a consistent and readable way, greater added value can be given to the institutional products themselves. This approach needs to be validated by the respective national economic authorities and simultaneously begin coordinating with national IT infrastructure to determine the possible need for a common platform with data being updated/curated by each separate department of the province. All over the world the use of DRM/DRR geospatial data is essential for many economic management procedures within the framework of climate change variability.


Tag: Effectiveness Human and Financial resources Monitoring and Evaluation Capacity Building

7.

3.1.6 - UNDP Comparative Advantage
UNDP has been assisting PNG in designing and implementing activities consistent with national sustainable development plans, while reflecting on the UNDP’s comparative advantage. This is supported by the UNDP’s global network of country offices, its experience in integrated policy development, human resource development, institutional strengthening, and non-governmental and community participation. Furthermore, UNDP assures an effective facilitation of partnerships with fellow UN Agencies and long- standing experience in the fields of policy support and capacity development. UNDP was well placed to provide the institutional and technical support required for this project.


Tag: Strategic Positioning UN Agencies

8.

3.1.7-Linkages between Project and other Interventions18
Within the UN framework there are several ongoing interventions focused on land use systems, village water supply, drought resistant food production, and enhanced governance for disaster risk management. These interventions indicate the continued relevance of the Project.
?


Tag: Environment Policy Natural Resouce management Monitoring and Evaluation Food Security Capacity Building

9.

3.1.8- Management Arrangements According to the original organigram [Box 1], the core arrangements for management were as follows.
? A Project Steering Committee (PSC) made up of high level representatives from CCDA, UNDP and other key stakeholders is co-chaired by the Executive Director of CCDA and the local UNDP Resident Representative. The Provincial Administrators or their delegates from the 5 provinces are also members of the PSC. There can be other PSC members from pertinent national organizations. The PMU assumes the function of secretariat.


Tag: Human and Financial resources Implementation Modality Project and Programme management Country Government UN Agencies

10.

3.2-Project Implementation 3.2.1-Adaptive Management
Adaptive management measures were implemented, stemming from the MTE recommendations submitted in February 2016. The MTE [pp 22-33] concluded that the Project had made almost no progress from October 2012 to February 2016. This was reflected starkly in one indicator. The Project had disbursed less than 3% of the AF grant while the timeline had already used 31% of its total. This sluggish progress reflected by the lethargic disbursement stemmed from the pattern of asymmetrical management, as partially discussed earlier. Despite the fact that in 2015 activity execution had shot up, without an extension in time, the Project would have compromised the achievement of an undetermined number of targets.


Tag: Communication Human and Financial resources Implementation Modality Knowledge management Project and Programme management

11.

3.2.2-Partnership Arrangements
Partnerships are vital as they provide the foundation for multi-sectoral collaboration to enhance the adaptive capacity of communities to build resilience. The Project has benefited from a multi-donor partnership between UNDP, Adaptation Fund and the Australian Government through the Department of Foreign Affairs and Trade [FPRR, page 19].
The working relationship between CCDA, NWS, NDC and CEPA has been strengthened. This has been helpful in establishing an integrated early warning system for Bumbu River flooding and developing integrated climate change adaptation and disaster risk management plans.


Tag: Climate Change Adaptation Resilience building Natural Resouce management Site Conservation / Preservation Bilateral partners Civil Societies and NGOs Country Government

12.

3.2.3-Feedback from M&E Activities used for Adaptive Management
Annual reports register the progress made during the past quarters and document the progress made against the project objective and outcomes. A review and update of the risks identified at the outset of the Project and the steps taken to mitigate these risks are also found in these reports. There is insufficient information about any adaptive management issues faced by the Project and management changes made. This is exemplified by the situation when the Project was facing difficulties in setting up an operational PMU at CCDA, including periods where there was no PM and then a change in PM between 2014 and 2015. The information available in these reports is undergoing a risk assessment, whereby a risk was logged that finding a PM is difficult because of the competition with private sector companies. This assessment is, however, moot because the skills profile needed  by the UNDP and other UN organizations, especially for project work, do not coincide with the skills normally in demand in the private sector, especially the extractive industry [MTE page 40].
The TE was conducted three months after the Project was closed. The project team has already disbanded to take on different assignments, and so the information for the evaluation came sparsely. The Final Project Review Report issued on 30 November 2017 came with the TOR as well as with a list of publications from the Project. There was, however, no information provided on the Adaptive Management matters, although there were references about the availability of funds from disbursements made possible by the extension of project end date.


Tag: Effectiveness Human and Financial resources Monitoring and Evaluation Project and Programme management

13.

3.2.4-Project Finance
The financial information comes from the Final Project Review Report [FPPR] submitted by project management on 30 November 2017; it is based on financial information for the project, with an operational closure date of 31 December 2017. On this date, as previously mentioned, the project team disbanded to take on new assignments. The Terminal Evaluation was launched on 9 April 2018 and all of the financial information used comes from the FPPR.


Tag: Climate Change Adaptation Efficiency Human and Financial resources Project and Programme management

14.

3.2.5-Monitoring & Evaluation
In conformity with UNDP standards and AF procedures, including the UNDP M&E practices for NIM projects, a comprehensive M&E plan was developed during the formulation of the project. The M&E budget of USD 83,000 was allocated representing only about 1.4% of the AF grant.
As is standard, the plan listed M&E activities that were to be implemented during the lifetime of the Project, including a mid-term evaluation and a terminal evaluation. For each M&E activity, the responsible party(ies) was/were identified, as well as a budget and schedule. The plan was based on the logical framework matrix that included a set of performance monitoring indicators along with their corresponding means of verification. During the inception phase, the M&E plan was reviewed and there were no modifications.


Tag: Climate Change Adaptation Resilience building Effectiveness Monitoring and Evaluation Oversight Project and Programme management

15.

3.2.6 UNDP and Implementing Partner Implementation / Execution
The Project design was complex and there was activity execution in five sites without practically any coordination. There was also a large number of training activities conducted, and it was not easy to ascertain what the expected results were. Also, as discussed above, a Project’s vision was missing, and all of these factors made it difficult to focus on results.
UNDP support for the implementing partner and project team were adequate and timely. In turn, the implementing partner was enthusiastic about the support provided by UNDP. UNDP carried out risk management activities to mitigate adverse impacts on achieving objectives and facilitated extensive international expertise to support project components, including South-South cooperation with a global climate adaptation initiative. Concretely, as part of capacity building for operation and maintenance of the Early Warning System, four staff of the National Weather Service and the Conservation, Environment and Protection Agency participated in a south to south learning exchange to Australia. It provided training for the establishment of the Bumbu river early warning system [FPRR page 17].


Tag: Climate Change Adaptation Effectiveness Human and Financial resources Implementation Modality Ownership Project and Programme management South-South Cooperation

16.

3.3-Project Results
3.3.1-Overall results [attainment of objectives] [*] The focus to assess overall results is at the outcome level. The Project has not attained the expected outcomes.
Under the circumstances, the Guidelines signposts procedures to assess project results in the absence of outcome results.25 Thus, from the output-based implementation results [Annex 12] and field observations, supported by focused beneficiary interviews [Annex 7], the Mission has triangulated intermediate outcomes [Annex 15] . Annex 15 is the Mission’s composite estimate of the Project’s progress in outcome achievement --derived from annexes 7,12 and 14. However on October 20, 2018 the CO shared with the Evaluator another estimate of outcome progress [Annex 17]. Therefore, Annex 17 was given preference when assessing the overall results.


Tag: Climate Change Adaptation Natural Resouce management Effectiveness Policies & Procedures Country Government Capacity Building Education

17.

3.3.2 Relevance Relevance Rating : Relevant [R]
PNG is on the frontline when it comes to climate change impacts, even though it is a small contributor to climate change. With rising sea levels and increasingly extreme weather events, the daily reality in the country leaves little room for climate skepticism. Environmental impacts are already irrevocably changing life in these island states – with rising sea levels exacerbating rising sea levels. The Project is in line with supporting development priorities at all levels of the GoPNG’s efforts to lead planning, coordination and on-the-ground implementation of measures to facilitate adaptation at all administrative levels. GoPNG intended to place particular focus on increasing vulnerable communities‘ resilience to climate change.
The available information indicates that target group priorities were dealt satisfactorily. However, the a priory distribution outputs to stimulate the communities to begin the process of CCA required a more refined methodology. This is to avoid introducing unintended inequities among the communities and members within the communities, as has been alleged.
UNDP has been assisting PNG in designing and implementing activities consistent with national sustainable development plans. UNDP’s comparative advantage is reflected in its global network of country offices, its experience in integrated policy development, human resource development, institutional strengthening, and nongovernmental and community participation.


Tag: Climate Change Adaptation Resilience building Relevance

18.

3.3.3 Effectiveness and Efficiency
Effectiveness and Efficiency Rating: Moderately Unsatisfactory [MU] Effectiveness
From October 2012 to February 2016, there was almost no progress in terms of activity implementation.29 The Project had disbursed less than 3% of the AF grant, while the timeline had already used 31% of its total. This lethargic disbursement seemed to stem from a pattern of asymmetrical management [Annex 14]. During 2015 activity implementation had picked up, however without an extension in disbursement time the Project would have suffered an even greater shortfall of implementation targets. The MTE recommended an extension to December 2017 to allow extra time for fund disbursement to help expediate development results—with particular reference to concrete actions in the adaptation to climate change-driven hazards affecting the target areas.


Tag: Effectiveness Efficiency Human and Financial resources

19.

3.3.4 Country Ownership
The project concept was in line with the GoPNG’s development priorities and plans reflected in the Climate Compatible Development Strategy (CCDS) set out in March 2010. At the same time, the Cabinet also established the Office of Climate Change and Development [OCCD] as well as the National Climate Change Committee [NCCC], who took full and exclusive responsibility of climate change and environmental sustainability in PNG. The OCCD then created four Technical Working Groups: REDD+, Adaptation, Consultations, and Low-carbon Growth [Prodoc page 10].
The GoPNG representatives were involved at the department level level and management arrangements were conducted through the CCDA. The CCDA played the role of implementing partner and five provincial governments were also involved in the implementation, including NGOs.
The government has not yet enacted any legislation or developed policies and regulations derived from the Project’s outputs. Some outputs are about to be completed. Five provincial governments are in the process of preparing plans to control and manage risk from coastal and riverine flooding. Likewise, the automatic early warning systems will be operational soon. Indeed, all Project outcomes are consistent with national policies and can be incorporated into the national and sectoral development plans. The national government has maintained financial commitment to the project


Tag: Climate Change Adaptation Ownership Country Government

20.

3.3.5 Mainstreaming
With the purpose of managing the hazards of flooding in the coastal areas of the North Coast and the Islands Region as settlements are usually located along the coasts, the Project has already conducted a systematic assessment of the vulnerability of these coastal and riverine communities to enable the necessary institutional and individual capacity at a national, provincial, district, and local level to enable decentralized and well-informed decision-making.


Tag: Climate Change Adaptation Resilience building Natural Resouce management Gender Mainstreaming

21.

3.3.6 Sustainability Overall Sustainability Rating: Moderately Likely [ML]
The Prodoc estimates that the prospects for long-term sustainability of Project achievements is rated as “likely sustainable.” The GoPNG has shown commitment to address climate change. Its intentions have been reflected in the country‘s CCDS, and the establishment of the NCCC and the CCDA. Indeed, this Project has been a direct response to government priorities in the climate change adaptation area. Therefore, it can be inferred that the GoPNG will continue financing and implementing CCA actions in the near and medium term timeframe.


Tag: Climate Change Adaptation Natural Resouce management Sustainability Project and Programme management

22.

3.3.7 Impact
The Project has potential to positively effect on the ecological status and contribute decreasing the stress on the ecosystem of two regions: [1] coastal flooding risks in selected communities of the Northern and Island Coastal Provinces of East Sepik, Madang, Morobe, New Ireland and Northern, [2] the inland flooding risk in selected river communities in Northern Coastal Provinces.


Tag: Climate Change Adaptation Ecosystem based adaption Natural Resouce management Impact

23.

3.3.8 Promoting Accountability and Transparency34
Broad concerns about accountability and transparency arose in the presence of the Evaluator, during focused interviews and unstructured interaction with stakeholders and beneficiaries, both during field visits and in working sessions with partners held both in the capital and the five provincial offices. The queries raised by beneficiaries and stakeholders are collected in one annex [Annex 13] for the attention of the Auditors. As it was important to listen to these concerns.


Tag: Oversight Procurement

24.

3.3.9 Project Convergence with UNDP Priorities Including UNDAF Harmonization
Although the Project has not achieved outcomes tied to the UNDAF outcome area “ Environment, Climate Change and Disaster Risk Management”, several sets of outputs share common ground with UNDAF.38 This budding set of outputs, ostensibly, provide an understanding of the root causes of poverty and conflict. This can be helpful to achieve MDG goals as well.
 


Tag: Climate Change Adaptation Disaster risk management Disaster Risk Reduction Natural Resouce management UN Country Team Coordination

Recommendations
1

Inconsistent application of UNDP Procurement Procedures.  

2

Mangrove rehabilitation requires community engagement in the design, planning and implementation of activities to accommodate traditional land tenure arrangements including traditional and cultural norms and practices to ensure socio-economic development aspirations are addressed.

3

Enhance opportunities for training in mainstreaming climate change adaptation.

4

Priory Output Delivery without Sustainable Strategy

5

Choice of Partners with Sustainability Strategy

6

The Role of Institutions in the Transfer of Technology

1. Recommendation:

Inconsistent application of UNDP Procurement Procedures.  

Management Response: [Added: 2019/05/07] [Last Updated: 2020/12/02]

UNDP’s procurement policies and guidelines are consistently applied to implementing partner, contractors and service providers in provinces.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
UNDP assisted the institutional review of Climate Change Development Authority and will assist strengthen project support unit for coordinating and managing donor supported initiatives including on the job-training for national government officials.
[Added: 2019/05/07]
Energy and Environment Unit 2019/01 Completed Project is financially and operationally closed History
2. Recommendation:

Mangrove rehabilitation requires community engagement in the design, planning and implementation of activities to accommodate traditional land tenure arrangements including traditional and cultural norms and practices to ensure socio-economic development aspirations are addressed.

Management Response: [Added: 2019/05/07] [Last Updated: 2020/12/02]

In this pilot intervention, UNDP focused and targeted awareness and trainings on climate change adaptation such as mangrove restoration and/or conservation as agreed with the stakeholders so that lessons learnt will be replicated in other provinces in the country.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Future projects with Climate Change Development Authority will ensure a more holistic approach to address an inclusive participatory approach that encompasses community development priorities in climate change adaptation actions through allowing more time for planning for community-based activities
[Added: 2019/05/07]
Energy and Environment Unit 2019/01 Completed Project is financially and operationally closed
3. Recommendation:

Enhance opportunities for training in mainstreaming climate change adaptation.

Management Response: [Added: 2019/05/07] [Last Updated: 2020/12/02]

In this pilot intervention, UNDP ensured focused and targeted trainings in five pilot provinces so that lessons learnt will be replicated in other sectors and provinces in the country.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Future projects with Climate Change Development Authority will ensure a more holistic approach in planning, coordination and delivery of training in mainstreaming climate change adaptation.
[Added: 2019/05/07]
Energy and Environment Unit 2019/01 Completed Project is financially and operationally closed
4. Recommendation:

Priory Output Delivery without Sustainable Strategy

Management Response: [Added: 2019/05/07] [Last Updated: 2020/12/02]

Project provided technical and strategic advisory services to implementing partner and the pilot provinces who successfully established their respective provincial climate change coordination committees, set-up climate change offices with allocation of budget and staff to institutionalize climate change adaptation to ensure smooth transition into sustaining UNDP supported interventions after project closure.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
New and ongoing projects supported by UNDP are co-led by government agencies during design and implementation to promote leadership and ownership for the long-term sustainability of internationally funded climate change actions. UNDP will further enhance its approach to ensure that sustainable approach is clearly and strictly applied to the projects.
[Added: 2019/05/07]
Energy and Environment Unit 2019/01 Completed Project is financially and operationally closed
5. Recommendation:

Choice of Partners with Sustainability Strategy

Management Response: [Added: 2019/05/07] [Last Updated: 2020/12/02]

UNDP’s programmes are aligned with the national priorities that are reflected in the UNDAF which endorsed by the Government. The selection of the implementing partner is based on set of criteria and careful considerations, such as accountability, ownership and broad stakeholder engagement as well as the ability to ensure sustainability. Throughout the implementation process, the IPs are to be consulted early and regularly throughout the programme and project formulation to ensure that priorities and approaches reflect partnership and funding commitments towards achieving results. UNDP in consultation with the Implementing Partner may enter into agreement with other organization known as Responsible Party to provide support that will contribute to achieving results. RP is selected based on assessment that reflects set of criteria to proofs RP’s eligibility and accountability to implement activities.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
UNDP will conduct implementing capacity assessment during the design of any project to inform the development of appropriate capacity development plans which will be implemented simultaneously with implementation of project activities.
[Added: 2019/05/07]
Energy and Environment Unit 2019/01 Completed Project is financially and operationally closed
Coordination and consultation process will be conducted regularly with all partners involved and are responsible to contribute to the national priorities.
[Added: 2019/05/07]
Energy and Environment Unit 2019/01 Completed Project is financially and operationally closed
UNDP Project Management and Procurement Trainings will be conducted for all implementing partners including government agencies, private sectors, non-government organisations and community-based organisations selected to implement activities.
[Added: 2019/05/07]
Energy and Environment Unit 2019/01 Completed Project is financially and operationally closed
6. Recommendation:

The Role of Institutions in the Transfer of Technology

Management Response: [Added: 2019/05/07] [Last Updated: 2020/12/02]

UNDP has put in place improved capacity to ensure adequate use of technology driven data to inform application of development planning towards addressing climate change adaptation. 

Key Actions:

Key Action Responsible DueDate Status Comments Documents
UNDP will continue to ensure regular engagement and dialogue on transfer of technology and skills to the implementing partner.
[Added: 2019/05/07]
Energy and Environment Unit 2019/01 Completed Project is financially and operationally closed

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org