Terminal Evaluation of GEF LDCF/ climate-resilience Angola´s Cuvelai River Basin Project

Report Cover Image
Evaluation Plan:
2020-2023, Angola
Evaluation Type:
Project
Planned End Date:
03/2022
Completion Date:
12/2021
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
30,000

Share

Document Type Language Size Status Downloads
Download document TOR_Terminal Evaluation_Cuvelai Project 14062021.pdf tor English 261.74 KB Posted 235
Download document DEL4_0132_UNDP_Angola_Cuvelai_FINAL_TE_Report_051221.pdf report English 3869.78 KB Posted 223
Title Terminal Evaluation of GEF LDCF/ climate-resilience Angola´s Cuvelai River Basin Project
Atlas Project Number: 00081003
Evaluation Plan: 2020-2023, Angola
Evaluation Type: Project
Status: Completed
Completion Date: 12/2021
Planned End Date: 03/2022
Management Response: Yes
Focus Area:
  • 1. Poverty
  • 2. Others
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 3.1.1 Core government functions and inclusive basic services4 restored post-crisis for stabilisation, durable solutions to displacement and return to sustainable development pathways within the framework of national policies and priorities
SDG Goal
  • Goal 1. End poverty in all its forms everywhere
  • Goal 11. Make cities and human settlements inclusive, safe, resilient and sustainable
  • Goal 13. Take urgent action to combat climate change and its impacts
  • Goal 15. Protect, restore and promote sustainable use of terrestrial ecosystems, sustainably manage forests, combat desertification, and halt and reverse land degradation and halt biodiversity loss
SDG Target
  • 1.5 By 2030, build the resilience of the poor and those in vulnerable situations and reduce their exposure and vulnerability to climate-related extreme events and other economic, social and environmental shocks and disasters
  • 11.5 By 2030, significantly reduce the number of deaths and the number of people affected and substantially decrease the direct economic losses relative to global gross domestic product caused by disasters, including water-related disasters, with a focus on protecting the poor and people in vulnerable situations
  • 13.1 Strengthen resilience and adaptive capacity to climate-related hazards and natural disasters in all countries
Evaluation Budget(US $): 30,000
Source of Funding: GEF project funds
Evaluation Expenditure(US $): 30,000
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Nationality
Fabiana Issler , CEO of EBD Global Optimum BRAZIL
GEF Evaluation: Yes
GEF Project Title: Promoting climate-resilient development and enhanced adaptive capacity to withstand disaster risks in Angolan’s Cuvelai River Basin
Evaluation Type: Terminal Evaluation
Focal Area: Climate Change
Project Type: FSP
GEF Phase: GEF-5
GEF Project ID: 5177
PIMS Number: 5166
Key Stakeholders: UNDP, MINAMB, IDA, GABHIC, INRH, Civil Protection, GEF, ADPP, DW, local government Cunene
Countries: ANGOLA
Lessons
1.

Lesson 1 - reference is made to section 2.7 of the evaluation report on the Limitations to the Evaluation Methodology

There are pros and cons in conducting evaluations remotely. It is likely that not as many individual stakeholders would have been contacted and interviewed by the TE, if the assignment included a mission to Angola, which tends to be a rushed process. Interacting with 24 unique individuals representing 14 different entities was only possible because of the remote nature of the TE. At the same time, it was not possible to directly interact with beneficiaries in the field and hear their perspective.

Lesson 2 – reference is made to section 4.1, and under the 'Lessons from other relevant projects incorporated into project design'

The development of Flood Forecast Early Warning System (FFEWS) needs to be approached through the step-by-step creation of pre-conditions. First, it is important to generate hydroclimatic data and then generate analysis and develop Early Warning Services. And in order to generate hydroclimatic data, measurement instruments must be installed early in the project’s lifetime.


Findings
1.

The TE’s main conclusion and FINDINGS can be thus summarized: The TE generally concludes that: the project  had an Overall Satisfactory Performance (S), all criteria considered, and considering that delivering results is the most important evaluation criterion.

Some of the project’s shortcomings could perhaps have been avoided, if a strong PMU had been set up in MINAMB/MCTA. The fact that UNDP and the implementing partner sustained a weak and understaffed PMU for the entire project duration is undermined project performance in many different way. It was the project’s highest risk, which remained unflagged and unaddressed throughout implementation.

It is worth noting that the project involves a large number of partners, including national institutions and NGOs, and it draws on the capacities of private sector entities, as well as south-south and foreign cooperation for delivering technically complex information and systems, and for building national capacities.

For managing complex processes, a strong, multi-disciplinary, capable and adequately staffed PMU would have been needed. The project never counted on one. Instead, the project struggled, and the sustainability of its results are now at risk. The TE believes that this can still be addressed with the few months of implementation remaining.

The project did deliver a number of interesting results, which should be capitalized upon and sustained, while there is still time.

Slated to close in February 2022, the project will likely need to return unused funds to the LDCF (possibly $0.8M to $1M, judging from the current pace of delivery). Such situations must be avoided in the future.


Recommendations
1

Recommendation 1: Low delivery, as demonstrated in Table 4. Execution and cumulative delivery on the LDCF Grant and on UNDP Core Resources.

2

Recommendation 2: There are apparently plans to conduct, at project end, a more comprehensive VRA study, but its scope and usefulness are not known by the TE. To date, a VRA using the same methodology as the one conducted by DW in 2018/9 had not been replicated across project localities in the Cuvelai River Basin, not conducted again in the same localities.

3

Recommendation 3: Amounts that will likely end up being returned to the GEF could also have been spent in consolidating project results to ensure a stronger sustainability. With strict limitations imposed on project duration and the number of milestone extensions, it will be difficult to consume the remaining LDCF budget until February 2022.

4

Recommendation 4: Although a 4-year duration tends to be the norm in several UNDP GEF projects, a duration this short for a project implemented in Angola, of the complexity of that of Cuvelai Project, and with a budget of $8.2M, is clearly too short. The limited absorptive capacity of government’s implementing partner and responsible parties plays also a role in the delays, as attested by several of the stakeholders interviewed, including UNDP. However, even in countries with stronger implementation capacity, the project’s complexity and budget size alone would warrant a longer duration, possibly of 5 to 6 years.

5

Recommendation 5: Although 18 months is currently the maximum time allowed by UNDP NCE in terms cumulative milestone extensions, it is not enough to compensate for the time loss in the beginning of the project, for the impacts of covid-19 on the project, and for the fact that the project’s original scoping of a 4-year duration had grossly overestimated the national absorptive capacity.

6

Recommendation 6: The TE analyzed the ‘smartness’ of project indicators in Section 3.4 (refer to Table 11). Several indicators and end-of-project targets are not specific enough to be easily measurable. This reflected negatively in the quality of reporting through the PIRs. If the VRA baseline VRA had been conducted during the PPG, it would be less of a problem, but this was not the case. The baseline only established very late and only in part.

7

Recommendation 7: From the point of view of adaptative management, the misalignment between project duration and expectations is shortcoming that could have been addressed in a timely manner, but was not.

8

Recommendation 8: More than a risk that was apparently not flagged and not adequately acted upon, a weak PMU ended up being a burden on those who were expected to oversee the project (officials in UNDP and MINAMB/MCTA), and to the extent that they stepped in to fulfil project management functions where there was a void. This situation created distortions in what would the ideal functions of different project players and it fuelled the likelihood of conflicts of interest. (Accumulating roles in implementation and oversight is considered a conflict of interests.)

9

Recommendation 9: The project has resulted in a lot of benefits, and it will rely on other projects to replicate and further upscale to a more significant level. A follow-up intervention is recommended to further secure the investment made by the GEF/LDCF, the government and UNDP.

1. Recommendation:

Recommendation 1: Low delivery, as demonstrated in Table 4. Execution and cumulative delivery on the LDCF Grant and on UNDP Core Resources.

Management Response: [Added: 2022/02/27]

Management Response: Recommendation fully accepted. UNDP recognizes all identified challenges from the delays on project start date to minimal project implementation progress up to the mid-term evaluation. The CO was also aware of the complexity of the project and challenges from not having an effective PMU at the IP, reason why this capacity was filled by a technical expert seconded to the Ministry to work with Cuvelai and Coastal Adaptation projects. Lessons from this project will be used to monitor progress in project implementation, identifying in timely manner potential risks that can affect the delivery and escalate to the established project governance mechanisms.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Work closely with the IP to ensure timely implementation of project planned activities.
[Added: 2022/02/27]
Env. & DRR Team 2022/12 Initiated
2. Recommendation:

Recommendation 2: There are apparently plans to conduct, at project end, a more comprehensive VRA study, but its scope and usefulness are not known by the TE. To date, a VRA using the same methodology as the one conducted by DW in 2018/9 had not been replicated across project localities in the Cuvelai River Basin, not conducted again in the same localities.

Management Response: [Added: 2022/02/27]

Management Response: Recommendation noted and accepted. However, given lack of clarity on the usefulness of such VRA by the TE, IP decides to not proceed with the study. The understanding is that this should be considered for future programs planned for the southern region of Angola, which is currently severely affected by recurring drought. The government expressed interest to build on from Cuvelai project lessons and seek for substantial investment through GCF to scale up good practices to other regions of the country facing similar challenges and/or exposed to potential threats. A VRA study will be relevant for the design phase of the future program and should be considered as a monitoring instrument to access effectiveness of proposed interventions at critical stages of project cycle (design, MTR and TE). The DW 2018/9 will serve as a benchmark on the tools and process to undertake VRA in Angola at community level to inform development initiatives aimed at uplifting communities’ livelihoods.

Key Actions:

3. Recommendation:

Recommendation 3: Amounts that will likely end up being returned to the GEF could also have been spent in consolidating project results to ensure a stronger sustainability. With strict limitations imposed on project duration and the number of milestone extensions, it will be difficult to consume the remaining LDCF budget until February 2022.

Management Response: [Added: 2022/02/27]

Management Response: Recommendation fully accepted. The CO worked closely with the IP to identify opportunities where to invest the remaining resources vs approved project results framework. Nevertheless, the unwillingness to re-structure the PMU from IP did not help such effort being made by UNDP. Hence, resources will have to be returned to the GEF. Going forward, as indicated in the recommendation 1, the CO will explore all possibilities to get an effective and functional project governance functions to identify risks and have them addressed using adaptive management principles, which should enable the IP and PMU to deliver on the project outputs within the established timelines in the project document.  

Key Actions:

4. Recommendation:

Recommendation 4: Although a 4-year duration tends to be the norm in several UNDP GEF projects, a duration this short for a project implemented in Angola, of the complexity of that of Cuvelai Project, and with a budget of $8.2M, is clearly too short. The limited absorptive capacity of government’s implementing partner and responsible parties plays also a role in the delays, as attested by several of the stakeholders interviewed, including UNDP. However, even in countries with stronger implementation capacity, the project’s complexity and budget size alone would warrant a longer duration, possibly of 5 to 6 years.

Management Response: [Added: 2022/02/27]

Management Response: Recommendation partially accepted. Although the initial project duration was set for 4 years, in practice this was actually the time that the project delivered on most of its results. Key challenges are related to non-responsiveness of the IP on critical aspect in project management. As it was well captured in the TE report, it took over one year for the LPAC meeting to be convened after approval of the project by the GEF Sec, similar delays were also observed for the Inception Workshop and establishment of the PMU. In addition, a strong PMU was required to manage such a complex project, which was not considered as relevant by the IP. This served as a lesson to UNDP to consider such lengthy processes and recent projects are now designed for 5 or more years of duration.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Interact with the IP and RTAs on the upcoming projects to advocate for relatively longer project implementation timeframes (minimum 5 years).
[Added: 2022/02/27]
Deputy RR and Env. & DRR Team. 2022/12 Initiated This is also envisioning the upcoming GEF8 programming and other donors funding opportunities.
5. Recommendation:

Recommendation 5: Although 18 months is currently the maximum time allowed by UNDP NCE in terms cumulative milestone extensions, it is not enough to compensate for the time loss in the beginning of the project, for the impacts of covid-19 on the project, and for the fact that the project’s original scoping of a 4-year duration had grossly overestimated the national absorptive capacity.

Management Response: [Added: 2022/02/27]

Management Response: Recommendation partially accepted. An exceptional additional 6 months extension was granted to the IP, to cater for the challenges deriving from COVID-19 imposed restrictions. However, key immediate actions agreed with the IP to ensure sustainability of outputs delivered towards the project end were not considered. IP major challenge is not related to the timeframe, rather proactiveness to ensure timely implementation of planed interventions. On the other hand, recurrent institutional changes at ministerial level (4 ministers in 4 years) negatively affected the project performance with a highly centralized government structure, where most of the decision had to come from the minister. The CO is currently exploring possibilities with different government structures in identifying ideal model to ensure that future projects will not face similar challenges.

Key Actions:

6. Recommendation:

Recommendation 6: The TE analyzed the ‘smartness’ of project indicators in Section 3.4 (refer to Table 11). Several indicators and end-of-project targets are not specific enough to be easily measurable. This reflected negatively in the quality of reporting through the PIRs. If the VRA baseline VRA had been conducted during the PPG, it would be less of a problem, but this was not the case. The baseline only established very late and only in part.

Management Response: [Added: 2022/02/27]

Management Response: Recommendation fully accepted. UNDP will ensure that future projects are designed with smart indicators, which are easier to measure considering the country context. The CO also noted that PPG processes are expected to provide all set of baseline data to minimize the risk of having baseline and targets being set during first year of project implementation, which may be biased as the PMU will be accountable to deliver on the targets that it can influence during first stage of project implementation. This also brings to the attention of the IP on the need for building strong institutional capabilities in data gathering and management to inform decision making processes.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
CO will work closely with the IP and Consultants assigned to support with development of Project Documents to ensure that projects indicators are smart and corresponding baseline data and targets agreed upon during the design phase.
[Added: 2022/02/27]
Env. & DRR Team. 2022/12 Initiated This will be applied with new project being developed for the GEF POPs, CBIT, as well as for the other potential donors such a joint proposal with UNICEF for the GCF, a proposal for IFAD and others.
7. Recommendation:

Recommendation 7: From the point of view of adaptative management, the misalignment between project duration and expectations is shortcoming that could have been addressed in a timely manner, but was not.

Management Response: [Added: 2022/02/27]

Management Response: Recommendation fully accepted. For future projects UNDP on its oversight role, will work closely with the IP to ensure that projects are implemented effectively, by addressing issues and risks timely to minimise the impacts of potential shortcomings.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Undertake monthly coordination meetings with the IPs and PMUs to access project planned activities progress on implementation.
[Added: 2022/02/27]
Env. & DRR Team. 2022/12 Initiated This will inform on the potential risks and address them timely.
Conduct at least 2 field missions to project sites.
[Added: 2022/02/27]
Env. & DRR Team and Ips/PMUs. 2022/12 Initiated
8. Recommendation:

Recommendation 8: More than a risk that was apparently not flagged and not adequately acted upon, a weak PMU ended up being a burden on those who were expected to oversee the project (officials in UNDP and MINAMB/MCTA), and to the extent that they stepped in to fulfil project management functions where there was a void. This situation created distortions in what would the ideal functions of different project players and it fuelled the likelihood of conflicts of interest. (Accumulating roles in implementation and oversight is considered a conflict of interests.)

Management Response: [Added: 2022/02/27]

Management Response: Recommendation fully accepted. UNDP CO recognises this as the major challenge, particularly with the National Directorate of Environment and Climate Action. A formal notification letter on the need to resolve most of issues related to accumulation of roles was submitted to the Ministry. This will be followed up with a meeting (subject to confirmation on the dates) with the new Minister to ensure that GEF portfolio is well managed. On a separate note, the recruitment and selection of PMU, in particular the project managers functions will need to go through a comprehensive assessment process, to certify that right person with right skills are selected through a competitive and transparent process.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
As part of its oversight functions, the CO will work closely with the IP to ensure that in future projects the selection of the PMUs is held following the defined qualifications and work experience in the ToRs. Whenever allowed and possible, the an Env.&DRR Team member will be appoint to serve as an observer in the selection process.
[Added: 2022/02/27]
Env. & DRR Team. 2022/12 Initiated
9. Recommendation:

Recommendation 9: The project has resulted in a lot of benefits, and it will rely on other projects to replicate and further upscale to a more significant level. A follow-up intervention is recommended to further secure the investment made by the GEF/LDCF, the government and UNDP.

Management Response: [Added: 2022/02/27]

Management Response: Recommendation fully accepted. UNDP CO is currently working with the government to develop a proposal for GCF consideration. This is expected to scale up some of the good results from Cuvelai project to other regions of the country.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Development of a project proposal jointly with UNICEF targeting GCF to scale up lessons from Cuvelai project in other regions of the country.
[Added: 2022/02/27]
Env. & DRR team. 2022/12 Not Initiated The proposal will look into building resilience of communities and institutions, with a component on WASH to be implemented jointly with UNICEF.

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org