Evaluation Detail

Final Evaluation: Environmental Governance Reform Project (EGR)

Evaluation Plan:: 2019-2023, Cambodia
Evaluation Type:: Final Project
Planned End Date:: 04/2019
Completion Date:: 10/2019
Status:: Completed
Management Response:: No
Evaluation Budget(US $):: 17,222

Document	Type	Language	Size	Status	Downloads
09_EGR FE Report - Final TRACKED_Steve_Aug 19_clean.docx	report	English	1006.75 KB	Posted	237
ToR_Final Evaluation_EGR.pdf	tor	English	24201.01 KB	Posted	77

Title

Final Evaluation: Environmental Governance Reform Project (EGR)

Atlas Project Number:

00088932

Evaluation Plan:

2019-2023, Cambodia

Evaluation Type:

Final Project

Status:

Completed

Completion Date:

10/2019

Planned End Date:

04/2019

Management Response:

Yes

UNDP Signature Solution:

1. Poverty

Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)

1. Output 1.1.1 Capacities developed across the whole of government to integrate the 2030 Agenda, the Paris Agreement and other international agreements in development plans and budgets, and to analyse progress towards the SDGs, using innovative and data-driven solutions
2. Output 2.1.1 Low emission and climate resilient objectives addressed in national, sub-national and sectoral development plans and policies to promote economic diversification and green growth

SDG Goal

Goal 15. Protect, restore and promote sustainable use of terrestrial ecosystems, sustainably manage forests, combat desertification, and halt and reverse land degradation and halt biodiversity loss

SDG Target

15.1 By 2020, ensure the conservation, restoration and sustainable use of terrestrial and inland freshwater ecosystems and their services, in particular forests, wetlands, mountains and drylands, in line with obligations under international agreements
15.2 By 2020, promote the implementation of sustainable management of all types of forests, halt deforestation, restore degraded forests and substantially increase afforestation and reforestation globally
15.4 By 2030, ensure the conservation of mountain ecosystems, including their biodiversity, in order to enhance their capacity to provide benefits that are essential for sustainable development

Evaluation Budget(US $):

17,222

Source of Funding:

USAID (cost-sharing)

Evaluation Expenditure(US $):

17,222

Joint Programme:

Joint Evaluation:

Evaluation Team members:

Name	Title	Email	Nationality
Steve Raaymakers	EcoStrategic Consultants

GEF Evaluation:

Key Stakeholders:

Countries:

CAMBODIA

Lessons
1.	The Cambodia EGR Project is assessed to be one of the most relevant projects that the Final Evaluation Consultant (FEC) has evaluated, in that it responds directly to a major national reform agenda as already formulated and initiated by the Minister for Environment. The Project was driven by the needs and priorities of the beneficiary country, rather than imposed by the development partners (demand- not supply-driven). In this regard it provides a best practice model for the design of other development projects.
2.	The technical components of the Project are assessed to be well conceived and designed based on best practices in the relevant areas of focus. However, the project design is found to be overly ambitious relative to limited timeframe and budget. A much larger and longer, phased design would have been more appropriate given the highly strategic, ambitious and complex nature of the EGR agenda.
3.	Despite major staff, time and budget constraints, the Project was still able to deliver a wide range of often high-quality activities and outputs across all four Project components, in support of the overall strategic objective of supporting the RGC’s EGR agenda. This is a major credit to all parties involved.
4.	Despite the high rate of achievement, many of the most important outputs remain to be finalized and fully implemented by MoE and NCSD, including inter alia: a) finalization, adoption, enactment and implementation of the Environment Code, b) operationalization of the integrated ecosystem mapping decision support system; and c) operationalization of the new human resources and related management systems at MoE.
5.	Given that those outputs of the Project that are designed to have the most significant impact have yet to be finalized and implemented – especially the Environment Code – it is too early to measure any significant impact. This needs to be assessed in future years once all Project outputs are fully implemented and operationalized.
6.	If the Code is actually finalized and operationalized, it is likely to have an extremely significant, positive impact, and move Cambodia towards being a leader in best practice environment protection, natural resource management and sustainable development.
7.	If the Code is not finalized and operationalized, then the overall impact of the EGR Project is likely to be minimal, and much of the Project investment will have been wasted. This highlights the need for an EGR Project Phase 2, as outlined in section 4.5.5.
8.	An annual financial audit was conducted in January 2018 covering the period January to December 2017, and a “spot check” audit was carried out in December 2018, covering the period January to September 2018, following UNDP Financial Rules. Given a major surge in expenditure since September 2018 it is recommended that a detailed financial audit be undertaken for the whole Project after financial closure.
Findings
1.	The EGR Project is assessed to be one of the most relevant projects that the Final Evaluation Consultant (FEC) has evaluated, in that it responds directly to a major national reform agenda as already formulated and initiated by the Minister for Environment. As such the Project was driven by the needs and priorities of the beneficiary country rather than imposed by the development partners (demand- not supply-driven). The Project also fits well with the United National Development Assistance Framework (UNDAF) for Cambodia, UN SDGs and USAID development objectives. Tag: Relevance Ownership Programme/Project Design
2.	During project design the beneficiary Ministry was able to work closely with UNDP, USAID, the Government of Japan and other partners and their project design consultants, to strongly determine the objectives, focus and design of the Project in accordance with its EGR agenda. The technical components of the Project are assessed to be were well conceived and designed based on best practices in the relevant areas of focus. Tag: Programme/Project Design Bilateral partners UN Agencies
3.	However, the project design is found to be overly ambitious relative to limited time-frame (initially only two years and extended to three years and two months) and limited budget (initially only US$2.9 million cash grants with an additional $1 million provided later by USAID). The project design also only allowed for a very small, two-person Project Management Unit (PMU) (a Project Manager and Project Assistant only). Tag: Human and Financial resources Programme/Project Design Project and Programme management
4.	The position classification of the Project Manager (UN P3) is also assessed to be too junior relative to the highly ambitious scope and complexity of the Project, the strategic political significance of the Project and the extremely senior level of engagement required with RGC officials (up to Secretary of State and even Minister). Ideally the project design (and budget) should have allowed for a P4 or even P5 classification. Tag: Human and Financial resources Programme/Project Design
5.	The Project design also suffered from some weaknesses in the Project Results Framework (PRF) and M&E plan (refer section 4.3). Tag: Monitoring and Evaluation Results-Based Management
6.	Overall the Project implementation arrangements and performance are assessed to be satisfactory, as exemplified by: Effective use of the Project Board, which enabled a high level of influence by MoE and NCSD over project implementation. A very high rate of delivery of activities and outputs relative to staff, time and budget constraints (a high delivery to resources ratio = efficient). Strong utilization of support from a wide range of partners – including UNE experts, experts from US Government Agencies other than USAID -through the US Department of Interior International Assistance Program and others. Efficient and timely UNDP procurement and disbursement processes - as reported by stakeholders - although delays in some cases Tag: Implementation Modality Operational Efficiency Project and Programme management Bilateral partners Country Government
7.	The Project players and partners also exhibited an outstanding capacity for adaptive management in response to changing circumstances, which is critical to the success of any project, including: Empowerment of the Project Board with discretion to steer the Project in response to changing circumstances, needs and priorities. A very rapid response by USAID to an identified need for additional funds to allow more substantive Project completion, with rapid approval of an additional US$1 million in funding to allow seamless continuity into an initial one year extension. Flexibility and commitment within UNDP to allocate core staff to the project management role during gaps in having a dedicated Project Manager (although such gaps should have been avoided in the first place – see Key Finding 4 below). Tag: Operational Efficiency Project and Programme management Bilateral partners
8.	The project design should have provided a staff structure, budget and timeframe that were better aligned (i.e. bigger and longer) with the extremely ambitious objectives, broad scope and complex nature of the EGR reform agenda, as outlined in Key Findings 1.5 to 1.7. Tag: Programme/Project Design
9.	There was an unacceptable six-month delay to Project start and an unacceptably high turnover of Project Managers (three in just two years), and significant gaps without a dedicated Project Manager. During these gaps core UNDP staff who may not be experienced in complex project management had to assume the project management function. This resulted in significant lulls and surges in implementation activity – with a marked “rush” of activity in the last year of the Project (see recommendation 4.1). Tag: Human and Financial resources Operational Efficiency Programme/Project Design Project and Programme management
10.	There are non-trivial questions about the use of a single-supplier private law firm to develop the Environment Code under a $750K contract equating to ~20% of total Project budget. Senior personnel reported that they felt that the contract for this major piece of work did not have adequate performance-based delivery, transparency and accountability checks and controls, reportedly making it difficult for UNDP to exercise sufficient control and oversight of the contractor’s performance (see recommendation 4.2). A more ideal approach might have been to develop the Code through a well-resourced, MoE-led inter-ministerial working group, supported by experts. Tag: Environment Policy Natural Resouce management Impact Policies & Procedures Procurement
11.	As outlined in Key Finding 3 the EGR Project has delivered a wide range of often high-quality activities and outputs across all four Project components, in support of the RGC’s overall strategic EGR objectives. However, as also outlined in Key Finding 3, many of the most important outputs that have been delivered by the Project remain to be finalized and fully implemented by MoE and NCSD, including inter alia the Environment Code, the integrated ecosystem mapping decision support system and the new human resources and related management systems Tag: Impact Sustainability Ownership Country Government
12.	The FEC finds that implementation of the EGR Project through UNDP benefitted from UNDP’s comparative advantage as a development enabler. This comparative advantage is based on the long-standing physical presence of the UNDP Country Office in Cambodia, with a long history of UN support to the RGC on a wide range of political, governance, social, economic and sustainable development issues. As a UN agency UNDP is trusted in Cambodia as a neutral, impartial, objective, a-political development partner, backed by an enormous global body of technical expertise and experience in international development. Tag: UN Agencies UNDP Management Advocacy
13.	The FEC assesses that given the high-level strategic significance of the EGR agenda, the revolutionary impact once fully implemented, and the wide scope and complexity of the Project’s four components, a more appropriate project design would have allowed for: a minimum budget of $20 million in grant support with at least four times in-kind support from partners, a time frame of 10 years divided into a five year development phase and a five year implementation phase; and an expanded PMU with three Technical Managers under the Project Manager, one for components 1 and 2 (MoE and NCSD institutional strengthening), one for component 3 on the legal code and one for component 4 on integrated ecosystem mapping and data management. Tag: Programme/Project Design
14.	The PRF for the EGR Project is presented in Table 3, as adapted from the ProDoc. Assessment of this identifies a number of unusual aspects, including inter alia: Tag: Impact Monitoring and Evaluation
15.	The ProDoc also contains an M&E plan. Review of this finds that while the M&E Plan should be clearly linked to the PRF, using the PRF Targets and Indicators as the basis for M&E – there are no such linkages in the M&E plan for the EGR Project. The planned M&E activities are also quite limited, comprising: Tag: Monitoring and Evaluation Oversight Quality Assurance
16.	A key element was missing from the M&E plan - a Mid Term Review (MTR) by an independent, external consultant. Given the highly ambitious and complex nature of the Project, an MTR would have been useful in bringing a fresh, external perspective to assist adaptive management. Restricting independent, external evaluation to project-end is of limited value as it is too late to make improvements to the Project once it is closed. Tag: Monitoring and Evaluation
17.	It appears that the PMU and supporting UNDP staff achieved extremely impressive results through innate project management skills, driven by personality, commitment, strong work ethic and natural intelligence, but there appears to have been limited use of the PRF as a project planning, management and monitoring tool, and M&E implementation was not fully effective. Assessing the APRs and other internal progress reports indicates a tendency to focus on just reporting long-lists of activities that have been undertaken, with limited analysis of whether these activities have actually been successful in generating the planned outcomes and impact. Tag: Monitoring and Evaluation Project and Programme management
18.	It is recommended that the design of future projects should give more careful consideration to ensuring a strong, clear and well articulated PRF and M&E plan, as these provide powerful tools for managing and monitoring project implementation. Tag: Monitoring and Evaluation Programme/Project Design Project and Programme management
19.	As far as could be ascertained through document review and consultation with stakeholders it appears that the DIM/LoA combination worked well, although it is not clear to the FEC that there was adequate financial auditing of expenditure trails relating to LoA disbursements. Given that the implementation status of some activities has been incorrectly reported (e.g. NCSD corridor branding, see point d) under Evaluation Limitations above), it might be prudent for UNDP to conduct a detailed financial audit once the Project is closed. Tag: Oversight Project and Programme management
20.	From review of board meeting minutes and other documents, and consultation with stakeholders, it appears that the Project Board was an effective component of the management arrangements for the EGR Project, including: enabling a high level of influence by MoE and NCSD over project implementation, ensuring close communication between all Project partners at a senior level; and empowering the Board with discretion to steer the Project in response to changing circumstances, needs and priorities. Tag: Oversight Project and Programme management Country Government
21.	To manage the day-to-day implementation of Project delivery UNDP established a small PMU comprising a Project Manager (UN P3) and Project Administration & Finance Assistant (UN SB3). The PMU staff worked alternately from an office provided at MoE for around three to four days a week and from the UNDP office for around one to two days a week. This is assessed as being a smart and effective modality – allowing both close, day-to-day integration with the beneficiaries at MoE and NCSD, and the ability to follow-up on procurement, disbursement and project reporting processes at UNDP. Tag: Implementation Modality Project and Programme management
22.	The FEC assesses that the Project players exhibited an outstanding capacity for adaptive management in response to changing circumstances, which is critical to project success. Examples included: Empowerment of the Project Board with discretion to steer the Project in response to changing circumstances, needs and priorities. A very rapid response by USAID to an identified need for additional funds to allow more substantive Project completion, with rapid approval of an additional US$1 million in funding to allow seamless continuity into an initial one year extension. Flexibility and commitment within UNDP to allocate core staff to the project management role during gaps in having a dedicated Project Manager (although such gaps should have been avoided in the first place – see section 4.4.3). Tag: Project and Programme management Bilateral partners
23.	In accordance with UNDP evaluation guidelines the FE includes an overall assessment of the financial aspects of the Project. However the FEC is not an accountant or financial auditor and no attempt at all has been made by the FEC to verify the Project financial data provided by UNDP – these data are accepted at face value. It should be noted also that despite requesting them, the FEC has not been provided with any financial audit reports for the EGR Project. Tag: Oversight

Recommendations
1	It is recommended that the outstanding responsiveness of the EGR Project to directly meeting the defined needs and priorities of the beneficiary country should be adopted as a best practice model for the design of other projects (development projects tend to have greater beneficial impact when they are demand- rather than supply-driven).
2	It is recommended that the design of future projects should give more careful consideration to ensuring that the level of resourcing and implementation timeframe are better aligned with the objectives and scope of the project.
3	It is recommended that the design of future projects should give more careful consideration to ensuring a strong, clear and well-articulated PRF and M&E plan, as these provide powerful tools for managing and monitoring project implementation.
4	It is recommended that MoE and NCSD should give high priority and to finalizing and fully implementing all key outputs that have been delivered by the Project but which have not yet been finalized, especially the Environment Code, HR plan and systems, and integrated ecosystem mapping.
5	It is recommended that UNDP, USAID and other development partners should support RGC as far as practicable in implementing Recommendation 4.
6	It is recommended that for future projects, in order to prevent loss of implementation continuity through high turnover of key staff like the Project Manager, as occurred with the EGR Project, UNDP should provide greater security of employment for these positions. The simplest reform would be to issue employment contracts that run for the full duration of the Project rather than the current, insecure year-by-year contracts. Performance criteria can easily be included to allow non-performing staff to be terminated if necessary. The position classification should also be appropriate to the significance of the project, in order to attract and retain the right candidates.
7	It is recommended that for future projects, when large private firms are engaged to implement major project components for relatively large fees, as was done under the EGR Project to develop the Environment Code, UNDP should ensure that contracting processes include stringent performance-based delivery, transparency and accountability checks and controls.
8	It is recommended that in finalizing the Environment Code, MoE should consider the issues identified in Annex 6 of this report.
9	It is recommended that in order not lose the benefits from EGR investments to date, as a matter of high priority UNDP should start working with MoE, NCSD and potential partners to start designing and resourcing an EGR Project Phase 2. Phase 2 should focus on implementing the major outputs of Phase 1 - especially the Environment Code. The Phase 2 Project should incorporate lessons learned from Phase 1 – including from this FE report. The Phase 2 Project could seek novel and innovative financing modalities, for example: - Partnering between UNDP, MoE, NCSD and infrastructure developers and financiers to demonstrate best practice EIA for major infrastructure such as the Phnom Penh – Sihanoukville expressway. - Partnering between UNDP, MoE, NCSD and the new “Green Belt & Road Initiative” to demonstrate best practice environment and social safeguards and green design in major projects. - Establishing Public-Private Partnerships (PPP) to implement viable waste management solutions in priority areas (UNDP has established PPP models).

Final Evaluation: Environmental Governance Reform Project (EGR)

Share

Evaluations by Unit

Latest Evaluations

Evaluation Snap Shot

Contact us