Terminal Evaluation of the Integrated Landscape Management for Improved Livelihoods and Ecosystem Resilience in Mount Elgon

Report Cover Image
Evaluation Plan:
2016-2020, Uganda
Evaluation Type:
Final Project
Planned End Date:
12/2020
Completion Date:
12/2020
Status:
Completed
Management Response:
No
Evaluation Budget(US $):
40,000

Share

Title Terminal Evaluation of the Integrated Landscape Management for Improved Livelihoods and Ecosystem Resilience in Mount Elgon
Atlas Project Number: 00088957
Evaluation Plan: 2016-2020, Uganda
Evaluation Type: Final Project
Status: Completed
Completion Date: 12/2020
Planned End Date: 12/2020
Management Response: Yes
UNDP Signature Solution:
  • 1. Poverty
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 1.4.1 Solutions scaled up for sustainable management of natural resources, including sustainable commodities and green and inclusive value chains
  • 2. Output 2.1.1 Low emission and climate resilient objectives addressed in national, sub-national and sectoral development plans and policies to promote economic diversification and green growth
SDG Target
  • 13.1 Strengthen resilience and adaptive capacity to climate-related hazards and natural disasters in all countries
  • 13.3 Improve education, awareness-raising and human and institutional capacity on climate change mitigation, adaptation, impact reduction and early warning
  • 15.1 By 2020, ensure the conservation, restoration and sustainable use of terrestrial and inland freshwater ecosystems and their services, in particular forests, wetlands, mountains and drylands, in line with obligations under international agreements
Evaluation Budget(US $): 40,000
Source of Funding: GEF
Evaluation Expenditure(US $): 40,000
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
Dr. Justine Braby International Consultant justine.braby@gmail.com
Dr. John Wasige National Consultant johnwasige@gmail.com UGANDA
GEF Evaluation: Yes
GEF Project Title: Terminal Evaluation of the Integrated Landscape Management for Improved Livelihoods and Ecosystem Resilience in Mount Elgon
Evaluation Type: Terminal Evaluation
Focal Area: Climate Change
Project Type: MSP
GEF Phase: GEF-5
GEF Project ID: 00088957
PIMS Number: 4634
Key Stakeholders: Ministry of Agriculture, animal industry and fisheries; Ministry of water and environment; Ministry of lands, housing and urban development; Ministry of trade and industry, NEMA, NFA, Busitema University University
Countries: UGANDA
Lessons
1.
  1. Lesson 1: It is important to conduct comprehensive capacity assessment and effective stakeholder engagement (including community empowerment options) at design phase
  2. Implementation challenges and changes were faced by the project. A lesson to be learnt from these challenges is the importance of conducting comprehensive capacity assessments of the executing agency in terms of capacity to manage the project but also to implement certain activities in-house. Conducting a more robust stakeholder engagement process and mapping of capacity within the country in relation to specific outputs (either at onset of project or, preferably, at design) can further enhance ownership and sustaining of project results. 
  3. Lesson 2: Community empowerment (and encouraging women leadership) can have a much more sustained impact (and be more cost-effective)
  4. The greatest success factor of this project was the change from design to implementation (based on stakeholder pressure) to do a grant mechanism and empower (through training and facilitative support) CBOs to lead and run the initiatives directly. This might be more risky in terms of financial oversight (good training, trust building and good but not too limiting oversight can overcome this risk) but the reward is much higher – especially in terms of longer-term impact and sustaining of project results.
  5. Lesson 3: Devolving grant mechanism coordination to an intermediary (preferably an NGO who can do this in a programmatic way) might simplify project management responsibilities for PMUs
  6. The UNDP CO and PMU spent much of their implementation time on administering the grant (without human resources or sufficient capacity) to 33 different parishes. It might simplify project management and implementation procedures to allow a capacitated NGO or other implementing partner to do this, and supporting them instead with a small management fee, possibly building in a programmatic approach here (or if there is an existing and operational government grant mechanism, using that). The UNDP Low Value Grant Guidance allows for intermediary administration of a grant mechanism provided a HACT assessment is conducted.
  7. Lesson 4: Championship is key to project results attainment and sustainability, catalytic role and replication
  8. This lesson can be taken in two contexts, namely (a) champion farmers and CBO representatives, who allow for social spread more rapidly as they have trust and respect in their communities, and (2) champion project partners, whose commitment goes beyond the tick-box project implementation, and often results in a much higher and more impactful project.
  9. Lifting, rewarding and further empowering those champions who go the extra mile will have a reinforcing feedback loop effect on longer-term impact.

Findings
1.

Main Findings

 

  1. The below section summarizes the key findings of the Terminal Evaluation detailed in the content of this report.
  2. Project Design: The project document and its results framework was country-driven and addressed key national priorities related to sustainable land management and climate change[1]. The design did not include a Theory of Change and one had to be reconstructed for the purpose, which was used to guide the evaluation in terms of overall project impact. The overall objective, components and outcomes were generally feasible and practical in terms of the time frame of the project.[2] Overall the results framework was clear with SMART indicators, and gender-disaggregation. Despite the results framework having clear ecosystem-based and capacity-development objective-level indicators (in the form of GEF tracking tools), the evaluators noted that there were no livelihood indicators to track progress towards “enhanced livelihoods”. As a result, the evaluators made some suggestions on how these could have been included in the design.[3] The results framework would have benefitted from having clear outcome-level indicators and targets. Examples of these have been suggested.[4] Based on the level of changes and challenges faced during implementation, the evaluators believe more effective stakeholder engagement and partnership, including capacity assessments should have been done at design.[5]
  3. Project Implementation and Execution: The project underwent changes in project implementation, all of which were formally recorded. These included (a) going directly through CBOs to implement their land use plans and SLM technologies and pilot demo sites (which the evaluators found was the most successful part of the project)[6] (b) small changes in collaboration partners (including a. and the partnership with Busitema that was not planned during design), (c) delays in implementation that necessitated an 18-month no-cost extension.
  4. The first two changes (a) and (b) affected the project positively in that results were achieved successfully, and evidence has demonstrated that the potential for sustaining these results are high.[7]
  5. The third (c), affected the project negatively in that certain results were not achieved within the timeframe of the project, despite an 18-month extension having been granted. Reasons for this are attributed to prolonged uncertainty on who and how the output (specifically Output 2.1) was to be achieved due to disagreements between MAAIF and UNDP, delays in procuring consultants (including challenges in finding suitable candidates), and finally when work plans and commitments had been put in place (more than a year into the 18-month extension), COVID-19 forced the project co further postpone resulting in the result not being achieved by project closure.[8]
  6. Unnecessary delays contributed to the lack of results achievement discussed above, most notably, the delay in setting up the PMU. The project was delayed by one year because MAAIF had not been able to set up the PMU. The PMU was eventually set up directly by UNDP as agreed by the Project Board in June 2017.[9]
  7. Project finance and expenditure was reported on annually. The evaluators found high variances between planned and actual expenditure by year, particularly in the first year. This can be attributed to the delays experience in the first year where virtually no activities were implemented. Huge discrepancies exist between co-financing committed at project design and actual co-financing realized during project implementation. Complete co-financing expenditure information was not received for the evaluation and thus the evaluators can only assume that either reporting was weak, or co-financing in the form that was committed was not realised.[10]
  8. The grant mechanism, although effectively implemented overall, posed some administrative challenges, and the project team highlighted that in hindsight it would have been more efficient (and opened up more time to do more technical aspects) to have used an intermediary to administer the grant.[11]
  9. M&E plan included the basic requirements, although the results framework would have benefitted from some minor improvements on the output-level indicators, and the inclusion of outcome-level indicators in the results framework.[12] Implementation of M&E was done through quarterly reporting, PIR reporting and the M&E framework.
  10. Risks management was generally well-managed with the exception of the risks to implementation arrangements that caused unnecessary delays in some aspects of the project already covered in the paragraphs above. 
  11. Project results and impacts: Outcome 1 has largely been successfully achieved. Land-use plans were developed for all parishes and these have been mainstreamed into district development plans in all three districts. Clause adoption has achieved more limited success, although evidence suggests that their adoption is ongoing beyond project closure. Monitoring and enforcement mechanisms have been put in place and integrated into district annual workplans.[13] Outcome 2 has had been very successfully achieved in terms of the empowerment aspect and demo sites with communities, but only partially achieved on some of its outputs by project closure. Implementation delays affected particularly Output 2.1 which was not achieved within the timeframe of the project (although it is clear from the evidence provided to the evaluators that workplans and commitments had been made to finalise this post-project). Some achievement had been made towards strengthening public-private collaboration to improve farmers’ access to inputs, but the Action Plan was not fully developed or implemented. The implementation of SLM and SFM at community level was highly successful and evidence suggests uptake, replication, and sustaining of project results. Monitoring frameworks for carbon sequestration and soil erosion had been developed and put in place, but implementation is limited due to low capacity at district level. A best practices document was not developed, although best practices from the SLM interventions have been replicated and upscaled into neighbouring areas of project intervention sites.[14]
  12. Relevance: The project was well-aligned to country priorities at government level (predominantly through the NDPII and ASSP), and within the UN country framework (UNDAF and CPAP). Stakeholder engagement during project implementation was strong (e.g. representation on the project board, representation at Inception Meeting including update of recommendations from stakeholders into project implementation such as the direct implementation of some outputs by communities, level of engagement of local-level government).
  13. Effectiveness: The project did not manage to fully achieve on all its outputs. Some activities that could have been realistically achieved within the project timeframe were not because of implementation-related delays (such as the action plan for public-private collaboration, the FFS training, the best practices documentation), other activities were overly ambitious (such as the carbon monitoring framework). Despite this, the project managed to make some impactful achievements in terms of the wider Theory of Change, particularly in relation to community empowerment and uptake towards SLM integration into farming to enhance resilience and ecosystem health in the project areas of Mt Elgon.[15]
  14. Efficiency: The project faced several delays including losing the first year because the PMU was not set up, which ultimately resulted in the project not achieving some of its outputs within the project timeframe despite the granting of an 18-month no-cost extension. There were some variances between years, but the project was generally cost-efficient in terms of its expenditure in relation to outcome.[16]
  15. Overall project outcome: Considering the above paragraphs, overall achievement of outputs is moderately satisfactory. Some elements were highly successful, especially the community engagement, the use of community championships, gender empowerment through women leadership of community implementation of SLM and SFM activities. The landscape planning and management processes were certainly an improvement from what was there before (the project managed to have digitized, and integrated land use planning conducted for all 33 parishes). Local communities were empowered to apply technologies and approaches to reverse land degradation and reduce GHG emissions through the adoption of technologies in the project areas, but also through the replication of neighbouring communities who saw for themselves that the technologies worked and supported improvements in livelihoods.
  16. Sustainability: Several examples of financial commitment suggest that project results will be sustained, especially at community level.[17] Empowerment of communities, including women empowerment, has resulted in community-championship of SLM interventions and further uptake; government at various levels has seen the success of community engagement and has demonstrated political support for further uptake.[18] In terms of governance and legal frameworks, land use plans and by-laws are progressing forward, the inter-ministerial taskforce is a good reflection of a wider and more programmatic approach to SLM and wider adoption of project successes have been demonstrated through a NEMA-led project to upscale SLM into the wider Mt Elgon landscape.[19] Environmental sustainability has been improved through the project in relation to the environmental risks encountered prior to project implementation.[20]
  17. Gender empowerment: Gender empowerment was particularly strong in this project as demonstrated by the level of women leadership and championship of SLM interventions in the project areas. The Gender Action Plan was developed late (half-way through project implementation) and the project would have benefitted from an action plan of this level of quality if it been done at PPG phase. However, the evaluators found that women empowerment particularly was demonstrated in the monitoring of the results framework (particularly at community level).

Conclusions

 

  1. The project faced implementation challenges that put project results attainment at risk. Unnecessarily delaying some output implementations until the very last months (March – August 2020) of an already maximised project time-frame[21] increased vulnerability of the project to external risks, which is exactly what happened when the COVID-19 pandemic restricted project implementation in the final months of the project. In fact, the project should have been under final operational closure in these months (and not in “starting phases” of output implementation). If it had been, it would not have been as affected by COVID-19 as it was.
  2. The finalisation of Output 2.1 (the FFS training) merits discussion, because the evaluators were provided with evidence that in fact funding was committed (funds transfer dated 26 August), and MAAIF commitment through work plans signed off by the Permanent Secretary had been approved by the Project Board, all before project closure. This evidence was provided during the finalisation of this TE report, and shows that activities of this output will be finalised by the end of the year (2020). However, the international evaluator is limited to assessing the project results attainment within the timeframe of the project (which officially closed 31 August 2020). Whether the output will or will not be achieved beyond project closure cannot be assessed by the TE. GEF rules stipulate that no extensions can be provided beyond the 18 months that had already been provided to the project. A recommendation is made by the evaluator in the finalisation of project results for specific outputs that were not achieved within the timeframe of this project, but this is placed in the context of GEF rules and regulations, and the IA, EA and GEF will need to take this matter forward accordingly. Specific recommendations are made below in relation to the finalisation and sustaining of some outputs that were not finalised within the project time frame (Recommendation Category A below).
  3. The Government of Uganda had made a large co-financing commitment in project design of which the majority was not realised, at least in terms of expenditure reporting made available to the evaluators. Similarly, expenditure reporting for the majority of the co-financing contribution from UNDP was also not available for the evaluators to assess where exactly the co-financing was used to achieve project results. GEF funding is supposed to be an incremental contribution, not the core contribution, towards what are essentially government-led projects. GEF additionality is a new section for evaluators to consider in GEF-funded project evaluations, and future project design will be focusing more on co-financing commitments and realisations thereof in project implementation. Co-financing that is committed at project design should be reported on, and a lesson from this project has been included in the lessons learnt for future project design.
  4. UNDP had to take over some of the executive functions of MAAIF during project implementation, including the setup of the PMU and direct recruitment and sometimes even management of project outputs. This leads the evaluators to question why the HACT capacity assessment that was used for project design had not picked up some limitations of MAAIF as an executing partner. Similarly, recruitment of appropriate staff and consultants proved difficult (e.g. FFS, gender). On the other hand, MAAIF was convinced that it had the in-house capacity to conduct the FFS training, and there was resultant to-and-fro between UNDP and MAAIF about whether external (or FAO) consultants should be used or not. Improved capacity assessment (HACT) of the EA may have supported a better understanding of this capacity or limits thereof.
  5. In summary, even with the project extension, some outputs were not fully delivered. Some other outputs demonstrated successful (or, in some cases over-) achievement. Results overall in terms of the project’s results framework were not fully achieved.
  6. That said, the project demonstrated achievements in terms of its outcomes and overall objective within the framework of the reconstructed Theory of Change that illustrates that the project was impactful in what it set out to achieve within the broader aims of SLM in Mt Elgon.
  7. The project aimed to decrease land degradation and enhance ecosystem health by using landscape management approaches and SLM, SFM and CCM technologies with communities to improve livelihoods and enhance resilience among people and ecosystems. The two strategic, higher-level questions guiding the evaluation linked to its two outcomes were (a) did the project success in integrating and improving landscape planning and management processes in the three project districts? and (b) did the project contribute to empowering communities in Mt Elgon to manage their production landscapes in an integrated manner?
  8. Based on the evidence provided to the evaluators of uptake and integration of land use planning in the districts, the integration of monitoring frameworks to support measuring land condition improvements, and the successful uptake and ownership at community-level of the SLM and SFM demonstrations, as well as the replication into areas outside of project intervention, it seems that the project did indeed contribute substantially to both improved landscape planning and improved community management of their production landscapes. The extent of this has been demonstrated in the results section of this report. The recommendations provided below will further strengthen this contribution toward impact and sustainability.
  9. Additionally, and in particular reference to community championship, the evaluators believe that the community-led approach had a significant impact in terms of sustaining project result, and move to impact in the TOC. This is evidenced by strong community ownership and uptake, longer-term savings schemes generated (through training and community empowerment as a result of allowing communities to lead activities instead of one company), and thus sustainability of results at least in terms of community-related SLM, SFM and CCM interventions.
  10. The project fit well within the larger programmatic approach of the Government (in relation to its inter-ministerial task force on SLM) relevant to country development priorities (e.g. NDPII) and should be seen as a leverage point for further catalytic action towards more systems and holistic SLM across wider landscapes.
  11. The evaluators believe that the community-led approach had a significant impact in terms of sustaining project result. This is evidenced by strong community ownership and uptake, longer-term savings schemes generated (through training and community empowerment as a result of allowing communities to lead activities instead of one company), and thus sustainability of results at least in terms of community-related SLM, SFM and CCM interventions. 
  12. Based on the above considerations, the project, overall, is given a rating of Moderately Satisfactory, with the summary table provided below (see Table 4 in main report for description of rating system).

 


 


Recommendations
1

Evaluation Recommendation 1:  Ensure bylaws are effectively finalized and enforced. NEMA to facilitate this process to fruition using the following steps: (a) bottom-up verification process at parish level should be conducted to get final buy-in from resistant community members, (b) submitting and tabling at sub-county, district, and finally Attorney General level (for verification and validation with existing laws) and (c) finally gazette the by-laws. 

2

Evaluation Recommendation 2: Ensure final reporting on lessons learnt and uptake as per Output .5 (particularly include CBO and women empowerment aspects of this project).

3

Evaluation Recommendation 3: Continue supporting the formation of cooperatives to improve farmer access to markets 

4

Evaluation Recommendation 4: Provide continued platform for successful CBOs to share stories, support training in future replication, farmer

5

Evaluation Recommendation 5:  This recommendation is specifically to be included in the project design for the project by NEMA/UNEP (concept development support from MAAIF) on SLM implementation in the broader Mt Elgon region. Integrate community empowerment, women leadership and lessons below into the GEF-cycle concept development that aims to catalyze and replicate aspects of this project into the entire Mt Elgon ecosystem.

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org