| 1. |
Main Findings
- The below section summarizes the key findings of the Terminal Evaluation detailed in the content of this report.
- Project Design: The project document and its results framework was country-driven and addressed key national priorities related to sustainable land management and climate change[1]. The design did not include a Theory of Change and one had to be reconstructed for the purpose, which was used to guide the evaluation in terms of overall project impact. The overall objective, components and outcomes were generally feasible and practical in terms of the time frame of the project.[2] Overall the results framework was clear with SMART indicators, and gender-disaggregation. Despite the results framework having clear ecosystem-based and capacity-development objective-level indicators (in the form of GEF tracking tools), the evaluators noted that there were no livelihood indicators to track progress towards “enhanced livelihoods”. As a result, the evaluators made some suggestions on how these could have been included in the design.[3] The results framework would have benefitted from having clear outcome-level indicators and targets. Examples of these have been suggested.[4] Based on the level of changes and challenges faced during implementation, the evaluators believe more effective stakeholder engagement and partnership, including capacity assessments should have been done at design.[5]
- Project Implementation and Execution: The project underwent changes in project implementation, all of which were formally recorded. These included (a) going directly through CBOs to implement their land use plans and SLM technologies and pilot demo sites (which the evaluators found was the most successful part of the project)[6] (b) small changes in collaboration partners (including a. and the partnership with Busitema that was not planned during design), (c) delays in implementation that necessitated an 18-month no-cost extension.
- The first two changes (a) and (b) affected the project positively in that results were achieved successfully, and evidence has demonstrated that the potential for sustaining these results are high.[7]
- The third (c), affected the project negatively in that certain results were not achieved within the timeframe of the project, despite an 18-month extension having been granted. Reasons for this are attributed to prolonged uncertainty on who and how the output (specifically Output 2.1) was to be achieved due to disagreements between MAAIF and UNDP, delays in procuring consultants (including challenges in finding suitable candidates), and finally when work plans and commitments had been put in place (more than a year into the 18-month extension), COVID-19 forced the project co further postpone resulting in the result not being achieved by project closure.[8]
- Unnecessary delays contributed to the lack of results achievement discussed above, most notably, the delay in setting up the PMU. The project was delayed by one year because MAAIF had not been able to set up the PMU. The PMU was eventually set up directly by UNDP as agreed by the Project Board in June 2017.[9]
- Project finance and expenditure was reported on annually. The evaluators found high variances between planned and actual expenditure by year, particularly in the first year. This can be attributed to the delays experience in the first year where virtually no activities were implemented. Huge discrepancies exist between co-financing committed at project design and actual co-financing realized during project implementation. Complete co-financing expenditure information was not received for the evaluation and thus the evaluators can only assume that either reporting was weak, or co-financing in the form that was committed was not realised.[10]
- The grant mechanism, although effectively implemented overall, posed some administrative challenges, and the project team highlighted that in hindsight it would have been more efficient (and opened up more time to do more technical aspects) to have used an intermediary to administer the grant.[11]
- M&E plan included the basic requirements, although the results framework would have benefitted from some minor improvements on the output-level indicators, and the inclusion of outcome-level indicators in the results framework.[12] Implementation of M&E was done through quarterly reporting, PIR reporting and the M&E framework.
- Risks management was generally well-managed with the exception of the risks to implementation arrangements that caused unnecessary delays in some aspects of the project already covered in the paragraphs above.
- Project results and impacts: Outcome 1 has largely been successfully achieved. Land-use plans were developed for all parishes and these have been mainstreamed into district development plans in all three districts. Clause adoption has achieved more limited success, although evidence suggests that their adoption is ongoing beyond project closure. Monitoring and enforcement mechanisms have been put in place and integrated into district annual workplans.[13] Outcome 2 has had been very successfully achieved in terms of the empowerment aspect and demo sites with communities, but only partially achieved on some of its outputs by project closure. Implementation delays affected particularly Output 2.1 which was not achieved within the timeframe of the project (although it is clear from the evidence provided to the evaluators that workplans and commitments had been made to finalise this post-project). Some achievement had been made towards strengthening public-private collaboration to improve farmers’ access to inputs, but the Action Plan was not fully developed or implemented. The implementation of SLM and SFM at community level was highly successful and evidence suggests uptake, replication, and sustaining of project results. Monitoring frameworks for carbon sequestration and soil erosion had been developed and put in place, but implementation is limited due to low capacity at district level. A best practices document was not developed, although best practices from the SLM interventions have been replicated and upscaled into neighbouring areas of project intervention sites.[14]
- Relevance: The project was well-aligned to country priorities at government level (predominantly through the NDPII and ASSP), and within the UN country framework (UNDAF and CPAP). Stakeholder engagement during project implementation was strong (e.g. representation on the project board, representation at Inception Meeting including update of recommendations from stakeholders into project implementation such as the direct implementation of some outputs by communities, level of engagement of local-level government).
- Effectiveness: The project did not manage to fully achieve on all its outputs. Some activities that could have been realistically achieved within the project timeframe were not because of implementation-related delays (such as the action plan for public-private collaboration, the FFS training, the best practices documentation), other activities were overly ambitious (such as the carbon monitoring framework). Despite this, the project managed to make some impactful achievements in terms of the wider Theory of Change, particularly in relation to community empowerment and uptake towards SLM integration into farming to enhance resilience and ecosystem health in the project areas of Mt Elgon.[15]
- Efficiency: The project faced several delays including losing the first year because the PMU was not set up, which ultimately resulted in the project not achieving some of its outputs within the project timeframe despite the granting of an 18-month no-cost extension. There were some variances between years, but the project was generally cost-efficient in terms of its expenditure in relation to outcome.[16]
- Overall project outcome: Considering the above paragraphs, overall achievement of outputs is moderately satisfactory. Some elements were highly successful, especially the community engagement, the use of community championships, gender empowerment through women leadership of community implementation of SLM and SFM activities. The landscape planning and management processes were certainly an improvement from what was there before (the project managed to have digitized, and integrated land use planning conducted for all 33 parishes). Local communities were empowered to apply technologies and approaches to reverse land degradation and reduce GHG emissions through the adoption of technologies in the project areas, but also through the replication of neighbouring communities who saw for themselves that the technologies worked and supported improvements in livelihoods.
- Sustainability: Several examples of financial commitment suggest that project results will be sustained, especially at community level.[17] Empowerment of communities, including women empowerment, has resulted in community-championship of SLM interventions and further uptake; government at various levels has seen the success of community engagement and has demonstrated political support for further uptake.[18] In terms of governance and legal frameworks, land use plans and by-laws are progressing forward, the inter-ministerial taskforce is a good reflection of a wider and more programmatic approach to SLM and wider adoption of project successes have been demonstrated through a NEMA-led project to upscale SLM into the wider Mt Elgon landscape.[19] Environmental sustainability has been improved through the project in relation to the environmental risks encountered prior to project implementation.[20]
- Gender empowerment: Gender empowerment was particularly strong in this project as demonstrated by the level of women leadership and championship of SLM interventions in the project areas. The Gender Action Plan was developed late (half-way through project implementation) and the project would have benefitted from an action plan of this level of quality if it been done at PPG phase. However, the evaluators found that women empowerment particularly was demonstrated in the monitoring of the results framework (particularly at community level).
Conclusions
- The project faced implementation challenges that put project results attainment at risk. Unnecessarily delaying some output implementations until the very last months (March – August 2020) of an already maximised project time-frame[21] increased vulnerability of the project to external risks, which is exactly what happened when the COVID-19 pandemic restricted project implementation in the final months of the project. In fact, the project should have been under final operational closure in these months (and not in “starting phases” of output implementation). If it had been, it would not have been as affected by COVID-19 as it was.
- The finalisation of Output 2.1 (the FFS training) merits discussion, because the evaluators were provided with evidence that in fact funding was committed (funds transfer dated 26 August), and MAAIF commitment through work plans signed off by the Permanent Secretary had been approved by the Project Board, all before project closure. This evidence was provided during the finalisation of this TE report, and shows that activities of this output will be finalised by the end of the year (2020). However, the international evaluator is limited to assessing the project results attainment within the timeframe of the project (which officially closed 31 August 2020). Whether the output will or will not be achieved beyond project closure cannot be assessed by the TE. GEF rules stipulate that no extensions can be provided beyond the 18 months that had already been provided to the project. A recommendation is made by the evaluator in the finalisation of project results for specific outputs that were not achieved within the timeframe of this project, but this is placed in the context of GEF rules and regulations, and the IA, EA and GEF will need to take this matter forward accordingly. Specific recommendations are made below in relation to the finalisation and sustaining of some outputs that were not finalised within the project time frame (Recommendation Category A below).
- The Government of Uganda had made a large co-financing commitment in project design of which the majority was not realised, at least in terms of expenditure reporting made available to the evaluators. Similarly, expenditure reporting for the majority of the co-financing contribution from UNDP was also not available for the evaluators to assess where exactly the co-financing was used to achieve project results. GEF funding is supposed to be an incremental contribution, not the core contribution, towards what are essentially government-led projects. GEF additionality is a new section for evaluators to consider in GEF-funded project evaluations, and future project design will be focusing more on co-financing commitments and realisations thereof in project implementation. Co-financing that is committed at project design should be reported on, and a lesson from this project has been included in the lessons learnt for future project design.
- UNDP had to take over some of the executive functions of MAAIF during project implementation, including the setup of the PMU and direct recruitment and sometimes even management of project outputs. This leads the evaluators to question why the HACT capacity assessment that was used for project design had not picked up some limitations of MAAIF as an executing partner. Similarly, recruitment of appropriate staff and consultants proved difficult (e.g. FFS, gender). On the other hand, MAAIF was convinced that it had the in-house capacity to conduct the FFS training, and there was resultant to-and-fro between UNDP and MAAIF about whether external (or FAO) consultants should be used or not. Improved capacity assessment (HACT) of the EA may have supported a better understanding of this capacity or limits thereof.
- In summary, even with the project extension, some outputs were not fully delivered. Some other outputs demonstrated successful (or, in some cases over-) achievement. Results overall in terms of the project’s results framework were not fully achieved.
- That said, the project demonstrated achievements in terms of its outcomes and overall objective within the framework of the reconstructed Theory of Change that illustrates that the project was impactful in what it set out to achieve within the broader aims of SLM in Mt Elgon.
- The project aimed to decrease land degradation and enhance ecosystem health by using landscape management approaches and SLM, SFM and CCM technologies with communities to improve livelihoods and enhance resilience among people and ecosystems. The two strategic, higher-level questions guiding the evaluation linked to its two outcomes were (a) did the project success in integrating and improving landscape planning and management processes in the three project districts? and (b) did the project contribute to empowering communities in Mt Elgon to manage their production landscapes in an integrated manner?
- Based on the evidence provided to the evaluators of uptake and integration of land use planning in the districts, the integration of monitoring frameworks to support measuring land condition improvements, and the successful uptake and ownership at community-level of the SLM and SFM demonstrations, as well as the replication into areas outside of project intervention, it seems that the project did indeed contribute substantially to both improved landscape planning and improved community management of their production landscapes. The extent of this has been demonstrated in the results section of this report. The recommendations provided below will further strengthen this contribution toward impact and sustainability.
- Additionally, and in particular reference to community championship, the evaluators believe that the community-led approach had a significant impact in terms of sustaining project result, and move to impact in the TOC. This is evidenced by strong community ownership and uptake, longer-term savings schemes generated (through training and community empowerment as a result of allowing communities to lead activities instead of one company), and thus sustainability of results at least in terms of community-related SLM, SFM and CCM interventions.
- The project fit well within the larger programmatic approach of the Government (in relation to its inter-ministerial task force on SLM) relevant to country development priorities (e.g. NDPII) and should be seen as a leverage point for further catalytic action towards more systems and holistic SLM across wider landscapes.
- The evaluators believe that the community-led approach had a significant impact in terms of sustaining project result. This is evidenced by strong community ownership and uptake, longer-term savings schemes generated (through training and community empowerment as a result of allowing communities to lead activities instead of one company), and thus sustainability of results at least in terms of community-related SLM, SFM and CCM interventions.
- Based on the above considerations, the project, overall, is given a rating of Moderately Satisfactory, with the summary table provided below (see Table 4 in main report for description of rating system).
|
ToR for Terminal Evaluation of _ILM Project_International Consultant.pdf
ToR of Terminal Evaluation of _ILM Project_National Consultant.docx