Terminal Evaluation for Improved Charcoal Production project

Report Cover Image
Evaluation Plan:
2016-2020, Uganda
Evaluation Type:
Final Project
Planned End Date:
01/2020
Completion Date:
01/2020
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
30,000

Share

Title Terminal Evaluation for Improved Charcoal Production project
Atlas Project Number: 00074620
Evaluation Plan: 2016-2020, Uganda
Evaluation Type: Final Project
Status: Completed
Completion Date: 01/2020
Planned End Date: 01/2020
Management Response: Yes
UNDP Signature Solution:
  • 1. Energy
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 1.5.1 Solutions adopted to achieve universal access to clean, affordable and sustainable energy
SDG Goal
  • Goal 7. Ensure access to affordable, reliable, sustainable and modern energy for all
SDG Target
  • 7.1 By 2030, ensure universal access to affordable, reliable and modern energy services
  • 7.2 By 2030, increase substantially the share of renewable energy in the global energy mix
Evaluation Budget(US $): 30,000
Source of Funding: GEF
Evaluation Expenditure(US $): 30,000
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
Veronica Nyawira Muthui Lead Consultant nyawira.muthui@gmail.com KENYA
Cliff Bernard Nuwakora Consultant cliff.nuwakora@gmail.com UGANDA
GEF Evaluation: Yes
GEF Project Title: Improved Charcoal Production Technologies and sustainable land management practices through an integrated approach
Evaluation Type: Terminal Evaluation
Focal Area: Multifocal Areas
Project Type: FSP
GEF Phase: GEF-5
GEF Project ID: 4644
PIMS Number: 4493
Key Stakeholders: Ministry of Energy
Countries: UGANDA
Comments:

See attached formal communication including project board minutes requesting for extension, partner letters and HQ response/approval.

Lessons
1.

Although the theory of change of the project on sustainable charcoal was weak, the project sought to put in place enabling conditions (institutional, policies and legislation, skills and technologies, incentives) for the widespread uptake of sustainable charcoal production processes nationally, which it effectively piloted in four districts. At TE, the project has delivered significant outcomes and contributed to the goal, by delivering some environmental benefits.

On advancing the use of knowledge and current information in decision-making on charcoal value chain, the national charcoal assessment was completed and has provided up to date information on the charcoal value chain. Both the BEST and NAMA have utilized the information provided. A data base has been established and an equipped sustainable charcoal research laboratory is spearheading research on sustainable charcoal.

On improving institutional coordination and legal provisions to mainstream sustainable charcoal production process into relevant district and national policies and programs (and increasing funding for charcoal value chain), the project has established a national level stakeholder coordination; it has produced District Charcoal Action Plans which have been mainstreamed into District Development Plans; thus sustainable charcoal is now recognized in the District budgets of these four districts.

On dissemination of technologies for sustainable charcoal production and climate smart agriculture, forty two Charcoal Producer Associations have been facilitated and registered, with over 800 members (40% women) across the 4 project districts. Three hundred and thirty seven (337) casamance and 15 retort kilns disseminated; dissemination of the Adam retort kiln was rightly stopped after the first 15 failed to function fully. There is a 28% increase in income per bag of charcoal produced via casamance. In addition, there is 100% increase in yields of annual crops and 28% in perennial crops.

It has contributed to environmental benefits. 120,741 metric tons of wood have been saved from the adoption of the casamance kiln and skills. This translates to 6,674 ha of avoided deforestation. Although both achievements are at 84.2% of the target, the TE Team concurs with the MTR that these targets were too ambitious.  30,621 hectares of forest land (natural and planted forest lands) (84.2% of target) have been put under improved management, enhancing carbon sequestration of 1,310,872 metric tons of carbon equivalent, delivering 84.2% of the target tCO2eq so far. However, the target of 2,100,000 tCO2eq is to be realized over a twenty-year period. 


Findings
1.

Criteria

Rating of this project

Remarks

IA&EA Execution: Satisfactory (S)

Overall quality of project implementation/execution

Satisfactory

There was strong management by the Board, which provided adequate oversight role and policy direction and included representatives from key stakeholders. However, coordination  roles and responsibilities were not sufficient, especially between upstream national Implementing Partners with downstream Local governments.

Implementing Agency execution

 Satisfactory

Executing Agency execution

Satisfactory

Monitoring and Evaluation (M&E):

Overall quality of monitoring and evaluation

Moderately Satisfactory

Whereas monitoring and evaluation endeavored to follow the UNDP GEF guidelines, the management hiccups as a result of National Implementation Modality (NIM) weakened the effectiveness of the M&E function. Whereas there was a Project Management Unit (PMU), decisions on project processes followed government structures and systems, which is often too slow for a project with a tight deadline.  Combined with the fact that there was no M&E Officer for the project, this weakened effectiveness of the M&E system. However, provision of an M&E tool by the PMU mitigated some of the weakness, and became an innovative step that augmented the M&E systems.

M&E design at project start up

Moderately Unsatisfactory

The design of the project M&E system fell short of the requirements as per GEF guidelines. This is because it had too many indicators with weak links between indicators, baseline values and targets. In addition, there was no provision for a project-specific M&E officer who would have refined the M&E system during the project implementation or consolidated the knowledge management for the project.

M&E plan implementation

Satisfactory

Various Stakeholders were involved UNDP, MEMD, MWE, NFA, DLGs and Political arm (Office of the President). Notwithstanding the inadequacies in M&E, the project management and Board were able to mitigate the weaknesses and risks, to deliver the project results in a Satisfactory manner.

Relevance: Relevant  (R), Not Relevant (NR)

Overall relevance of the project

Relevant

The Green Charcoal project was highly relevant to the development needs of Uganda, given the importance of charcoal as a source of energy and the need to reduce deforestation from the charcoal value chain, through a managed process combining adoption of efficient technologies and inclusion of energy plantations as part of natural resources management at the household and forestry sector levels.

GEF and UNDP strategic alignment

Relevant

The Green Charcoal project is highly relevant to GEF and UNDP strategic objectives. It contributed to three GEF Focal Areas (Climate Change Mitigation, Sustainable Forest and Land Management), contributing to four Focal Area Objectives and seven outcomes (Table 9). It contributed to Outcome 2 of the UN Development Assistance Framework (UNDAF) for 2012-2014[1] and outcome 2.3 of the 2010-2014 UNDP CPAP[2]

National policy frameworks and ownership

Relevant

The Green Charcoal project as indicated extensively in the prodoc is relevant to various national regulatory, policy, strategic and development plans such as the National constitution, NDP, National Environment Act, National forestry act, Local government Act, Land act, National Forestry and Tree planting, National forest plan as well as all relevant policies etc.

3. Assessment of Outcomes

Overall Quality of Project Outputs and Outcomes

Satisfactory

To overcome the challenge of evaluating the twelve outcomes of the project, the TE reconfigured 4 core outcomes, which it used to assess performance (Table 6). The project has delivered on all the majority of outputs (75% implementation level) and delivered satisfactorily on outcomes. As a result, 120,741 metric tons of wood have been saved from the adoption of the casamance kiln and skills. This translates to 6,674 ha of avoided deforestation. Although both achievements are at 84.2% of the target, the TE Team concurs with the MTR that these targets were too ambitious.  30,621 hectares of forest land (natural and planted forest lands) (84.2% of target) have been put under improved management, enhancing carbon sequestration of 1,310,872 metric tons of carbon equivalent, delivering 84.2% of the target tCO2eq so far. A total of 800 beneficiaries including 240 women in the pilot districts have been equipped with skills to efficiently utilize the improved charcoal production technologies and conservation agriculture practices. Adoption of climate smart agriculture (61% women) has led to over 100% increase in yields of annual crops and 28% for perennial crops. Approximately 300,000 households (2.5 million persons – M:1,700,00; F:800,000) have been sensitized on charcoal regulatory frameworks and guidelines through 116 live radio talk-shows and radio spot messages, community meetings and multi-stakeholder dialogues facilitated by the project.

Outcome 1: Knowledge systems established and used to provide up to date information for planning for the charcoal sub-sector

Satisfactory

National charcoal assessment completed. Both BEST and NAMA  have utilized the information provided;

 

The sustainable charcoal research laboratory has been established and equipped; it is undertaking research using staff and graduate students.

 

Analogue database in place (needs to be digitized); land cover maps have been provided; but they need to be converted to land use maps.

Outcome 2: Institutional coordination and legal provisions established to mainstream sustainable charcoal production process into relevant district and national policies and programs (increasing funding for charcoal value chain).

Satisfactory

National level stakeholder coordination put in place; District Charcoal Action Plans in place and mainstreamed into District Development Plans (sustainable charcoal is recognized in the District budgets of the four districts); national charcoal standards have been approved and district charcoal ordinances have been finalized; more than 30 million US dollars have been raised to support the work of the new Renewable Energy department and priorities of BEST; NAMA completed which is likely to add over 60 million USD when funded, $50 million from the private sector.

Outcome 3: Adoption of technologies for sustainable charcoal production and climate smart agriculture, supported by local capacities and institutions

Moderately Satisfactory

337 casamance and 15 retort kilns disseminated; dissemination of the Adam retort kiln rightly stopped after the first 15 failed to function fully;

Forty Charcoal Associations with over 800 members (40% women) have been established and registered across the 4 project districts.

28% increase in income per bag of charcoal produced via casamance

100% increase in yields of annual crops and 28% in perennial crops.

Outcome 4: Sustainable forest management and tree cropping support sustainable charcoal production

Moderately Satisfactory

About 5,888 hectares of well grown planted sustainable charcoal woodlots of mainly eucalyptus tree species have been established in the four pilot districts. This acreage is determined after planting 6,542,000 seedlings and factoring in the seedling survival rate of 72% and considering farmer practices.

 

About 1,800 tree planters growing trees, 18% of whom are women.

 

30,621 hectares of forest land (natural and planted forest lands) across four pilot districts have been brought under improved multi-functional forest management

Effective and efficiency:

Effectiveness

Satisfactory

At the output levels a number of indicators and targets were realized all above 75%

Efficiency

Satisfactory

The project having used output/activity based budgeting was able to institute control measures but this was bogged down by the bureaucracies that were brought about by NIM modality, which was experienced delays in disbursement of funds and approvals. However, with use of government and DLG structures and systems the project was able to mitigate the negative impacts of the government bureaucratic processes.

Partnership:

Overall partnerships built

Satisfactory

Partnerships were forged between national and district levels as well as with both private and Civil society. These were both formal and informal in nature. At the upstream national level government ministries and semi-autonomous agencies were brought on board such as MEMD, MWE, NFA, FSSD while at the downstream the DLGs and civil society organizations also formed a strong partnership well community level beneficiaries. A key partnership with the CleanStart did not function as envisaged at project design, impacting delivery of some activities.

Overall stakeholders participation

Highly Satisfactory

As intimated above stakeholder engagement involved Government, Development Partners, CSOs, Private sector and communities

Sustainability: Likely (L) Moderate Likely (ML), Moderately Unlikely (MU), Unlikely (U)

Overall likelihood of Sustainability

ML – Moderately Likely

Sustainability of the production of charcoal by improved technologies is threatened by the fact that neither the casamance nor the retort kilns are available in the markets (can only be obtained via the project); the modified retort kiln is not yet functional; although completed, the district ordinances and not yet approved; charcoal is still sold by volume in significant markets (hence no real reward for sustainably produced charcoal); and 90% of the seedlings planted are eucalyptus, which is in high demand for timber and poles. However, if the NAMA takes off, it will tackle these challenges.

Financial resources

ML – Moderately Likely

Even though charcoal has a budget line in the district development plans (in the four districts), financing the budget is likely to suffer in the regular budget shortfalls. District revenues continue to be dominated by forest products, yet sustainable charcoal discourse has not been in part of the revenue collection by contractors.  Work on charcoal continues to be largely project funded; the absence of an exit strategy for the project is particularly worrying, given that many initiatives are just starting and will need continued support.

Socio-economic and Environmental aspects

Moderately Likely

There is need to closely monitor and follow up the socio economic aspects so far realized since they are likely to vanish if the relevant improved charcoal and SLM/SFM activities are not up scaled.

Institutional systems

 Likely

The creation of the Renewable Energy Department at the MEMD, the approval of the national charcoal standards, the formulation of the District Charcoal Ordinances and the formation of the Charcoal and Conservation Farmers Associations will all go a long way in sustaining institutional arrangements for sustainable charcoal production.

Impact: Significant (S), Minimal (M), Negligible (N) There are clear indicators that would lead to great impact over time through mentioned best practices

Environmental status improvement

Significant

The Green Charcoal project highlighted and showcased improved charcoal and SLM/SFM practices which addressed the environmental concerns as well as sustainable charcoal production measures.

Social-economic status improvement

Minimum

The Green charcoal project was positive in directly addressing social and economic aspects especially with the direct beneficiaries including tackling gender issues in sustainable charcoal production by bringing women on board in the entire value chain. However short implementation period will render them hanging without being consolidated.

Overall Project Results

Satisfactory

The Green Charcoal project delivered over 75% of the outputs; delivered close to 80% of the reconstructed impact indicators. However,  the sustainability of these results will only be secured if the proposed NAMA materializes to tackle the considerable threats to sustainability.

 


[1] Vulnerable segments of the population increasingly benefit from sustainable livelihoods and in particular improved agricultural systems and employment opportunities to cope with the population dynamics, increasing economic disparities, economic impact of HIV&AIDS, environment shocks and recovery challenges by 2014. Notably: Outcome 2.2 Vulnerable communities, Government, civil society and the private sector are sustainably managing and using the environment and natural resources for improved livelihoods and to cope with the impact of climate change.

[2] CPAP Outcome 2.3:Capacity of Selected Institutions Strengthened for Sustainable Environment and Natural Resources Management (ENRM) as well as Climate Change (CC) Adaptation/ Mitigation and Disaster Risk Management


Recommendations
1

Develop an exit strategy: there are many project initiated activities that need funding to take root, deliver and/or sustain the benefits already manifesting. 

2

Prioritize and Increase budgetary allocation to the Natural Resources and Forestry Units

3

The district revenue collection system (contracting) should take on board sustainable charcoal production principles. 

4

The charcoal certification and national standards are now in place and implementation needs to be prioritized to move to labeling as well as proposals for future directions

5

The District Councils, Forest Sector Support Department (FSSD) and the MEMD should investigate further which parts of the eucalyptus woodlots will provide sustainable biomass for charcoal production versus timber and poles to the building industry. 

6

Ensure NAMA funding is mobilized/realized. 

7
  1. Knowledge Management: Produce KM products such as documentaries, videos, technical publications summarizing knowledge products/messages, to capture best practices and lessons for the future replication. 
1. Recommendation:

Develop an exit strategy: there are many project initiated activities that need funding to take root, deliver and/or sustain the benefits already manifesting. 

Management Response: [Added: 2020/01/07]

Agree

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Integrate the initiated activities in development plans for continuity and follow-up
[Added: 2020/01/07]
Project Steering Committee/MEMD 2024/12 Initiated
2. Recommendation:

Prioritize and Increase budgetary allocation to the Natural Resources and Forestry Units

Management Response: [Added: 2020/01/07]

Agree

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Prioritize the capacity building and financing of the units at the Parish levels where the services of the extension service are required
[Added: 2020/01/07]
District Authority and the Forestry Department, with support from the NAMA PSC 2024/12 Initiated This is an on-going activity
3. Recommendation:

The district revenue collection system (contracting) should take on board sustainable charcoal production principles. 

Management Response: [Added: 2020/01/07]

Agree

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Ensure sustainable charcoal initiatives are not implemented in parallel with the revenue collection activities.
[Added: 2020/01/07]
District Councils and the Forestry Department 2024/12 Initiated On-going activity
4. Recommendation:

The charcoal certification and national standards are now in place and implementation needs to be prioritized to move to labeling as well as proposals for future directions

Management Response: [Added: 2020/01/07]

Agree

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Ensure implementation is in compliance with charcoal certification and national standards
[Added: 2020/01/07]
The Renewable Energy Department at the MEMD, with support for the NAMA PSC 2024/12 Initiated On-going activity
5. Recommendation:

The District Councils, Forest Sector Support Department (FSSD) and the MEMD should investigate further which parts of the eucalyptus woodlots will provide sustainable biomass for charcoal production versus timber and poles to the building industry. 

Management Response: [Added: 2020/01/07]

Agree

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Use research information to plan.
[Added: 2020/01/07]
The District Councils, Forest Sector Support Department (FSSD) and the MEMD 2024/12 Initiated On-going activity
6. Recommendation:

Ensure NAMA funding is mobilized/realized. 

Management Response: [Added: 2020/01/07]

Agree

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Mobilize NAMA funding to ensure continuity
[Added: 2020/01/07]
The Renewable Energy Department at the MEMD 2024/12 Initiated On-going activity
7. Recommendation:
  1. Knowledge Management: Produce KM products such as documentaries, videos, technical publications summarizing knowledge products/messages, to capture best practices and lessons for the future replication. 
Management Response: [Added: 2020/01/07]

Agree

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Produce KM products such as documentaries, videos, technical publications summarizing knowledge products/messages, to capture best practices and lessons for the future replication.
[Added: 2020/01/07]
Renewable Energy Department with support from the PSC partners. 2024/12 Initiated On-going activity

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org