- Evaluation Plan:
- 2016-2017, UNCDF
- Evaluation Type:
- Mid Term Others
- Planned End Date:
- 10/2017
- Completion Date:
- 10/2017
- Status:
- Completed
- Management Response:
- Yes
- Evaluation Budget(US $):
- 110,000
Mid-term Evaluation of the CleanStart Programme Accessing Clean Energy for the Poor
| Document | Type | Language | Size | Status | Downloads |
|---|---|---|---|---|---|
 CleanStart mid-term evaluation_Final Report .pdf |
report | English | 2583.82 KB | Posted | 451 |
| TOR_CleanStart_Mid-Term evaluation_.pdf |
tor | English | 1241.51 KB | Posted | 172 |
| Title | Mid-term Evaluation of the CleanStart Programme Accessing Clean Energy for the Poor | ||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Atlas Project Number: | |||||||||||||||||||||||||||||
| Evaluation Plan: | 2016-2017, UNCDF | ||||||||||||||||||||||||||||
| Evaluation Type: | Mid Term Others | ||||||||||||||||||||||||||||
| Status: | Completed | ||||||||||||||||||||||||||||
| Completion Date: | 10/2017 | ||||||||||||||||||||||||||||
| Planned End Date: | 10/2017 | ||||||||||||||||||||||||||||
| Management Response: | Yes | ||||||||||||||||||||||||||||
| Focus Area: |
|
||||||||||||||||||||||||||||
| Corporate Outcome and Output (UNDP Strategic Plan 2014-2017) | |||||||||||||||||||||||||||||
| Evaluation Budget(US $): | 110,000 | ||||||||||||||||||||||||||||
| Source of Funding: | |||||||||||||||||||||||||||||
| Evaluation Expenditure(US $): | 109,920 | ||||||||||||||||||||||||||||
| Joint Programme: | No | ||||||||||||||||||||||||||||
| Joint Evaluation: | No | ||||||||||||||||||||||||||||
| Evaluation Team members: |
|
||||||||||||||||||||||||||||
| GEF Evaluation: | No | ||||||||||||||||||||||||||||
| Key Stakeholders: | |||||||||||||||||||||||||||||
| Countries: | CAMBODIAETHIOPIAMYANMARNEPALUGANDA |
| Lessons | |
|---|---|
| Findings |
| Recommendations | |
|---|---|
| 1 | |
| 2 | Evaluation Recommendation or Issue 1: 1. STRATEGY LEVEL The programme should continue with the piloting approach, and: a. Concentrate on the five existing countries of implementation; b. Focus on the development and research of specific initiatives at global and regional level to share best practices, capitalize lessons learned and try to attract additional funds; c. If possible within this programme, or eventually in future interventions, distinguish objectives and expected results between direct and indirect interventions to attribute a different weight to the expected contribution to the programme (and consequently to monitor their achievements); d. Starting from the existing country business plans (or market assessments), to design (or revise) a specific action plan for each country of implementation in order to establish the ‘appropriate’ entry level and the best sequence to upstream and/or downstream connections at different levels to identify and address market bottlenecks, challenges and opportunities; and programmes in order to establish clear institutional relationships in the countries and to avoid gaps or overlapping amongst the Programme Implementation Units. |
| 3 | Evaluation Recommendation or Issue 2: OUTREACH LEVEL Focus also on a broader target definition, i.e. micro-entrepreneurs (as also declared in the final outcome) and community economies (namely centers of aggregation such as hospitals, schools, market places, etc.) with a view to accelerate programme accomplishment and increase awareness in communities. In addition, consider to improve the definition and monitoring of household’s level of income or poverty rate to ensure of reaching the expected segment of population (low income households). |
| 4 | Evaluation Recommendation or Issue 3: GOVERNANCE LEVEL Reinforce the current Investment Committee with a dedicated Committee (Advisory or Steering Committee) to maintain a more effective relationship with donors, global stakeholders and other relevant international energy initiatives towards strengthening the programme’s governance structure, gaining larger participation of relevant international actors and building capacity to attract additional funds. |
| 5 | Evaluation Recommendation or Issue 4: MANAGEMENT LEVEL a. Develop more diversified supporting tools in terms of: i) financial tools that apart from grants also include other funding mechanisms (such as senior debt and/or sharing risk credit and guarantee funds); and ii) capacity building by first assessing the effective needs of the institutions and then providing targeted assistance and/or training; b. Improve the monitoring and reporting system by including programme achievements beyond the micro level; c. Encourage implementing partners to identify and promote projects based on innovative and strategic core-business models avoiding to support projects that are residual activities respect to their main business; d. Reorganize the Programme Implementation Unit through increased presence at country level of dedicated staff (preferably local). |
1. Recommendation:
Management Response: [Added: 2017/12/06]
Overall comments: Although it was clearly a challenge for the Mid-Term Evaluation team to account for the complex and evolving ecosystem and material constraints that the CleanStart (CS) Programme operates in, and that CS (as part of UNCDF), promotes a market development approach, the management of CS recognizes the value of this exercise not only in terms of accountability, but also for advice on moving forward. Taking this into account, CS hopes to continue influencing and play an active role in how UNCDF strategically use its catalytic investment mandate to accelerate market development in LDCs thereby working toward the objectives of SDG 7 and others goals.
CS had some challenges to achieve ambitious targets for the following reasons: i) The program started seriously underfunded, with only 30 percent of the budget funded (there is still a funding gap today of USD11Million out of USD26Million), ii) by delays in program implementation in several countries, given the ambitious and comprehensive nature of the CS approach and, iii) the incorrect assumption that traditional Microfinance Institutions were best placed to lead the energy access push—which forced a change in approach by including energy service providers more aggressively in the strategy. More realistic targets (especially on the intermediate output level) can now be set based on a more flexible approach, experience and track record. It should be noted, however, that as of end 2016—with revised timelines—the outreach target was basically on track.
The MTE noted that limited capacity building support was provided to partners. Although recognizing that capacity building can be improved qualitatively, the CS team does not fully agree with this statement. Extensive pre-investment TA is provided to all partners (FSPs, ESPs), irrespective of the approach (sector-based, investment), particularly on how to develop a commercially-viable business model and an implementable business plan. Further, extensive implementation TA through trainings and one-on-one advice have been provided to financial service provider partners in Nepal and in Africa through the PAMIGA partnership—with several training tools for MFIs development. The “investment” approach which CS has recently preferred, by nature, embeds the external TA requirements into the business plan (if deemed required through company’s own thinking and TA/due diligence discussions with CS team) so that the company takes ownership, and gets the type of support when the company most needs it. But still, for the “investment” partners, CS provides value-added support, such as referrals to product/service provider partners such as digital payment service providers, solar providers; face-time with Government to discuss business challenges; linkages to funders; participation in customer research.
Responding to the MTE statement that CS has modest influence on broader system and multiplier effect, the CS team would respond that CS has been actively developing other pieces of the market system (macro, meso) through convening country-level market stakeholders (Think-shops on targeted market development topics), and providing TA to national funds dedicated to energy lending (Central Renewable Energy Fund in Nepal, Development Bank of Ethiopia) so that working capital needs are at a much larger scale. In Nepal, CS is a close partner of the Alternative Energy Promotion Center, who looks to CS to provide advice on financing whether that is CREF or the GCF application. In Uganda, CS has a reputation for having a finger on the pulse when it comes to the energy access market and latest business models; it has since been selected as a SDG Financing For Development pilot by UNDP to demonstrate new approached to blended finance. CS represents multilateral organizations on the Ministry of Energy and Mineral Development and DFID-led Steering Committee to develop the household solar sector, and now approached by development partners to support the clean cooking and solar energy associations. Given modest resources in the programme it was not possible to adopt comprehensive intervention strategies for Myanmar and Cambodia in the same manner as Nepal and Uganda, for instance. However, even in Cambodia, CS has provided substantial inputs to the design of the UNDP-GCF solar programme.
Global level activities, such as engagement with SE4ALL and the Global Off-Grid Lighting Association and IFC on sector KPIs should have been reflected more strongly in the analysis. CS has played (and is playing) a constructive and sector-supporting role here. MACRO seems un-necessarily confined to country level. It is mentioned in the report, but not further analyzed, although information on this was made available in time. GOGLA nor IFC/World Bank (global), two major global players in the sector that know CS could have provided commentary on CS, its role and collaboration with these two entities to create an enabling environment and performance standards for the off-grid solar sector.
Overall, the MTE was coordinated well with the CS team, although communications on logistics and final review of the draft document could have been better managed to minimize transaction costs and time on all sides. The CS team wishes to thank the UNCDF evaluation unit for its continuous guidance and its professional, objective advice throughout the process.
Key Actions:
2. Recommendation:
Evaluation Recommendation or Issue 1: 1. STRATEGY LEVEL
The programme should continue with the piloting approach, and:
a. Concentrate on the five existing countries of implementation; b. Focus on the development and research of specific initiatives at global and regional level to share best practices, capitalize lessons learned and try to attract additional funds;
c. If possible within this programme, or eventually in future interventions, distinguish objectives and expected results between direct and indirect interventions to attribute a different weight to the expected contribution to the programme (and consequently to monitor their achievements);
d. Starting from the existing country business plans (or market assessments), to design (or revise) a specific action plan for each country of implementation in order to establish the ‘appropriate’ entry level and the best sequence to upstream and/or downstream connections at different levels to identify and address market bottlenecks, challenges and opportunities; and programmes in order to establish clear institutional relationships in the countries and to avoid gaps or overlapping amongst the Programme Implementation Units.
Management Response: [Added: 2017/12/06]
Management Response: a. Where possible, CS will strive to do this. However, this also depends on the funding is available; in many instances funding is earmarked for specific countries or regions, limiting the amount flexibility in terms of deploying in those countries. In some countries, the ecosystem for energy access financing may also change quite suddenly (for example, the case of Myanmar, where a subsidy programme brought severe difficulties for several of the CS partners and where our overall position in the market would need to be re-assessed). NO SPECIFIC ACTION DEEMED REQUIRED. b. CS’s energy customer journey research outputs (energy ladder, energy and financial diaries, client impact study) will soon be publicized. The calibre of the data partners, research team, and advisory committee (e.g. GOGLA, GSMA, Acumen Fund) should be a testament to how innovative the research is better understanding customers. Several further research initiatives are in the pipeline, some also considering big data. CS Connect (event) and Connections (magazine) will be revitalized in more bite-sized form, with more emphasis on local events and social media to share best practice. c. Indeed, this is taken on board, with more attention to be given to DCED and similar monitoring and reporting systems that might be more appropriate for CleanStart, with more of an emphasis on market development approaches. Work by the UNCDF SHIFT programme, which was also instrumental in providing support to the CS challenge fund activities can assist with this (it recently contributed to the development of the newly released Measuring Market Development Handbook by CGAP). d. A country tracking framework, taking into account these different levels and clear CS entry points, as well as interaction with strategic partners will be established. This will be a light, live document that will be adjusted as programmatic realities evolve—which can also be used for reporting and board information.
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| 1.a. Publish three customer journey research pieces.
[Added: 2017/12/06] [Last Updated: 2018/05/16] |
PMU | 2018/03 | Completed | CleanStart published three blog pieces on customer journey. History | |
| 1.b. Re-activate Connect (local or international) events and Connections series (choosing one of the following platforms: publications, modest case-studies, information sharing—primarily using social media).
[Added: 2017/12/06] [Last Updated: 2019/03/11] |
PMU | 2019/12 | Initiated | Series of think-shop events organized in Uganda. Updates and lessons learned from investments shared through blogs. Blog and twitter reads increasing. Case studies for Window 1 partners completed and being disseminated now until April/May 2019. Planning to organize Connect event end of 2019. History | |
| 1.c. CleanStart will adjust its results measurement system and reporting to include Market Development Measurement best practice indicators.
[Added: 2017/12/06] [Last Updated: 2018/12/14] |
PMU | 2019/12 | Initiated | KPI and results measurement framework updated. RM system will continue to be updated to reflect programming in new countries and new interventions such as LDCIP investments, DRC, and support to industry association. History | |
| 1.d A country tracking framework prepared for Uganda.
[Added: 2017/12/06] [Last Updated: 2018/10/05] |
PMU | 2018/03 | Completed | As part of the RECF logframe and results tracking framwork History | |
| 1.e A country tracking framework prepared for other countries where CleanStart is active with critical mass.
[Added: 2017/12/06] [Last Updated: 2018/10/05] |
PMU | 2018/06 | Completed | As of now, only Uganda has critical mass History |
3. Recommendation:
Evaluation Recommendation or Issue 2: OUTREACH LEVEL
Focus also on a broader target definition, i.e. micro-entrepreneurs (as also declared in the final outcome) and community economies (namely centers of aggregation such as hospitals, schools, market places, etc.) with a view to accelerate programme accomplishment and increase awareness in communities. In addition, consider to improve the definition and monitoring of household’s level of income or poverty rate to ensure of reaching the expected segment of population (low income households).
Management Response: [Added: 2017/12/06]
Management Response: Impact on poverty is complex to measure, especially midway through the programme (we note that some attempts to do so in the MTE lack robustness, given the modest sample size. PPI requires at a minimum a sample size of 30 in a given context). However, CS will strive to find what the poverty outreach of the programme is, using different tools. Assessing the use of energy for productive use (as per the SE4ALL and SDG7 measurement frameworks) will be given more emphasis, both on an individual, enterprise and collective basis. It will attempt to do so using the aforementioned measurement parameters, in combination with action point taken on 1.c above.
Awareness raising is certainly seen as a key activity for CS moving forward (a significant commitment to this has already been made in Uganda, for example).
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| 2. Key SE4ALL/SDG 7 parameters included in impact metrics (in combination with action point 1.c.
[Added: 2017/12/06] [Last Updated: 2018/12/14] |
PMU | 2018/12 | Completed | Impact indicators mapped against SDGs 1, 5, 7, 8, 13, 17 and tracked through quarterly/annual KPIs reported by partners and through qualitative surveys with customers conducted by UNCDF History |
4. Recommendation:
Evaluation Recommendation or Issue 3: GOVERNANCE LEVEL
Reinforce the current Investment Committee with a dedicated Committee (Advisory or Steering Committee) to maintain a more effective relationship with donors, global stakeholders and other relevant international energy initiatives towards strengthening the programme’s governance structure, gaining larger participation of relevant international actors and building capacity to attract additional funds.
Management Response: [Added: 2017/12/06] [Last Updated: 2017/12/06]
Management Response: Indeed, this is an area for strengthening and is accepted as such. Attempts to include key donors in the global programme board structure have not been successful thus far, but new steps to expand board members to peer organizations, industry representatives or other leaders in the field will be made. Given the ability to attract a high-quality investment committees for the CS challenge fund rounds, candidates from these investment committees may be sought. That said, engagement with a whole variety of partners, including donors has worked very well at the country level, which may also offer another way to engage in the governance structure. For example, in Uganda, UNCDF is part of the DFID-sponsored energy compact
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| 3. Board expanded from current level to include at least two additional external stakeholders (e.g. donor, academic/industry expert, investor/industry representative, etc…..)
[Added: 2017/12/06] [Last Updated: 2018/12/14] |
PMU | 2019/06 | Initiated | Sweden member of board since June 2018. Will take stock of CleanStart’s strategic direction with current Board members in Jan 2019. Then will proceed to recruit 1 new member that can best steer the Programme to maximize strategic priorities identified by current Board members. History |
5. Recommendation:
Evaluation Recommendation or Issue 4: MANAGEMENT LEVEL
a. Develop more diversified supporting tools in terms of: i) financial tools that apart from grants also include other funding mechanisms (such as senior debt and/or sharing risk credit and guarantee funds); and ii) capacity building by first assessing the effective needs of the institutions and then providing targeted assistance and/or training; b. Improve the monitoring and reporting system by including programme achievements beyond the micro level; c. Encourage implementing partners to identify and promote projects based on innovative and strategic core-business models avoiding to support projects that are residual activities respect to their main business; d. Reorganize the Programme Implementation Unit through increased presence at country level of dedicated staff (preferably local).
Management Response: [Added: 2017/12/06]
Management Response: a.i. UNCDF has recently operationalized its LDC Investment Platform that allows it to more aggressively push a number of different funding instruments such as loans and guarantees. SIDA has agreed to provide additional funding to CS to test some of these instruments within 2018, which it intends to do—in partnership with the LDC IP. a.ii. CS aims to do more work on Capacity Building, Knowledge and Policy Advocacy. Initial work on this has already been done, especially in Uganda and Nepal. Extensive capacity building activities are planned in Ethiopia with UNDP funding. A concept note has been submitted to DFID and NORAD that have strong components for Uganda by doing this through industry associations. b. Although CS does have various and robust tools to monitor its performance (staff monitoring system, a partner KPI reporting system, and independent research), it recognizes that more can be do strengthen the overall coherence of its monitoring. In response, CS has set up a qualitative database on NVIVO to conduct systematic case study analyses on best and weak practices of partners. CS will implement lessons learned from previous challenge fund rounds into the next window. KPI systems currently operational will be also simplified. Distinction between direct and indirect interventions needs to be worked on. This should all combine to improve the monitoring and reporting system by including programme achievements beyond the microlevel and into the portfolio level. Focus on the development and research of specific initiatives at global and regional level to share best practices will be made, to capitalize lessons learned, make policy interventions and try to attract additional resources. c. The challenge fund app explicitly supports innovation—typically in terms of new business models, approaches to groups or geographies not covered earlier, or different partnership arrangements in a market (for example between an energy service provider and a bank). However, given that this entails a high risk of failure also, it has taken steps to capture lesson from failure also. Further, it will seek to adopt a mixed portfolio of investments, with some highly innovative, high risk investments, mixed with investments into projects that can real opportunities for scale. NO SPECIFIC ACTION DEEMED REQUIRED. d. For countries where these is a critical mass of activities and funding, this has been practice (for example Nepal) and continues to be a priority. Uganda is well funded into 2020 and this will allow the team to recruit a national project coordinator. The recruitment process will start within 2017, with the aim of having person on board early in 2018. Similar steps will be taken for countries/regions which are able to secure resourcing for this. For countries which have less resources, a more flexible staffing approach will be taken
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| 4.a.i) CleanStart will provide at least three loans or guarantees
[Added: 2017/12/06] [Last Updated: 2018/12/14] |
PMU and LDC-IP | 2018/12 | Completed | 3 Loans and 1 guarantee to three companies in Uganda. One more in the pipeline in Nepal to be completed in Q1 19. History | |
| 4.a.ii) CleanStart will undertake structured capacity building activities ESPs and FSPs in Uganda and Ethiopia focusing on institutional strengthening and financial planning.
[Added: 2017/12/06] [Last Updated: 2018/12/14] |
PMU | 2019/12 | Initiated | TA needs assessments and investment-readiness training completed with Uganda portfolio. Trainings organized with energy and financial service providers in Ethiopia so that more providers can submit viable business plans to Development Bank of Ethiopia (CleanStart’s IP). Capacity building activities will continue throughout the life of the programme. History | |
| 4.b) Unified KPI system to be in place for all active CS countries
[Added: 2017/12/06] [Last Updated: 2018/10/05] |
PMU | 2018/06 | Completed | Unified KPI software used for all current CS investees. History | |
| 4.d) National project coordinator to be hired in Uganda
[Added: 2017/12/06] [Last Updated: 2018/12/14] |
PMU | 2018/12 | Completed | Recruitment in final stages i.e. offer has been made. History |