Success message
error message
warn message
Mid-Term Evaluation: Sustainable Energy Solutions for Rural Livelihoods in DPRK (SES)
Commissioning Unit: DPR Korea
Evaluation Plan: 2011-2019
Evaluation Type: Project
Completion Date: 08/2018
Unit Responsible for providing Management Response: DPR Korea
Documents Related to overall Management Response:
 
1. Recommendation:

Recommendations to Improve the Sustainability and Impact of Results

R1. Contingency planning for RE activities

In light of the continue delays in procurement approvals for materials for RE installations, it is recommended that the project develop a contingency plan whereby no RE activities can be implemented, refocusing on wider implementation of EE installations and consolidating knowledge gains among engineers and decision makers at the provincial and county level in order to prepare for (eventual) scale-up of the initiative by the government, including supporting such activities as in-country study tours, and bringing together national partners, including NTDCs to discuss and share lessons and areas for intervention. Such a contingency plan would allow the project to capitalize on both the interest of communities in EE technologies, as well as the ownership of the government by sustaining momentum in activity implementation.

Management Response: [Added: 2018/09/20] [Last Updated: 2020/12/05]

SES project is experiencing significant delays in procurement of RE and EE systems that need to be piloted. Therefore, the project shall develop a contingency plan to refocus on wider implementation of EE installations and consolidating knowledge gains among engineers and decision makers at the provincial and county level in order to prepare for (eventual) scale-up of the initiatives by the Government.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
1.1 Conduct in-country study tours (bringing together national partners) including NTDCs to discuss and share lessons and areas for intervention a) National consultants select most qualified experts at the NTDCs, Counties and Ris with the support of SES project team. b) Such visits could be expanded beyond the project and to the places such as Jangchon Co-operative farm, Pyongyang City and Natural Energy Institute under the State Academy of Science. c) Produce a report following in-country study tours and commitment received from its participants in scaling up of the EE interventions.
[Added: 2018/10/03] [Last Updated: 2019/09/25]
PA/SES Project team 2019/09 Completed Implemented History
1.1 Establish a regular communication with communities in the implementation of EE technologies with the ownership of MOEPI by sustaining momentum in activity implementation.
[Added: 2018/10/03] [Last Updated: 2019/09/25]
PA/SES Project team 2019/09 Completed Regular communication established with relevant parties History
2. Recommendation:

Recommendations to Improve the Sustainability and Impact of Results

R2. Data collection on users of public buildings

In order to better understand the direct humanitarian impact of the project, it is recommended that the project team work closely with county-level stakeholders (i.e.: managers of public buildings and national consultants) to improve their data collection on how many people (disaggregated by sex, age, disability) access services, and the impacts that the RE and EE improvements have had on particular humanitarian outcomes, particularly health. While it is understood that this may be a difficult undertaking given that access to data is inconsistent, it will nonetheless support qualitative evidence that the project is substantially easing the humanitarian burden that counties face in terms of providing safe and reliable public services such as health care and education.

Management Response: [Added: 2018/10/03] [Last Updated: 2020/12/05]

Project team works closely with county-level stakeholders (i.e.: managers of public buildings and national consultants) to improve their data collection on how many people (disaggregated by sex, age, disability) access services, and the impacts that the RE and EE improvements have had on particular humanitarian outcomes, particularly health.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
2.1 Firm up of the methodology of data collection from local stakeholders a) Rely on a good expertise of national consultants hired under SES b) Train them on the data collection on how many people (disaggregated by sex, age, disability) access public services.
[Added: 2018/10/03] [Last Updated: 2019/09/26]
PA/SES Project team 2019/09 Completed The project is continuing to reply on the expertise of national consultants in terms of data collection for the interventions completed as on Q1/2018. ROAR 2018 already reported sex disaggregated data, also by age and people with disabilities benefitting from those interventions. Updated information will be reported again in APR 2019 as well as in ROAR 2019. History
3. Recommendation:

Recommendations to Improve the Sustainability and Impact of Results

R3. Improved qualitative data collection at the output level

One of the main challenges encountered during this MTR was assessing the qualitative changes effected by the project when monitoring data was limited to quantitative data at the activity level. Moreover, because the project engages in substantial capacity building of the enabling environment and individual technical capacity, it is critical that the project team monitor results of capacity building at the output level, beyond demonstrating the successful implementation of capacity building activities. Below are suggestions for output indicators, and targets, which aim to allow for the collection of data which can be used to analyse the meaningful change in capacity and quality effected by the project to date. These suggestions aim to identify possible entry points for the project, conscious of data access limitations. 

Output 1: Extent to which assessment and audit methodologies are incorporated into the workplans of the NTDCs (Target: Assessment and audit methodologies are formally adopted by NTDCs) 

Output 2: Extent to which County Energy Management Plans receive budgetary support for implementation from the country government (Target: County Energy Management Plans receive at least 75% of necessary funding)

Output 3: Extent to which NTDCs are operational (Target: plans and budgets for 3 NTDCs approved); % of training recipients also serving as peer-to-peer trainers or providing training in other counties (Target: at least 10% of training recipients engaging in knowledge transfer related to RE and EE technologies in some form).

Output 4: % reduction in coal and firewood use for heating by targeted public buildings (Target: at least at 60% reduction in coal and firewood use); % change in patients receiving emergency medical care in targeted hospitals and clinics (Target: at least a 40% increase in emergency/urgent care treatment); % change in absences among 5-7-year-old children in target kindergartens between November-March (Target: at least a 50% reduction in absences). 

Management Response: [Added: 2018/10/03] [Last Updated: 2020/12/05]

It is critical that the project team monitor results of capacity building at the output level, beyond demonstrating the successful implementation of capacity building activities. Below proposed output indicators, and targets, which aim to allow for the collection of data which can be used to analyze the meaningful change in capacity and quality effected by the project to date. These suggestions aim to identify possible entry points for the project, conscious of data access limitations.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
3.1 In consultation with MOEPI following indicators will be monitored by SES project. These will be reported in the QPRs: a. Extent to which County Energy Management Plans receive budgetary support for implementation from the county governments (Target: County Energy Management Plans receive at least 75% of necessary funding) b. Extent to which NTDCs are operational (Target: plans and budgets for 3 NTDCs approved) c. % change in improved attendance of 5-7-year-old children in target kindergartens between November-March (Target: at least a 50% increase)
[Added: 2018/10/03] [Last Updated: 2019/09/26]
PA/SES Project team 2019/09 Completed Suggested indicators are monitored and reported as below: a. UNDP included short term and immediate measures listed in the County Energy Management Plans (CEMPs) as part of its procurement plan in 2018. This procurement plan has been put on hold since early 2018 without any progress till date. As the project couldn’t showcase the implementation of CEMPs, counties refrained their budgetary support for CEMPs. b. Six NTDCs are operational across six counties. They are actively involved in the capacity building activities, mainly training. c. As the country office is functioning under cash conservation mode, no new activities can be commissioned under the project to initiate surveys for the assessment of children attendance in kindergartens. History
4. Recommendation:

Recommendations to Improve the Sustainability and Impact of Results

R4. Standardized monitoring tools

Based on documents reviewed and discussions with project and programme staff, it is evident that although there are comprehensive guidelines for project and programme monitoring in the Country Office, the lack of appropriate tools for data collection and analysis severely impacts what type of data is being collected and by whom. It is recommended that instead of having joint reports following field visits, whether or implementation and monitoring purposes, team members should submit individual BTORs, with project and programme aspects kept separate. A standardized quarterly monitoring report should be used to consolidate data from the BTORs on a quarterly basis only, providing ease in data analysis. Other country offices in the Asia-Pacific region have implemented a similar tool, an example of which is attached as Annex 10. The report should be completed by the project team (lead by the Project Manager), with quality assurance of the data and analysis undertaken by M&E Specialist. This process would improve the storage and analysis of information, both at activity level, and at output level, where analysis to date is weak. This also provides a clear delineation between the role of the project and programme in monitoring and reporting at the project level. 

Management Response: [Added: 2018/10/03] [Last Updated: 2020/12/05]

It is recommended that instead of having joint reports following field visits, whether it is for implementation and/or monitoring purposes, team members should submit individual BTORs separately for project and programme

Key Actions:

Key Action Responsible DueDate Status Comments Documents
4.1 Projects and programme team will submit separate BTORs upon field missions.
[Added: 2018/10/03] [Last Updated: 2019/09/25]
SES Project team 2019/03 Completed Implemented History
4.2 Quarterly Progress Reports (QPRs) should be completed by the project team (lead by the Project Manager), with quality assurance of the data and analysis undertaken.
[Added: 2018/10/03] [Last Updated: 2019/09/25]
SES Project team/PA/M&E Specialist 2019/03 Completed Implemented History
4.3 Report qualitative changes of the projects in ROAR through captured qualitative results from projects.
[Added: 2018/10/03] [Last Updated: 2019/09/15]
SES Project team/PA/M&E Specialist 2019/09 Completed Since there is a deadlock in procurement, actual actions on ground were limited. Despite, the project captured qualitative changes on ground through its activities. These were reported in 2018 ROAR. History
5. Recommendation:

Recommendations to Improve the Sustainability and Impact of Results

R5. Communication of project results

 Political issues surrounding the relevance of the project in terms of its humanitarian role have created challenges in terms of how to communicate the results of the project. If results are communicated at the activity level through purely quantitative data, it is difficult to understand the longer-term, life-saving impact that the project has and will have. With the inclusion of more qualitative indicators at the output level, it is hoped that more meaningful analysis of the humanitarian importance of the project will be captured, and it is recommended that the UNDP Country Office put significantly more effort into communicating these results within the wider UN system in order to reinforce why UNDP’s presence in DPRK is essential, as well as providing evidence for the need to ease some procurement restrictions on UNDP for more effective project implementation and the easing of the humanitarian burden on other agencies. 

Management Response: [Added: 2018/10/03] [Last Updated: 2020/12/05]

With the inclusion of more qualitative indicators at the output level, it is hoped that more meaningful analysis of the humanitarian importance of the project will be captured, bearing in mind the sensitivities in sharing project results publicly due to the complex geopolitical context under which UNDP operates in DPRK. Following are some of the key actions that will be taken to improve the reporting of qualitative changes that the project is leading on the ground.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
5.1 SES project to share communication material (videos/brochures…) with relevant parties including BRH
[Added: 2018/10/03] [Last Updated: 2019/09/25]
SES Project team/PA/ UNDP DPRK CO Com. focal point 2019/09 Completed Project team prepared communication materials and will make public when condition allows History
6. Recommendation:

Recommendations to Improve the Sustainability and Impact of Results

R6. Exit strategy

The project has already instituted activities that will support the sustainability and possible scale-up of results once the project is closed. It is recommended that the project team prepare a strategy for the formal handover of tools and information that would be useful for future roll-out/scale-up to the six NTDCs which it is supporting.  

Management Response: [Added: 2018/10/03] [Last Updated: 2020/12/05]

As the SES project will end by December 2019 according to the ProDoc, it’s the right time to start deploying its exit strategy to meaningfully consolidate the results achieved till date and ensure the sustainability of the project activities and results

Key Actions:

Key Action Responsible DueDate Status Comments Documents
6.1 SES project shall organize a National Partners Meet to assess what was done better and what else need to be completed in fulfilling the needs of the communities. a) At this meet, SES project must encourage the local communities to make in-kind contributions in the absence of procurement activities. b) Consolidating SES project results till date
[Added: 2018/10/03] [Last Updated: 2019/09/15]
SES Project team/PA 2019/03 Completed Completed: National Partners Meet was conducted during 20-22 March. Follow up activities are being implemented in best possible way, despite of deadlock in procurement. History
7. Recommendation:

Going Forward: Programming Scenarios

Office closure. In the case where decisions are taken by UNDP HQ to close the DPRK Country Office due to the reasons that are geopolitical or lack of financial resources available within the country for continued operation of office as there is no existing banking channel or any internal legal reasons, it is advised that the project team must have a contingency plan ready similar to that of above recommendation on exit strategy, whereby knowledge products and tools for RE and EE planning and management to improve energy access (for the most vulnerable) can be easily transferred to the NTDCs (and a relevant UN agency such as WHO and UNICEF), in order to ensure that a) the knowledge products are not lost, and b) UN agencies are able to use the materials in their own work to support the possible roll-out of the SES model in the humanitarian sectors in which they work.

Complete projects. Similar approach as with office closure, but with a more formal handover of materials to a nominated UN agency(ies), as well as identifying a focal agency to continue support to participating counties and NTDCs for improved humanitarian outcomes.

Small scale up. Should UNDP decide to develop a new CPD for the country office, a second phase of the SES project would be appropriate, replicating the original model in new target communities (potentially 5-10 communities based on the availability of funds), and scaling-up the intervention to improve energy access in public buildings and support RE activities in relation to land/slope management in the original 15 communities. If the opportunity is presented, the project should aim to target the NTDCs with regularized knowledge transfer and leadership capacities. 

Large scale up. Although highly unlikely, in an ideal situation, large scale-up of the SES project based on a new CPD would require formalized partnerships with other UN agencies and concerned ministries and departments at the national level engaged in health, education and food security outcomes, targeting communities where those agencies currently work in order to supplement their work with improved energy access and safer environments (i.e.: temperature controlled buildings, improved air quality), as well as improve coordination with the CBDRM project so that energy access targets installations critical in DRR and disaster recovery. Large scale-up would necessitate a greater focus on the monitoring the healthrelated impacts on the users of public buildings (in particularly clinics, hospitals and schools) than on the number of type of RE and EE installations made. In this scenario, it is assumed that sanctions were lifted. Further, it assumes that UNDP makes significant changes to its existing country specific ICF that has limitations on the procurement.

Management Response: [Added: 2020/12/04] [Last Updated: 2020/12/05]

Key Actions:

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org