Mid-Term Evaluation: Sustainable Energy Solutions for Rural Livelihoods in DPRK (SES)

Report Cover Image
Evaluation Plan:
2011-2019, DPR Korea
Evaluation Type:
Mid Term Project
Planned End Date:
09/2018
Completion Date:
08/2018
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
12,000

Share

Document Type Language Size Status Downloads
Download document TOR_SES_CBDRM_Project_Final21Feb18.pdf tor English 488.26 KB Posted 263
Download document FinalMTRReport_SES_August 2018.pdf report English 1141.09 KB Posted 349
Download document FinalMTRReport_SES_Summary_August2018.docx summary English 1174.87 KB Posted 263
Title Mid-Term Evaluation: Sustainable Energy Solutions for Rural Livelihoods in DPRK (SES)
Atlas Project Number: 00090996
Evaluation Plan: 2011-2019, DPR Korea
Evaluation Type: Mid Term Project
Status: Completed
Completion Date: 08/2018
Planned End Date: 09/2018
Management Response: Yes
Focus Area:
  • 1. Environment & Sustainable Development
  • 2. Others
Corporate Outcome and Output (UNDP Strategic Plan 2014-2017)
  • 1. Output 1.5. Inclusive and sustainable solutions adopted to achieve increased energy efficiency and universal modern energy access (especially off-grid sources of renewable energy)
Evaluation Budget(US $): 12,000
Source of Funding: TRAC
Evaluation Expenditure(US $): 10,000
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
Denika Blacklock Independant Evaluation Consultant djbkarim@gmail.com
GEF Evaluation: No
Key Stakeholders:
Countries: DPRK -DEMOCRATIC PEOPLE'S REPUBLIC OF KOREA
Comments:

The evaluation has been planned in the Project Document

Lessons
Findings
1.

Section 3: Analysis of Findings based on the OECD DAC for Development Evaluations

Project Strategy: To what extent is the project strategy relevant to country priorities, country ownership and the best route towards expected results?

Relevance of the project design.

This project is both extremely relevant to the current humanitarian context of DPRK, as well as to the priorities of the government to promote the use of renewable energy technologies, particularly in rural and remote areas which have had their energy access negatively impacted by the numerous extreme weather events, including severe flooding, over the past two decades. While the project initially envisioned targeting households as well as public service infrastructure for renewable energy and energy efficiency interventions, given the wide-ranging need across the country, the project instead opted to focus singularly on interventions in public service delivery and other community-wide options such as micro-hydro and improved biomass management. These interventions would have a wider reach, and avoid choosing one household over another, both being equally in need, because of budget constraints. 

It was noted on more than one occasion during the MTR country mission that this project has identified the communities most in need, assessing need over reach, and reinforcing the objective of the SDGs which is to ‘leave no one behind.’ Government counterparts noted that in many cases, project staff would walk 1-2 km to the site where renewable energy (RE) or energy efficiency (EE) interventions had taken place for monitoring and verification purposes. Often villages this remote would not be covered by development projects due to access issues and concerns about value for money – how many people are impacted for each dollar spent. However, although some sites do not site within the traditional parametres of humanitarian intervention, it is evident that the 15 Ris (local level, below county) selected for intervention, and the public buildings prioritized for support, were appropriately selected by the project and demonstrate a suitable model for roll-out across the country, and internationally as a good practice. During the project design phase, it was assumed that energy interventions would benefit men more than women as improved livelihoods was a major focus of the project. However, the project has evolved over time to accommodate the shifting geopolitical landscape and a need to implement activities that are more traditionally humanitarian in nature. Therefore, the project has resulted in more impacts for woman and children than initially presumed, which will be important both in terms of longer term humanitarian impacts, but also in terms of SDG monitoring. 


Tag: Rural development Vulnerable Energy Relevance Programme/Project Design Results-Based Management Country Government Humanitarian development nexus Technology Agenda 2030 Leaving no one behind

2.

Progress towards results: To what extent have the expected outputs of the project been achieved so far?

Assessment of progress towards targets. This MTR provides an assessment of progress towards current output targets based on monitoring data provided and supplementary qualitative information captured during the MTR country mission. (see table in the report)

Factors influencing project implementation. Aside from the overall geopolitical and operational issues which negatively impact the implementation of the project, the primary factor related to the achievement of the results of the project to date, despite the many challenges faced by the project team, has largely been due to the technical expertise and experience of the project manager. Without detracting from the hard work and capacity of the other project team members, it was noted by project stakeholders at national and local level that the project manager’s ability to adjust activity implementation to the everchanging operational environment, as well as his technical background, benefits the project similar to having a full-time chief technical advisor, and has meant that the quality of the activities implemented to date are far and away the highest experienced by the country in relation to the energy sector (as quoted from the Ministry of Electric Power Industry). Moreover, the project management capacities have allowed for the maximum of activity implementation despite the highly restrictive environment. 


Tag: Effectiveness Human and Financial resources Operational Efficiency Ownership Project and Programme management UNDP Management Capacity Building Technology Technical Support

3.

Project Implementation and Adaptive Management

Project efficiency.

Due to the geopolitical situation and current ICF of the UNDP Country Office, the project is implemented through the Direct Implementation (DIM) modality. With an international project manager, national programme and project staff, and supported by an international Operations Manager and international M&E Specialist (at the programme level), the project has sufficient technical and management capacity to be effectively implemented. It should be noted that cost-savings in project management come from cost-sharing of the costs of the project team with the CBDRM project. Not only does that improve synergies between the projects where project sites overlap, but also improves operational efficiencies in terms of activity implementation. However, the projects1 carry the programme costs (senior management, operations staff salaries) as well as office premise costs (operations and maintenance of the office building) has significantly increased the proportion of the project budget utilized for project management beyond what would be considered acceptable even in the most management-intensive environments. For example, with delays in procurement pushing most activities from 2016 to 2017, the proportion of the project management costs in the budget in 2016 was approximately 55%, while in 2017 it was 16-18% (depending on whether the APR or AWP is referenced, respectively). These extra costs are being carried by the project to the detriment of communities that could be benefit from RE and EE interventions which that money should be used for. Thus, while the project is rated very well for its saving in project management (sharing project management team costs between projects), the additional project management costs incurred due to procurement delays (inefficient use of project management team when activities cannot be implemented) mean that project cannot be considered efficient in its budget use. While the project does not have partnerships for implementation in the traditional sense of the term, the decision by the project manager to bundle activities into fewer contracts with consulting firm RENAC (Germany) and Novi Sad University (Serbia) significantly improved the operational efficiency of the project (in terms of contracting, payments, travel, visas, activity planning), as well as ensured a continuity in soft interventions as one integrated work plan for linked activities could be carried out by the contractors without interruption (which would have likely been the case if a large number of individual consultants had been contracted to carry out individual activities), and later by national consultants. It has become standard practice for this evaluator to assess the priorities of the project from quality and value for money perspectives. Often times, donor pressure for value for money in implementation is prioritized over quality in implementation and targeting the most in need (where fewer people benefit but the impact is greater). The SES project presents an interesting case whereby project funds come solely from UNDP core budget so there is no pressure to meet the value for money expectations of external donors, while the employment of a project manager with an engineering background has allowed for the highest level of quality in the implementation of RE and EE interventions due to his hands-on approach and the requirement in the ICF for verifications of the installation of materials by international (as well as national) staff members. Despite delays in implementation due to sanctions and procurement issues, this project is a model for good practice in what it means to leave no one behind.

Finally, in terms of adaptive management by the project, it should be noted that while the project team has contingency plans for adjusting the annual work plans based on delays in procurement and RE installation, the continued delays which unduly impact the ability of the project team to deliver its activities indicate that the project has a high tolerance for uncertainty before changes in the work plan are implemented. It is recommended that the project develop a contingency plan whereby no RE activities can be implemented, refocusing on wider implementation of EE installations and consolidating knowledge gains among engineers and decision makers at the provincial and county level in order to prepare for (eventual) scale-up of the initiative by the government and local counterparts. Such a contingency plan would allow the project to capitalize on both the interest of communities in EE technologies, as well as the ownership of the national and local counterparts by sustaining momentum in activity implementation. 

Financial Controls and In-kind Contributions. As noted routinely in this report, the project team has been extremely capable at managing its resources and adjusting planning in order to ensure that the project makes the most efficient use of its time and money in light of the many procurement challenges it faces. There is excellent planning, and in a ‘normal’ operating context, this would result in excellent financial planning and management. However, the project team is faced with many constraints, not least the issue of the banking channel, which impact how quickly it can access funds. In the view of this MTR, the project team is doing an excellent job within the constraints that it is implementing activities and should not be reviewed against issues beyond its direct control. The project team also makes good use of the in-kind contributions of communities related to structural and works. While it was not possible to assess the inkind contribution of each community, it was evident from the communities which were visited that the structural interventions that are in progress or will be implemented next year (provided procurement requests are approved) would be either impossible or significantly more expensive without community participation to make reasonable initial preparations for structural interventions (i.e.: contributions that would equate to less than 15% of total activity cost). 


Tag: Efficiency Implementation Modality Monitoring and Evaluation Operational Efficiency Partnership Policies & Procedures Programme Synergy Results-Based Management Risk Management

4.

Sustainability.

Given the nature of RE and EE, the benefits of the project to date will be very sustainable in the short term, mostly sustainable in the medium term, and likely sustainable in the long term of county-level technical knowledge and energy planning and management capacities are improved and consolidated over the remainder of the project. EE technologies require minimal future interventions beyond basic maintenance and will provide safer and more temperature-controlled premises for health care and education. RE technologies need technical support when maintenance is required, and counties need to plan for budgeting for replacement parts (for example, Solar PV has a lifetime of 25 years, but the back-up battery has a life of only seven years). Knowledge transfer will be sustainable, particularly with the set-up of the NTDCs, which will provide not only technical support, but could also act as a training centre itself for the future scale-up of the initiatives by the government (when feasible). 

In terms of the sustainability of the impacts, given that the project has only target public buildings, most of which serve humanitarian purposes, as long as maintenance of the buildings and technologies is sustained by the counties, the humanitarian burden (particularly in terms of health burdens) will be reduced, with improved health outcomes due to improved access to health facilities and timely treatment. An important aspect of this project is that the set-up of the NTDCs to support RE and EE installation and maintenance at the county-level also serves as an appropriate institution for the handover of responsibilities at the county level, which is a reasonable exit strategy for UNDP in light of the operational uncertainty that the Country Office is currently facing. With NTDCs as part of the energy management institutional framework at the county level, the framework for eventual scale-up of the initiative is already in place. Should international funding for such initiatives become more available in the future, the NTDCs will be well-placed to lead such initiatives, for example with WHO which is interested in supporting more EE interventions in hospitals not targeted by SES. Likewise, should the government desire, it can use the NTDCs to implement its own funds to roll-out RE and EE technologies to a wider selection of public buildings. 

However, there are always risks to sustainability, and while SES has put in place a number of processes (training, energy management plans, NTDCs) which would allow the counties to carry on with implementation should the SES project have to close due to operational constraints, there are a few issues which the project should plan for. While financial and socio-economic risks to the project’s results are minimal at this stage, the ongoing issue of delayed procurement due to sanctions issues will impact the finalization of RE and EE installations, as well as impact environmental sustainability of RE in that it undermines the ability of the project to support the application of quick-rotation crops for agro-forestry, supporting both renewable energy options and disaster reduction in terms of reducing soil erosion and the risk of landslides. The other issue is sustainability of commitment by the counties if their expectations related to larger RE installations are not met, including for micro-hydro dams and channeling. The project will need to improve its management of beneficiary expectations, and likely identify alternative activities for communities where procurement delays will result in fewer RE installations. 


Tag: Rural development Disaster Risk Reduction Natural Disaster Energy Natural Resouce management Sustainability Resource mobilization Health Sector Knowledge management Risk Management Country Government Capacity Building Technology

5.

Assessment of Cross-cutting Issues The focus on capacity building assistance. While the project is precluded from providing technical support directly to the national level, it has been able to provide a rather holistic capacity development approach at the county and local level, with support to the enabling environment (energy management plans), institutional arrangements (NTDCs), and individuals (tools and training on RE and EE technologies). Significant technical capacity building of individuals (both decision makers and engineers) has taken place and while training does not equate to improved skills, the implementation of the study tour to Novi Sad was instrumental in terms of providing examples of how RE and EE works in practice, which, based on feedback during the MTR country mission, improved both commitment to and leadership of processes to improve energy access in target communities. However, because the project lacks qualitative indicators, it is difficult to assess how much of the knowledge that has been transferred has been retained. Recommendations for data collection related to capacity building are provided below. 


Tag: Rural development Energy Gender Equality Gender Mainstreaming Local Governance Health Sector Results-Based Management Humanitarian development nexus Capacity Building Technical Support Data and Statistics

Recommendations
1

Recommendations to Improve the Sustainability and Impact of Results

R1. Contingency planning for RE activities

In light of the continue delays in procurement approvals for materials for RE installations, it is recommended that the project develop a contingency plan whereby no RE activities can be implemented, refocusing on wider implementation of EE installations and consolidating knowledge gains among engineers and decision makers at the provincial and county level in order to prepare for (eventual) scale-up of the initiative by the government, including supporting such activities as in-country study tours, and bringing together national partners, including NTDCs to discuss and share lessons and areas for intervention. Such a contingency plan would allow the project to capitalize on both the interest of communities in EE technologies, as well as the ownership of the government by sustaining momentum in activity implementation.

2

Recommendations to Improve the Sustainability and Impact of Results

R2. Data collection on users of public buildings

In order to better understand the direct humanitarian impact of the project, it is recommended that the project team work closely with county-level stakeholders (i.e.: managers of public buildings and national consultants) to improve their data collection on how many people (disaggregated by sex, age, disability) access services, and the impacts that the RE and EE improvements have had on particular humanitarian outcomes, particularly health. While it is understood that this may be a difficult undertaking given that access to data is inconsistent, it will nonetheless support qualitative evidence that the project is substantially easing the humanitarian burden that counties face in terms of providing safe and reliable public services such as health care and education.

3

Recommendations to Improve the Sustainability and Impact of Results

R3. Improved qualitative data collection at the output level

One of the main challenges encountered during this MTR was assessing the qualitative changes effected by the project when monitoring data was limited to quantitative data at the activity level. Moreover, because the project engages in substantial capacity building of the enabling environment and individual technical capacity, it is critical that the project team monitor results of capacity building at the output level, beyond demonstrating the successful implementation of capacity building activities. Below are suggestions for output indicators, and targets, which aim to allow for the collection of data which can be used to analyse the meaningful change in capacity and quality effected by the project to date. These suggestions aim to identify possible entry points for the project, conscious of data access limitations. 

Output 1: Extent to which assessment and audit methodologies are incorporated into the workplans of the NTDCs (Target: Assessment and audit methodologies are formally adopted by NTDCs) 

Output 2: Extent to which County Energy Management Plans receive budgetary support for implementation from the country government (Target: County Energy Management Plans receive at least 75% of necessary funding)

Output 3: Extent to which NTDCs are operational (Target: plans and budgets for 3 NTDCs approved); % of training recipients also serving as peer-to-peer trainers or providing training in other counties (Target: at least 10% of training recipients engaging in knowledge transfer related to RE and EE technologies in some form).

Output 4: % reduction in coal and firewood use for heating by targeted public buildings (Target: at least at 60% reduction in coal and firewood use); % change in patients receiving emergency medical care in targeted hospitals and clinics (Target: at least a 40% increase in emergency/urgent care treatment); % change in absences among 5-7-year-old children in target kindergartens between November-March (Target: at least a 50% reduction in absences). 

4

Recommendations to Improve the Sustainability and Impact of Results

R4. Standardized monitoring tools

Based on documents reviewed and discussions with project and programme staff, it is evident that although there are comprehensive guidelines for project and programme monitoring in the Country Office, the lack of appropriate tools for data collection and analysis severely impacts what type of data is being collected and by whom. It is recommended that instead of having joint reports following field visits, whether or implementation and monitoring purposes, team members should submit individual BTORs, with project and programme aspects kept separate. A standardized quarterly monitoring report should be used to consolidate data from the BTORs on a quarterly basis only, providing ease in data analysis. Other country offices in the Asia-Pacific region have implemented a similar tool, an example of which is attached as Annex 10. The report should be completed by the project team (lead by the Project Manager), with quality assurance of the data and analysis undertaken by M&E Specialist. This process would improve the storage and analysis of information, both at activity level, and at output level, where analysis to date is weak. This also provides a clear delineation between the role of the project and programme in monitoring and reporting at the project level. 

5

Recommendations to Improve the Sustainability and Impact of Results

R5. Communication of project results

 Political issues surrounding the relevance of the project in terms of its humanitarian role have created challenges in terms of how to communicate the results of the project. If results are communicated at the activity level through purely quantitative data, it is difficult to understand the longer-term, life-saving impact that the project has and will have. With the inclusion of more qualitative indicators at the output level, it is hoped that more meaningful analysis of the humanitarian importance of the project will be captured, and it is recommended that the UNDP Country Office put significantly more effort into communicating these results within the wider UN system in order to reinforce why UNDP’s presence in DPRK is essential, as well as providing evidence for the need to ease some procurement restrictions on UNDP for more effective project implementation and the easing of the humanitarian burden on other agencies. 

6

Recommendations to Improve the Sustainability and Impact of Results

R6. Exit strategy

The project has already instituted activities that will support the sustainability and possible scale-up of results once the project is closed. It is recommended that the project team prepare a strategy for the formal handover of tools and information that would be useful for future roll-out/scale-up to the six NTDCs which it is supporting.  

7

Going Forward: Programming Scenarios

Office closure. In the case where decisions are taken by UNDP HQ to close the DPRK Country Office due to the reasons that are geopolitical or lack of financial resources available within the country for continued operation of office as there is no existing banking channel or any internal legal reasons, it is advised that the project team must have a contingency plan ready similar to that of above recommendation on exit strategy, whereby knowledge products and tools for RE and EE planning and management to improve energy access (for the most vulnerable) can be easily transferred to the NTDCs (and a relevant UN agency such as WHO and UNICEF), in order to ensure that a) the knowledge products are not lost, and b) UN agencies are able to use the materials in their own work to support the possible roll-out of the SES model in the humanitarian sectors in which they work.

Complete projects. Similar approach as with office closure, but with a more formal handover of materials to a nominated UN agency(ies), as well as identifying a focal agency to continue support to participating counties and NTDCs for improved humanitarian outcomes.

Small scale up. Should UNDP decide to develop a new CPD for the country office, a second phase of the SES project would be appropriate, replicating the original model in new target communities (potentially 5-10 communities based on the availability of funds), and scaling-up the intervention to improve energy access in public buildings and support RE activities in relation to land/slope management in the original 15 communities. If the opportunity is presented, the project should aim to target the NTDCs with regularized knowledge transfer and leadership capacities. 

Large scale up. Although highly unlikely, in an ideal situation, large scale-up of the SES project based on a new CPD would require formalized partnerships with other UN agencies and concerned ministries and departments at the national level engaged in health, education and food security outcomes, targeting communities where those agencies currently work in order to supplement their work with improved energy access and safer environments (i.e.: temperature controlled buildings, improved air quality), as well as improve coordination with the CBDRM project so that energy access targets installations critical in DRR and disaster recovery. Large scale-up would necessitate a greater focus on the monitoring the healthrelated impacts on the users of public buildings (in particularly clinics, hospitals and schools) than on the number of type of RE and EE installations made. In this scenario, it is assumed that sanctions were lifted. Further, it assumes that UNDP makes significant changes to its existing country specific ICF that has limitations on the procurement.

1. Recommendation:

Recommendations to Improve the Sustainability and Impact of Results

R1. Contingency planning for RE activities

In light of the continue delays in procurement approvals for materials for RE installations, it is recommended that the project develop a contingency plan whereby no RE activities can be implemented, refocusing on wider implementation of EE installations and consolidating knowledge gains among engineers and decision makers at the provincial and county level in order to prepare for (eventual) scale-up of the initiative by the government, including supporting such activities as in-country study tours, and bringing together national partners, including NTDCs to discuss and share lessons and areas for intervention. Such a contingency plan would allow the project to capitalize on both the interest of communities in EE technologies, as well as the ownership of the government by sustaining momentum in activity implementation.

Management Response: [Added: 2018/09/20] [Last Updated: 2020/12/05]

SES project is experiencing significant delays in procurement of RE and EE systems that need to be piloted. Therefore, the project shall develop a contingency plan to refocus on wider implementation of EE installations and consolidating knowledge gains among engineers and decision makers at the provincial and county level in order to prepare for (eventual) scale-up of the initiatives by the Government.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
1.1 Conduct in-country study tours (bringing together national partners) including NTDCs to discuss and share lessons and areas for intervention a) National consultants select most qualified experts at the NTDCs, Counties and Ris with the support of SES project team. b) Such visits could be expanded beyond the project and to the places such as Jangchon Co-operative farm, Pyongyang City and Natural Energy Institute under the State Academy of Science. c) Produce a report following in-country study tours and commitment received from its participants in scaling up of the EE interventions.
[Added: 2018/10/03] [Last Updated: 2019/09/25]
PA/SES Project team 2019/09 Completed Implemented History
1.1 Establish a regular communication with communities in the implementation of EE technologies with the ownership of MOEPI by sustaining momentum in activity implementation.
[Added: 2018/10/03] [Last Updated: 2019/09/25]
PA/SES Project team 2019/09 Completed Regular communication established with relevant parties History
2. Recommendation:

Recommendations to Improve the Sustainability and Impact of Results

R2. Data collection on users of public buildings

In order to better understand the direct humanitarian impact of the project, it is recommended that the project team work closely with county-level stakeholders (i.e.: managers of public buildings and national consultants) to improve their data collection on how many people (disaggregated by sex, age, disability) access services, and the impacts that the RE and EE improvements have had on particular humanitarian outcomes, particularly health. While it is understood that this may be a difficult undertaking given that access to data is inconsistent, it will nonetheless support qualitative evidence that the project is substantially easing the humanitarian burden that counties face in terms of providing safe and reliable public services such as health care and education.

Management Response: [Added: 2018/10/03] [Last Updated: 2020/12/05]

Project team works closely with county-level stakeholders (i.e.: managers of public buildings and national consultants) to improve their data collection on how many people (disaggregated by sex, age, disability) access services, and the impacts that the RE and EE improvements have had on particular humanitarian outcomes, particularly health.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
2.1 Firm up of the methodology of data collection from local stakeholders a) Rely on a good expertise of national consultants hired under SES b) Train them on the data collection on how many people (disaggregated by sex, age, disability) access public services.
[Added: 2018/10/03] [Last Updated: 2019/09/26]
PA/SES Project team 2019/09 Completed The project is continuing to reply on the expertise of national consultants in terms of data collection for the interventions completed as on Q1/2018. ROAR 2018 already reported sex disaggregated data, also by age and people with disabilities benefitting from those interventions. Updated information will be reported again in APR 2019 as well as in ROAR 2019. History
3. Recommendation:

Recommendations to Improve the Sustainability and Impact of Results

R3. Improved qualitative data collection at the output level

One of the main challenges encountered during this MTR was assessing the qualitative changes effected by the project when monitoring data was limited to quantitative data at the activity level. Moreover, because the project engages in substantial capacity building of the enabling environment and individual technical capacity, it is critical that the project team monitor results of capacity building at the output level, beyond demonstrating the successful implementation of capacity building activities. Below are suggestions for output indicators, and targets, which aim to allow for the collection of data which can be used to analyse the meaningful change in capacity and quality effected by the project to date. These suggestions aim to identify possible entry points for the project, conscious of data access limitations. 

Output 1: Extent to which assessment and audit methodologies are incorporated into the workplans of the NTDCs (Target: Assessment and audit methodologies are formally adopted by NTDCs) 

Output 2: Extent to which County Energy Management Plans receive budgetary support for implementation from the country government (Target: County Energy Management Plans receive at least 75% of necessary funding)

Output 3: Extent to which NTDCs are operational (Target: plans and budgets for 3 NTDCs approved); % of training recipients also serving as peer-to-peer trainers or providing training in other counties (Target: at least 10% of training recipients engaging in knowledge transfer related to RE and EE technologies in some form).

Output 4: % reduction in coal and firewood use for heating by targeted public buildings (Target: at least at 60% reduction in coal and firewood use); % change in patients receiving emergency medical care in targeted hospitals and clinics (Target: at least a 40% increase in emergency/urgent care treatment); % change in absences among 5-7-year-old children in target kindergartens between November-March (Target: at least a 50% reduction in absences). 

Management Response: [Added: 2018/10/03] [Last Updated: 2020/12/05]

It is critical that the project team monitor results of capacity building at the output level, beyond demonstrating the successful implementation of capacity building activities. Below proposed output indicators, and targets, which aim to allow for the collection of data which can be used to analyze the meaningful change in capacity and quality effected by the project to date. These suggestions aim to identify possible entry points for the project, conscious of data access limitations.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
3.1 In consultation with MOEPI following indicators will be monitored by SES project. These will be reported in the QPRs: a. Extent to which County Energy Management Plans receive budgetary support for implementation from the county governments (Target: County Energy Management Plans receive at least 75% of necessary funding) b. Extent to which NTDCs are operational (Target: plans and budgets for 3 NTDCs approved) c. % change in improved attendance of 5-7-year-old children in target kindergartens between November-March (Target: at least a 50% increase)
[Added: 2018/10/03] [Last Updated: 2019/09/26]
PA/SES Project team 2019/09 Completed Suggested indicators are monitored and reported as below: a. UNDP included short term and immediate measures listed in the County Energy Management Plans (CEMPs) as part of its procurement plan in 2018. This procurement plan has been put on hold since early 2018 without any progress till date. As the project couldn’t showcase the implementation of CEMPs, counties refrained their budgetary support for CEMPs. b. Six NTDCs are operational across six counties. They are actively involved in the capacity building activities, mainly training. c. As the country office is functioning under cash conservation mode, no new activities can be commissioned under the project to initiate surveys for the assessment of children attendance in kindergartens. History
4. Recommendation:

Recommendations to Improve the Sustainability and Impact of Results

R4. Standardized monitoring tools

Based on documents reviewed and discussions with project and programme staff, it is evident that although there are comprehensive guidelines for project and programme monitoring in the Country Office, the lack of appropriate tools for data collection and analysis severely impacts what type of data is being collected and by whom. It is recommended that instead of having joint reports following field visits, whether or implementation and monitoring purposes, team members should submit individual BTORs, with project and programme aspects kept separate. A standardized quarterly monitoring report should be used to consolidate data from the BTORs on a quarterly basis only, providing ease in data analysis. Other country offices in the Asia-Pacific region have implemented a similar tool, an example of which is attached as Annex 10. The report should be completed by the project team (lead by the Project Manager), with quality assurance of the data and analysis undertaken by M&E Specialist. This process would improve the storage and analysis of information, both at activity level, and at output level, where analysis to date is weak. This also provides a clear delineation between the role of the project and programme in monitoring and reporting at the project level. 

Management Response: [Added: 2018/10/03] [Last Updated: 2020/12/05]

It is recommended that instead of having joint reports following field visits, whether it is for implementation and/or monitoring purposes, team members should submit individual BTORs separately for project and programme

Key Actions:

Key Action Responsible DueDate Status Comments Documents
4.1 Projects and programme team will submit separate BTORs upon field missions.
[Added: 2018/10/03] [Last Updated: 2019/09/25]
SES Project team 2019/03 Completed Implemented History
4.2 Quarterly Progress Reports (QPRs) should be completed by the project team (lead by the Project Manager), with quality assurance of the data and analysis undertaken.
[Added: 2018/10/03] [Last Updated: 2019/09/25]
SES Project team/PA/M&E Specialist 2019/03 Completed Implemented History
4.3 Report qualitative changes of the projects in ROAR through captured qualitative results from projects.
[Added: 2018/10/03] [Last Updated: 2019/09/15]
SES Project team/PA/M&E Specialist 2019/09 Completed Since there is a deadlock in procurement, actual actions on ground were limited. Despite, the project captured qualitative changes on ground through its activities. These were reported in 2018 ROAR. History
5. Recommendation:

Recommendations to Improve the Sustainability and Impact of Results

R5. Communication of project results

 Political issues surrounding the relevance of the project in terms of its humanitarian role have created challenges in terms of how to communicate the results of the project. If results are communicated at the activity level through purely quantitative data, it is difficult to understand the longer-term, life-saving impact that the project has and will have. With the inclusion of more qualitative indicators at the output level, it is hoped that more meaningful analysis of the humanitarian importance of the project will be captured, and it is recommended that the UNDP Country Office put significantly more effort into communicating these results within the wider UN system in order to reinforce why UNDP’s presence in DPRK is essential, as well as providing evidence for the need to ease some procurement restrictions on UNDP for more effective project implementation and the easing of the humanitarian burden on other agencies. 

Management Response: [Added: 2018/10/03] [Last Updated: 2020/12/05]

With the inclusion of more qualitative indicators at the output level, it is hoped that more meaningful analysis of the humanitarian importance of the project will be captured, bearing in mind the sensitivities in sharing project results publicly due to the complex geopolitical context under which UNDP operates in DPRK. Following are some of the key actions that will be taken to improve the reporting of qualitative changes that the project is leading on the ground.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
5.1 SES project to share communication material (videos/brochures…) with relevant parties including BRH
[Added: 2018/10/03] [Last Updated: 2019/09/25]
SES Project team/PA/ UNDP DPRK CO Com. focal point 2019/09 Completed Project team prepared communication materials and will make public when condition allows History
6. Recommendation:

Recommendations to Improve the Sustainability and Impact of Results

R6. Exit strategy

The project has already instituted activities that will support the sustainability and possible scale-up of results once the project is closed. It is recommended that the project team prepare a strategy for the formal handover of tools and information that would be useful for future roll-out/scale-up to the six NTDCs which it is supporting.  

Management Response: [Added: 2018/10/03] [Last Updated: 2020/12/05]

As the SES project will end by December 2019 according to the ProDoc, it’s the right time to start deploying its exit strategy to meaningfully consolidate the results achieved till date and ensure the sustainability of the project activities and results

Key Actions:

Key Action Responsible DueDate Status Comments Documents
6.1 SES project shall organize a National Partners Meet to assess what was done better and what else need to be completed in fulfilling the needs of the communities. a) At this meet, SES project must encourage the local communities to make in-kind contributions in the absence of procurement activities. b) Consolidating SES project results till date
[Added: 2018/10/03] [Last Updated: 2019/09/15]
SES Project team/PA 2019/03 Completed Completed: National Partners Meet was conducted during 20-22 March. Follow up activities are being implemented in best possible way, despite of deadlock in procurement. History
7. Recommendation:

Going Forward: Programming Scenarios

Office closure. In the case where decisions are taken by UNDP HQ to close the DPRK Country Office due to the reasons that are geopolitical or lack of financial resources available within the country for continued operation of office as there is no existing banking channel or any internal legal reasons, it is advised that the project team must have a contingency plan ready similar to that of above recommendation on exit strategy, whereby knowledge products and tools for RE and EE planning and management to improve energy access (for the most vulnerable) can be easily transferred to the NTDCs (and a relevant UN agency such as WHO and UNICEF), in order to ensure that a) the knowledge products are not lost, and b) UN agencies are able to use the materials in their own work to support the possible roll-out of the SES model in the humanitarian sectors in which they work.

Complete projects. Similar approach as with office closure, but with a more formal handover of materials to a nominated UN agency(ies), as well as identifying a focal agency to continue support to participating counties and NTDCs for improved humanitarian outcomes.

Small scale up. Should UNDP decide to develop a new CPD for the country office, a second phase of the SES project would be appropriate, replicating the original model in new target communities (potentially 5-10 communities based on the availability of funds), and scaling-up the intervention to improve energy access in public buildings and support RE activities in relation to land/slope management in the original 15 communities. If the opportunity is presented, the project should aim to target the NTDCs with regularized knowledge transfer and leadership capacities. 

Large scale up. Although highly unlikely, in an ideal situation, large scale-up of the SES project based on a new CPD would require formalized partnerships with other UN agencies and concerned ministries and departments at the national level engaged in health, education and food security outcomes, targeting communities where those agencies currently work in order to supplement their work with improved energy access and safer environments (i.e.: temperature controlled buildings, improved air quality), as well as improve coordination with the CBDRM project so that energy access targets installations critical in DRR and disaster recovery. Large scale-up would necessitate a greater focus on the monitoring the healthrelated impacts on the users of public buildings (in particularly clinics, hospitals and schools) than on the number of type of RE and EE installations made. In this scenario, it is assumed that sanctions were lifted. Further, it assumes that UNDP makes significant changes to its existing country specific ICF that has limitations on the procurement.

Management Response: [Added: 2020/12/04] [Last Updated: 2020/12/05]

Key Actions:

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org