End of Project Evaluation: Sustainable Energy Solutions for Rural Livelihoods in DPRK (SES)

Evaluation Plan:: 2011-2019, DPR Korea
Evaluation Type:: Final Project
Planned End Date:: 03/2020
Completion Date:: 01/2020
Status:: Completed
Management Response:: Yes
Evaluation Budget(US $):: 60,000

Document	Type	Language	Size	Status	Downloads
IC_ToR_SES_19Sept2019_Final_Ad.pdf	tor	English	318.87 KB	Posted	220
UNDP DPRK_SES_Terminal Eval Report_Jan2020 FINAL clean version.pdf	report	English	1468.67 KB	Posted	228
UNDP DPRK_SES_Terminal Eval Report_22Jan2020_Final_Summary.pdf	summary	English	220.86 KB	Posted	171

Title

End of Project Evaluation: Sustainable Energy Solutions for Rural Livelihoods in DPRK (SES)

Atlas Project Number:

00090996

Evaluation Plan:

2011-2019, DPR Korea

Evaluation Type:

Final Project

Status:

Completed

Completion Date:

01/2020

Planned End Date:

03/2020

Management Response:

Yes

Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)

1. Output 1.5.1 Solutions adopted to achieve universal access to clean, affordable and sustainable energy
2. Organisational Output 2.1 UNDP recognized as a development partner of choice

Evaluation Budget(US $):

60,000

Source of Funding:

TRAC

Evaluation Expenditure(US $):

30,000

Joint Programme:

Joint Evaluation:

Evaluation Team members:

Name	Title	Email	Nationality
Dr Jeff Fang	Evaluator	jeff@icmac.asia

GEF Evaluation:

Key Stakeholders:

Countries:

DPRK -DEMOCRATIC PEOPLE'S REPUBLIC OF KOREA

Comments:

The evaluation has been planned in the Project Document

Lessons

5.1 Conclusions and Lessons Learned

Conclusion #1: Significant external factors/challenges severely affected the project

Significant external factors/challenges beyond the control of the UNDP DPRK CO were encountered throughout the entire SES project implementation, and severely affected the timely delivery of project outputs relating to procurement-related activities.

Table 9 below shows the timeline of how the significant external factors/challenges overlapped each other, hence the SES Project Team would not be free of any constraints at any point of time between 2015 to 2019 to effectively and efficiently implement the project outputs relating to procurement activities to fully achieve the desired project outcome.

Table 9: Timeline of External Factors/Challenges Faced by UNDP DPRK CO

In particular, the evaluation highlights below the 2 external factors/challenges as the main constraints.

1. 6 Rounds of UN Sanctions on DPRK (2016-2017); and

2. Extended Period of Banking Channel Disruptions/Closure

The UN Security Council imposed two UN Sanctions (UN Resolutions #2270 and #2321) in 2016 and another four UN Sanctions (UN Resolutions #2356, #2371, #2375 and #2397) in 2017 were imposed on DPRK which included (among many measures) import, financial and economic restrictions. As a result, the UNDP DPRK CO and SES Project Team were severely constrained and the SES project’s delivery negatively impacted as follow:

The complicated, lengthy and increasingly difficult process to obtain clearance or exemptions for international procurement from UN Sanctions Committee 1718 which oversees the implementation of the UN Sanctions on DPRK.
The recurred disruption/closure of the banking channel prevented funds transfer into DPRK for the UNDP DPRK CO to fully implement local activities and local procurement. This also led to the UNDP DPRK CO having to activate cash conservation mode and enforce stringent internal measures to sustain the office operations, which resulted in (1) restrictions for in-country/local procurement, and (2) increased complexity and time to implement the SES Project’s procurement activities.

The SES PRODOC had appropriate risk assessments which identified a total of 15 risks (1 governance risk, 3 operational risks, 5 strategic risks, 3 financial/fiduciary risks, and 3 sustainability risks) with impact and probability ratings, and prepared corresponding counter-measures/management responses which were appropriate at that point of time and during the project implementation (2015 to 2019).

The risk assessments could be further extended by identifying the key risks and appropriate counter-measures/management response for each of project outputs within the Results and Resources Framework.

The evaluation noted that the risk analysis did not plan for scenarios of extreme UN sanction measures and the extended banking channel disruption/closure. Furthermore, the implementation of the SES PRODOC’s counter-measures/management responses did not appropriately resolve the significant change of events caused by the UN Sanction measures and the extended banking channel disruption/closure during the project implementation.

Lesson Learned:

Delayed efforts to complete procurement-related interventions, especially those listed as part of the feasibility studies severely disrupted county and village community (Ri) development plans/activities, resulting in potential economic hardship/losses and supply chain sustainability/productivity not fully realized.
Long-term scenario planning together with annual reviews for change of direction should form part of risk assessment and mitigations in special country context projects.

Conclusion #2: The UNDP SES Project Team has done their best but there is room for improvement in project implementation

Despite the challenging circumstances, The SES Project Team has done their best and laid strong foundations to enable sustainable energy solutions at the village community (Ri) level. The SES Project Team was able to implement the project despite encountering the significant external factors and challenges that were beyond the control of the UNDP DPRK CO throughout the entire SES Project by:

displaying good project management abilities and effectively utilising appropriate project management tools to implement the SES Project to the best of their abilities and resulted in:
- Energy resource assessments and energy demand assessments completed for 15 village communities (Ri).
- 24 feasibility studies integrating cost-benefit assessment methodology for RE/EE interventions completed.
- 20 energy audits for basic-necessity facilities, agricultural processes and public community buildings completed.
- Dissemination, promotion and demonstration on the uses/applications of RE/EE solutions at village community (Ri) level through NTDCs.
applying effective adaptive management in planning procurement activities in order to accomplish the following project results:
- Successfully delivered and installed 200 sets of solar PV systems in 170 public institutions across 15 Ris (from project reports):
  - 29 sets of solar PV systems in 26 productive facilities and implemented energy efficiency interventions in 11 productive processes.
  - 171 sets of solar PV systems in 144 social service institutions.
- Successfully implemented energy efficiency interventions in 67 public institutions (improved Ondol floor heating system, thermal insulation in buildings including double glazed windows and doors, retrofitting with foamed cement bricks on building envelop, EE coal stoves, EE biomass stoves, and solar PV systems) in 62 community social service providers across 15 Ris such as kindergarten, nursery, schools, hospitals and clinics.
  
  However, improvements/consistencies could still be further strengthened in the following areas:

Registering/updates of assets/delivered items list and tagging of assets/delivered items by project team, in full compliance and adherence to relevant UNDP Policies and Procedures and UNDP DPRK Guidelines for Field Monitoring Visits, should be more consistent to ensure complete and proper physical verification and handover for the intended use/purpose.
Signed acceptance at time of delivery and physical verification of all assets/items from the project, while continuing to monitor on the use of delivered items and assets in full operations, should be more consistent. This ensures successful delivery onsite and the use of the delivered items for their intended purpose to achieve the desired project results.
Field data collection to measure effectiveness and impact on completed project activities.
For improved financial accountability and transparency purposes as part of demonstrating the efficient use of funding on project output-based activities, future financial reporting processes and templates of UNDP DPRK projects should:

track and report consistent financial figures (budget and actual expenditure).
have consistent comparisons between budget and actual expenditure, as per project outputs, based on project CDRs, for submissions of all relevant project reports (including annual progress reports and submissions to PSC meetings).

Lesson Learned:

To maintain sustainability and determine any project output/activity effectiveness and impact, even after any formal hand-over and/or completion of project output technical support and assistance, it is important that project teams, at minimum during project implementation, still continue monitoring and reporting on post project initiatives, including the use of the assets and delivered equipment items after handover to project beneficiaries. This would ensure successful delivery onsite and the use of the delivered items for their intended purpose to achieve the desired project results.

For improved financial accountability and transparency purposes, financial reporting processes and templates should be consistent, especially on the:

tracking and reporting of financial figures (budget and actual expenditure).
consistent comparisons between budget and actual expenditure to demonstrate the efficient use of funding on project output-based activities.

Conclusion #3: SES model has potential for replication across DPRK but requires strong national ownership and commitment as the key to overcome any difficulties faced and achieve optimum results

An important result demonstrated in the SES Project was how the intended project outputs addressed country priorities and also fit within the county development priorities with new strategies and initiatives being planned for sustainable living and livelihoods. This was further strengthened with strong support and commitment from National and Local Counterparts.

The high level of national and local ownership ensured sustainability and positive environmental impact, despite the SES Project encountering external challenges that severely constrained the project beneficiaries.

The SES model has the potential to be replicated across DPRK in close partnership collaboration with National and Local Counterparts. To ensure the continuity and also strengthening of national ownership, future replication projects should also be accompanied by appropriate capacity building activities at local county and village levels. However, this replication must also be complemented with fully sustainable and well-equipped energy supply chains to benefit the end-users at the county and village community (Ri) level.

Lesson Learned:

Strong national ownership combined with strong commitment/support and participation from CPCs and village communities (Ris) is key to accelerate the SES model to overcome any difficulties faced and achieve/bear lasting results.
Replication of knowledge/operational capabilities and capacities of National Consultants to enhance the pool of national and local resources are strongly recommended.

Conclusion #4: Significant delays through the sanctions exemptions/clearance process and the extended banking channel disruption/closure hindered project implementation and severely affected UNDP’s reputation of not being able to effectively deliver

Significant delays through the sanctions exemptions/clearance process and the extended banking channel disruption/closure hindered project implementation and have severely affected UNDP’s reputation as an organization of not being able to effectively deliver.

However, many other significant achievements in the SES Project at village community (Ri) level through the use of solar PV systems in 170 public institutions and EE retrofitting measures in 67 public community buildings across 15 village communities (Ris) should be given more on-the-ground recognition for UNDP’s unique contributions.

Lesson Learned:

Stronger on-the-ground visibility on UNDP’s unique contributions would be required at current SES project sites and future SES-related interventions (such as UNDP logos, nameplates, asset/delivered item tags), and communication of project results among international and national stakeholders (through a suitable communications platform for active sharing of information and lessons learned). UNDP’s reputation as an organization to deliver results would need to be restored.

It is important to:

better manage village community (Ri) expectations to avoid/minimize potential economic losses to counties/Ris due to extensive surveys, project document preparation, frequent site visits, and extended/delayed/disrupted delivery times of UNDP assets/items to project sites
impart knowledge to local counties/village communities (Ris) on more effective electricity usage and better control of the demand and energy consumption
observe and pay attention to safety measures and procedures for RE/EE equipment to minimize/prevent occupational accidents and hazards from occurring
conduct an independent impact evaluation study as a future project output/activity component to measure impact effectiveness, final end-line indicators and actual benefits gained
ensure the use of assets/delivered items for their intended purposes

Findings

3. findings

3.1 Project Design

3.1.1 Project Document (PRODOC) Formulation

The SES PRODOC indicated that the earliest commencement of the SES Project formulation was a UNDP fact-finding mission to conduct a rural energy survey in June 2014. A detailed assessment indicated to implement local-level energy solutions in rural areas through an approach that entails:

the establishment of delivery models enabling the sustainable supply and operation of energy solutions in rural areas
the introduction of renewable energy technologies (RE), and solutions for more efficient energy use (EE) and energy conservation (EC)
the increase of county-level energy self-reliance by enhanced ownership and technical and managerial competencies for the sustainable use of local renewable energy resources.

The SES PRODOC developed TORs to recruit suitable project team members to implement and manage the SES Project. The evaluation determined that the SES Project Team (comprising one International Project Manager, one National Technical Coordinator and One National Administrative Assistant) had the project management expertise and suitable technical expertise to deliver the project outputs which are technically complex and required specialised expertise and knowledge in RE/EE.

3.1.2 Analysis of Results and Resources Framework (Project Logic/Strategy and Indicators)

In reviewing the effectiveness and efficiency of the SES Project in meeting its outcome, the evaluation reviewed the SES Project’s Results and Resources Framework in relation to the UNDP DPRK CPD (2011 to 2015) and UNSF (2011 to 2016, 2017 to 2021) on the strategic priorities, outcomes, outputs and the primary applicable key environment and sustainable development key result areas (KRAs). The evaluation assessment also addressed the SES Project’s strategy, indicators, baseline, end of project target, source of verification, and risk and assumptions.

The evaluation reviewed that the SES Project’s Results and Resources Framework design has taken careful consideration of the UNDP DPRK CPD and UNSF outcomes and was aligned to the key environment and sustainable development KRAs. Furthermore, the SES Project’s Results and Resources Framework was prepared with in-depth thinking, accurately described the end of project goals, listed the sources of verification, and appropriately identified the risks and the assumptions.

The Results and Resources Framework was clearly described with the indicative activities and end of project targets. There were 15 indicators in total which reflected against outputs and activities.

The SES Project took extensive consideration to stakeholder participation in project design, decision making, planning, implementation and monitoring. For example, the National Counterparts and Local Counterparts were invited to contribute to designing of project interventions and technical discussions on the output activities. This translated to an increase in confidence and ownership of project activities in the SES Project implementation.

The SES Project’s outcome and outputs were consistent with the DRPK Government’s national priorities. A consultative approach with the National and Local Counterparts was followed in the development and design of project outputs and activities, resulting in strong project ownership and commitment.

The SES Project’s proposed outcome and outputs of the Project individually addressed specific needs identified and collectively presented a comprehensive solution to strengthen local capacity for improved nutrition and food security.

The SES Project also aligned with local county development plans and reinforced stakeholders’ engagement and supported their achievement of priorities. The SES Project design was also strategically aligned and consistent with the UN MDGs and subsequent UN SDGs.

The evaluation further noted that the SES Project’s expected results in the SES PRODOC are more output-oriented (WHAT IS BEING PRODUCED - EFFICIENCY) than outcome-oriented (WHAT IS THE VALUE/BENEFIT/ CHANGE/IMPACT - EFFECTIVENESS). While this is not an assessment of the SES Project Team’s performance, the evaluation is of a view that future PRODOC design should consider a balance of expected results with outcome-oriented targets/indicators to determine the effectiveness.

3.1.3 Risks and Assumptions

The SES PRODOC had appropriate risk assessments with impact and probability ratings, and prepared corresponding counter-measures/management responses which were appropriate at that point of time and for the project duration (2015 to 2019). The SES Project identified a total of 15 risks:

1 governance risk
3 operational risks
5 strategic risks
3 financial/fiduciary risks
3 sustainability risks

However, the evaluation reviewed that the risk assessments could be further extended to be part of the Results and Resources Framework to identify the key risks and appropriate counter-measures/management response for each of the 4 SES Project outputs. Many of these activities would have governance, operational risks, strategic risks, financial/fiduciary and/or sustainability risks that would require appropriate counter-measures/management responses.

The evaluation also determined that the SES PRODOC’s risk analysis did not account for scenarios of extreme UN sanction measures and the extended banking channel disruption/closure. Furthermore, the implementation of the SES PRODOC’s counter-measures/management responses did not appropriately resolve the significant change of events caused by the UN Sanction measures and the extended banking channel disruption/closure over the project duration.

3.1.4 Lessons from Other Relevant Projects Incorporated into Project Design

The evaluation observed that the SES Project Team took opportunity to align the SES Project with the CBDRM Project to maximize the synergy effects (more details found in Section 3.3.8).

The SES Project was also built from the experience and lessons learned from two previous UNDP DPRK projects:

Small Wind Energy Development Programme Rural Areas (SWEDPRA) Project
Sustainable Rural Energy Development (SRED) Project

For example:

the SES Project focused on the attainment of effective and sustainable local energy solutions that generate positive impact among rural beneficiaries, rather than involving in technology development.
the SES Project focused on implementing RE/EE technologies and solutions at community-level (such as public institutions , buildings and facilities) rather than household level (individual homes). This would be through area-based clustering or crowding-in approach to maximize demonstration effects and awareness raising.

3.1.5 Planned Stakeholder Participation

The SES Project generated strong stakeholder interest, especially at the DPRK national/central government ministries and Local Counterparts such as CPCs and NTDCs in Kaechon City and Hoechang, Singye, Unsan, Yonsan, and Yangdok Counties.

In terms of project design, the proxy indicators would be the number of stakeholders involved in planning and attendance during the project formulation/planning meetings. The evaluation interviews with National and Local Counterparts indicated sufficient evidence of direct involvement based on detailed accounts of the project outputs.

The minutes of the PSC meetings recorded perfect attendance and representations from the National Counterparts. The proxy indicators from M&E Field Monitoring Visits for participation at the project implementation stage indicated high project output ownership, perfect attendance at project field site meetings, capacity development/knowledge dissemination activities, and the visible evidence of construction/installation taking place. During the evaluation interviews, there were high levels of project output-ownership as the Local Counterparts and beneficiaries were able to provide extensive technical details of their project outputs.

3.1.6 Replication Approach

Replication and up-scaling are fundamental to the SES Project as it provides the opportunity to build on best practices and lessons learned, and expand the reach and impact of its project outputs. As such, UNDP, government agencies, international agencies/organizations and the private sector would utilize these given opportunities to support the replication and up-scaling of the most successful projects and practices through their networks and contacts.

The SES Project has the potential for replication in other provinces/counties in DPRK through:

Methodologies and approaches for energy resource assessments, energy demand assessments, energy audits and feasibility studies including cost-benefit analysis
Development and implementation of County Energy Management Plans (CEMPs)
Dissemination, promotion and demonstration on the uses/applications of RE/EE solutions at village community (Ri) level through NTDCs
Roll-out implementation of suitable RE/EE interventions at public buildings, institutions and facilities
Establishment of institutional frameworks and energy supply chains at county level to continue the sustainability of RE/EE equipment and interventions.

3.1.7 Management Arrangements

Execution Modality: In accordance with the SES PRODOC, the SES Project modality was Direct Implementation Modality (DIM) which meant the project execution and implementation would be undertaken directly by UNDP DPRK in accordance with UNDP Programme and Operations Policies and Procedures (POPP). The overall decision, including financial accountability would rest with UNDP DPRK and the SES Project was to be executed in coordination with relevant partners, including at the local county level, with a view to ensuring that effective assistance flowed directly to targeted beneficiaries.

Project Steering Committee (PSC): The PSC was established to provide high-level oversight and to steer the SES Project. The PSC is responsible for high-level management decisions and policy guidance required for implementation of the project, including recommendations and approval of project plans, budget and revision. The PSC membership comprised the following key stakeholders:

UNDP DPRK:
- Deputy Resident Representative of UNDP DPRK (PSC Chairperson)
- SES Project Manager
- Programme Analyst
- M&E Specialist
Government of the DPRK:
- Coordinator of National Coordinating Committee (NCC) for UNDP (PSC Co-Chairperson)
- Representative of Ministry of Electric Power Industry (MEPI)
- Representative of State Academy of Sciences (SAOS)
- Representative of State Commission of Science and Technology (SCoST)
- Representative of Central Bureau of Statistics (CBS)
  
  The evaluation reviewed that PSC decisions in relation to the SES Project were effective and adhered to standards that ensure efficiency, cost effectiveness, transparency, effective institutional coordination, and harmony with overall priorities of the Government of DPRK and UNDP.
  
  The PSC was first constituted in April 2016 and met regularly every quarterly. The meeting minutes for all meetings made available showed that the PSC effectively provided important directions and oversight. In addition, the PSC was also successful in advising on technical aspects of project implementation, discussions and deliberations on the external/environmental challenges faced in relation to procurement and prioritization of interventions keeping project cost considerations in view.
  
  UNDP: As the DIM agency, UNDP offered substantive support services to the SES Project, which included project management/administration, financial reporting, procurement support, and technical advisory services. The SES Project updates to the PSC, Project Annual Progress Reports, Programme and Project Field Monitoring Visits (FMV) Reports were comprehensive and timely produced. These reports covered many details and provided insights into project implementation, overall management, the many challenges faced in project implementation and mitigations/counter-measures to overcome the barriers.
  
  Project Counterparts: At the National/Central level, the DPRK government agencies involved in the project were:
National Coordinating Committee (NCC) for UNDP
Ministry of Electric Power Industry (MEPI)
State Academy of Sciences (SAOS)
State Commission of Science and Technology (SCoST)
Central Bureau of Statistics (CBS)

At the local level, the main project partners were CPCs, NTDCs and beneficiaries of:
- Hoechang County, South Pyongan Province
- Singye County, North Hwanghae Province
- Yonsan County, North Hwanghae Province
- Unsan County, North Pyongan Province
- Kaechon City, South Pyongan Province
- Yangdok County, South Pyongan Province

The SES Project Team travelled to the respective county locations to hold regular and quarterly meetings with the project partners to review the project progress and initiate early corrective actions.

The SES Project FMV reports indicated effective discussions to resolve project management and coordination issues, and also contained details of reviews and actions taken. The Programme FMV, led by the M&E Specialist and CO Management, validated the results achieved. All recommended actions were consistently followed up and presented by the M&E Specialist at PSC meetings and captured in the quarterly programme and oversight FMV reports. These reports were subsequently sent to the UNDP Regional HQ Bureau as required by the UNDP DPRK ICF. The evaluation reviewed that there was a focus on results and activity scheduling across activities and outputs. Progress was reviewed against the objectives and targets set in the SES PRODOC’s Results and Resources Framework. The Project and Programme FMV reports were written to reflect the progress achieved against targets.

Project Management Unit (PMU): Being a DIM agency, the UNDP formed a PMU comprising one International Project Manager, one National Technical Coordinator and one National Administrative Assistant.

The PMU would be fully responsible for the coordination of National/Local Counterparts for project execution in a timely manner and within budget. The PMU facilitated effective project planning, that included preparation of annual work plans and project monitoring and reporting. The PMU was charged with coordinating and facilitating the procurements. As a curator, the evaluation reviewed that the PMU had effectively and efficiently held all the records, publications and minutes of meetings pertaining to the SES Project.

3.2 Project Implementation

3.2.1 Adaptive Management

The SES Project was formally signed off on 26 August 2015. However, there were prolonged delays at the start of the project due to the:

extended period of banking channel disruption/closure for funds transfer to the UNDP DPRK CO

Due to the early UN Sanctions on DPRK (UN Resolutions #2087 and #2094), the UNDP DPRK CO had to implement prolonged periods of organizational cash conservation mode due to the lack of funds being transferred into DPRK. Hence, there were minimal funds to implement any project activities and eventually slow progress in delivering project results.

Late recruitment of the SES Project Team

The extended period of banking channel disruption/closure created uncertainties for the UNDP DPRK CO and possibly resulted in the late recruitment of the SES Project Team. The Project Manager, National Technical Coordinator and Project Administrative Assistant were eventually on board in the 1st quarter of 2016.

Despite the early and recurring setbacks, the evaluation reviewed that the SES Project Team displayed good project management abilities and effectively utilised appropriate project management tools to implement the SES Project to the best of their abilities.

The project implementation was delayed by 7 months from August 2015 until March 2016, with the first PSC Meeting involving the SES Project Team on board held on 21 April 2016. The SES Project Team effectively applied adaptive management in planning by having to reschedule the timelines for activities in order to accomplish the project outputs, with activities starting in 2016.

Table 1 below showed the implementation status of each SES Project output as assessed by the evaluation. The evaluation noted that the SES Project would have produced a significantly different implementation status if there were no UN Sanctions imposed on DPRK and there was no banking channel disruption/closure issue to deal with.

Table 1: SES Project Implementation Status

SES PRODOC	Implementation Status1
Outcome Increased standards of living and sustainable livelihood
Output 1 Information about energy resources and feasible RE/EE solutions updated and made accessible to local beneficiaries.	Fully Achieved
Output 2 Increased technical know-how of county-level personnel for energy planning and sustainable management of local renewable energy resources.	Almost Fully Achieved
Output 3 Strengthened supply chains for the delivery of appropriate RE/EE solutions for local communities in rural areas.	Partially Achieved
Output 4 Increased energy security and self-reliance of rural population through the implementation of RE/EE solutions for local communities.	Partially Achieved

Note:

The implementation status is purely based on the desired results of the SES PRODOC. It has not been moderated based on the implications and resultant consequences attributed to the 6 UN Sanctions imposed on DPRK in 2016 and 2017, and the extended period of banking channel disruption/closure which severely disrupted funds being transferred into DPRK to implement project activities.

In the case of Outputs 3 and 4, the evaluation reviewed that these implications and resultant consequences were beyond the control of the SES Project Team and the UNDP DPRK CO, and there were minimal or no alternative adaptive management measures that could have produced a better outcome.

The evaluation assessed and made the following observations relating to safety and durability:
- Safety handling of RE technologies/solutions – there is a need to pay close attention to handling live wirings/connections, and the cleaning/maintenance and durability of the RE technologies/solutions during extreme weather conditions such as typhoons, heavy snow storms. In relation to open live equipment, care and attention should be undertaken to minimize exposure to potential electrical shock and injuries.
- Access to the site housing the RE technologies/solutions – there is a need to improve the pathways to access the sites to minimize potential injuries and hazard
  
  The evaluation assessed that specific communication aspects of the SES Project would need to be strengthened as follow:

Communications – enhanced visibility and communication of project results
Even though facing significant setbacks on procurement-related activities to deliver RE/EE equipment and materials, the SES Project did make significant impacts at village community (Ri) level such as the successful delivery and implementation of:
- 200 sets of solar PV systems in 170 public institutions across 15 Ris (from project reports):
  - 29 sets of solar PV systems in 26 productive facilities and implemented energy efficiency interventions in 11 productive processes.
  - 171 sets of solar PV systems in 144 social service institutions.
- energy efficiency interventions in 67 public institutions (improved Ondol floor heating system, thermal insulation in buildings including double glazed windows and doors, retrofitting with foamed cement bricks on building envelop, EE coal stoves, EE biomass stoves, and solar PV systems) and 62 community social service providers across 15 Ris such as kindergartens, nurseries, schools, hospitals and clinics.
  
  However, during the field visit to project sites, the evaluation noted many items were confirmed as funded by the UNDP SES Project but missing UNDP-registered name tags, name plates and/or logos that would enable local counties and village communities to recognize and acknowledge UNDP’s contributions to improving livelihoods. The evaluation determines that UNDP’s unique contributions should be given more on-the-ground recognition through stronger visibility and communication of project results at project sites such as UNDP logos, nameplates, asset/delivered item tags.
  
  The evaluation also observed that a number of items procured/provided under the SES Project were not updated accordingly into the asset/delivered items list. This could potentially lead to be externally viewed/perceived as not physically accepted at time of delivery by the SES Project Team and/or not physically verified by the M&E Specialist, as required in the UNDP DPRK Guidelines for Field Monitoring Visits.

Communications – Management of expectations and reputational risk
There is a need to manage village community (Ri) expectations on (1) UNDP’s “inconsistent” delivery of items (such as RE/EE equipment) to different Ris to minimize the occurrence of unhealthy comparisons and unhealthy competitions between project Ris, and (2) prolonged delays in UNDP interventions to minimize/avoid potential economic and productivity losses to Counties/Ris.

UNDP DPRK has gained a reputation among national and local counterparts as an organization that failed to deliver on its promises. Restoring UNDP’s reputation as an organization that can effectively deliver would be a key priority. The evaluation would therefore find it beneficial for UNDP DPRK by:

prescribing conditions and mechanisms to implement “Force Majeure” or early termination of projects if need to; and
continuing field visits, as practical and as relevant as required during the project implementation period, to maintain relationships and communications with village communities (Ris).

3.2.2 Partnership Arrangements

The SES Project generated strong stakeholder interest and participation from National/Local Counterparts in DPRK. The stakeholders at the National/Central level were:

National Coordinating Committee (NCC) for UNDP
Ministry of Electric Power Industry (MEPI)
State Academy of Sciences (SAOS)
State Commission of Science and Technology (SCoST)
Central Bureau of Statistics (CBS)

At the local level, the main project partners were CPCs, NTDCs and beneficiaries of:
- Hoechang County, South Pyongan Province
- Singye County, North Hwanghae Province
- Yonsan County, North Hwanghae Province
- Unsan County, North Pyongan Province
- Kaechon City, South Pyongan Province
- Yangdok County, South Pyongan Province

There was evidence of strong interest and commitment at the local county level through stakeholder contributions (both financial and in-kind), roles and responsibilities to implement the SES Project activities.

Despite the external factors/challenges that were beyond the control of the UNDP DPRK CO, the partnership arrangement between SES Project Team and the National and Local Counterparts endured the challenging 4.5-year project period, and demonstrated great patience, understanding and resilience to overcome the difficulties faced. The fruits of this partnership agreement in challenging circumstances were the successful completion of many SES Project interventions as follow:

Energy resource assessments and energy demand assessments completed for 15 village communities (Ri).
24 feasibility studies integrating cost-benefit assessment methodology for RE/EE interventions completed.
20 energy audits for basic-necessity facilities, agricultural processes and public community buildings completed.
Dissemination, promotion and demonstration on the uses/applications of RE/EE solutions at village community (Ri) level through NTDCs.
Successfully delivered and installed 200 sets of solar PV systems in 170 public institutions across 15 Ris (from project reports):
- 29 sets of solar PV systems in 26 productive facilities and implemented energy efficiency interventions in 11 productive processes.
- 171 sets of solar PV systems in 144 social service institutions.
Successfully implemented energy efficiency interventions in 67 public institutions (improved Ondol floor heating system, RE hybrid system units, thermal insulation in buildings including double glazed windows and doors, retrofitting with foamed cement bricks on building envelop, EE coal stoves, EE biomass stoves, and solar PV systems) and 62 community social service providers across 15 Ris such as kindergartens, nurseries, schools, hospitals and clinics.

3.2.3 Project Finance

The SES Project had a duration of about 4.5 years (August 2015 – December 2019) with an approved funding of US$6,117,572. The details of the planned financing allocation based on the SES PRODOC are as follow:

Table 2: SES Project – Original Planned Budget as per SES PRODOC

SES Project	2015 and 2016 (US$)	2017 (US$)	2018 (US$)	2019 (US$)
Output 1	543,500	354,500	162,500	122,500
Output 2	141,750	539,250	199,250	156,750
Output 3	251,750	468,750	292,750	207,750
Output 4	142,750	1,168,322	827,750	537,750
Total	1,079,750	2,530,822	1,482,250	1,024,750

While the SES PRODOC did not include any co-financing from National/Local Counterparts, the evaluation reviewed that the Local Counterparts provided in-kind contributions (labor and construction materials) to assist the timely completion of SES Project activities.

The budget and actual expenditure of the SES Project is provided below in Table 3.

Table 3: Summary of Budget and Actual Expenditure (SES Project)

SES Project	2015 and 2016 (US$)	2017 (US$)	2018 (US$)	2019 (US$)
Operational Expenses (Actual)	1,573.41	323,321.58	4,878.37	4120.92
Output 1 (Actual)	190,077.29	480,374.36	174,522.36	82,493.57
Output 2 (Actual)	33,795.07	313,228.68	244,935.66	105,478.13
Output 3 Actual)	34,144.28	79,916.14	75,458.84	464,993.32
Output 4 (Actual)	68,803.31	1,157,608.30	166,697.68	162,968.91
Total (Actual)1	328,233.98	2,354,449.06	666,492.91	820,054.85
Utilization Rate (Actual/PRODOC Budget)	30%	93%	45%	80%

Note:

Actual figures are based on financial system extracts provided by the UNDP DPRK CO

The evaluation noted that the SES Project under-spent its allocated total project funds by about 32% and its utilization with an average of 62%. This was due to the banking channel disruption/closure, caused by the UN Sanctions, which disrupted funds from being transferred into DPRK. This further resulted in the SES Project’s inability to obtain funds to implement the SES Project procurement-related activities.

In considering the UN sanction measures together with recurring and extended banking channel disruption/closure which led to the UNDP DPRK CO activating the cash conservation mode to sustain the office operations, the SES Project Team displayed sound financial management processes to implement the relevant SES Project activities which were not affected by the UN sanction measures.

However, the evaluation reviewed that there were inconsistencies (and inconsistent templates) in the SES Project Team’s financial reporting processes due to different reporting requirements given.
1. Different reporting of SES Project budget figures
  
  The budget figures in the SES Project Annual Work Plans were different from that of the SES Project Annual Progress Reports. This was due to the different submission timelines required by different reports which led to different budget figures being reported, e.g. due to budget revisions made during the year.
2. Inconsistent reporting of SES Project actual expenditure figures
  
  The actual expenditure provided to the evaluation was based on actual expenditure according to project outputs. However, the actual expenditure in the SES Project Annual Progress Reports were based on actual expenditure, as per financial reporting templates being provided by UNDP DPRK CO, according to the categories of Project Activity, Management and Staff, General Operations Expenditure, and/or Common Services.
  
  3. Inconsistent reporting on comparison of SES Project budget versus actual expenditure figures
  
  The SES Project Team did not provide budget and actual expenditure figures in PSC meetings. However, the SES Project Annual Progress Reports report these comparisons for the calendar year period but not at output levels as the SES Project Team followed the financial reporting templates being provided by UNDP DPRK CO. The evaluation further noted that only the SES Project’s CDR run was attached at the time of the report submission.
  
  For improved financial accountability and transparency purposes as part of demonstrating the efficient use of funding on project output-based activities, future financial reporting processes and templates of UNDP DPRK projects should:

track and report consistent financial figures (budget and actual expenditure)
have consistent comparisons between budget and actual expenditure, as per project outputs, based on project CDRs, for submissions of all relevant project reports (including annual progress reports and submissions to PSC meetings)

3.2.4 Monitoring and Evaluation (M&E) Activities Used for Adaptive Management

The M&E framework consisted of local monitoring and reporting as well as international independent evaluations. Both the SES Project Manager and the UNDP DPRK Programme M&E Specialist were responsible for the preparation and submission of the M&E reports and evaluations at project and programme levels respectively, as stated in the SES PRODOC. Table 4 below summarizes the achievement of monitoring actions as required by the SES PRODOC.

Table 4: M&E Plan and Completion Status

Type of M&E Activity/Report	Frequency/ Timing	Status	Comments
Detailed Quarterly Workplan	Every beginning of the quarter	Completed	Detailed workplans for 2016, 2017, 2018 and 2019 completed
Annual Workplan and Budget	Beginning of each year	Completed	Detailed workplans with budget for 2016, 2017, 2018 and 2019 completed
Quarterly Progress Report	Quarterly	Completed	Reports completed every quarter in 2016, 2017, 2018 and 2019
Annual Progress Report	End of year	Completed	Reports completed in 2016, 2017 and 2018. The Annual Progress Report for 2019 is to be completed by the SES Project Manager in December 2019
Mid-Term Review	End of Year 2	Completed	This M&E activity was delayed with one MTR report completed by an independent evaluator in August 2018
Terminal Report	End of Project (end of Year 4)	In progress	One Terminal Report to be completed by the SES Project Manager in December 2019
Terminal Evaluation and Audit	End of Project (end of Year 4)	In progress	One Terminal Evaluation report to be completed by an independent evaluator in January 2020
Mission reports	After each mission	Completed	Mission reports by individual experts (International and National) completed
Other Reports and Deliverables	After each TA or sub-contract	Completed	Reports and deliverable by individual experts (International and National) completed
Monitoring Reports	After every field visits	Completed	Field Monitoring reports by SES Project Team and UNDP DPRK Programme M&E Team completed
Financial records & reporting	Continuous	Completed	Financial records and reporting completed

The UNDP DPRK CO and the SES Project Team proactively responded with specific adaptive management measures to recommendations from MTR as shown below in Table 5:

Table 5: Management Response to SES Project MTR Recommendations

SES Project MTR Recommendation

Management Response

SES project is experiencing significant delays in procurement of RE and EE systems that need to be piloted. Therefore, the project shall develop a contingency plan to refocus on wider implementation of EE installations and consolidating knowledge gains among engineers and decision makers at the provincial and county level in order to prepare for (eventual) scale-up of the initiatives by the government.

Conduct in-country study tours (bringing together national partners) including NTDCs to discuss and share lessons and areas for intervention

National consultants select most qualified experts at the NTDCs, Counties and Ris with the support of SES project team.
Such visits could be expanded beyond the project and to the places such as Jangchon Co-operative farm, Pyongyang City and Natural Energy Institute under the State Academy of Science.
Produce a report following in-country study tours and commitment received from its participants in scaling up of the EE interventions.

Establish a regular communication with communities in the implementation of EE technologies with the ownership of MEPI by sustaining momentum in activity implementation.

Project team works closely with county-level stakeholders (i.e. managers of public buildings and national consultants) to improve their data collection on how many people (disaggregated by sex, age, disability) access services, and the impacts that the RE and EE improvements have had on particular humanitarian outcomes, particularly health.

Firm up of the methodology of data collection from local stakeholders

Rely on a good expertise of national consultants hired under SES
Train them on the data collection on how many people (disaggregated by sex, age, disability) access public services.

It is critical that the project team monitor results of capacity building at the output level, beyond demonstrating the successful implementation of capacity building activities. Below proposed output indicators, and targets, which aim to allow for the collection of data which can be used to analyze the meaningful change in capacity and quality effected by the project to date. These suggestions aim to identify possible entry points for the project, conscious of data access limitations.

In consultation with MEPI following indicators will be monitored by SES project. These will be reported in the Quarterly Progress Reports:
- Extent to which County Energy Management Plans receive budgetary support for implementation from the county governments (Target: County Energy Management Plans receive at least 75% of necessary funding)
- Extent to which NTDCs are operational (Target: plans and budgets for 3 NTDCs approved)
- % change in improved attendance of 5-7-year-old children in target kindergartens between November-March (Target: at least a 50% increase)

It is recommended that instead of having joint reports following field visits, whether it is for implementation and/or monitoring purposes, team members should submit individual BTORs separately for project and programme.

Projects and programme team will submit separate BTORs upon field missions.
Quarterly Progress Reports (QPRs) should be completed by the project team (lead by the Project Manager), with quality assurance of the data and analysis undertaken.
Report qualitative changes of the projects in ROAR through captured qualitative results from projects.

With the inclusion of more qualitative indicators at the output level, it is hoped that more meaningful analysis of the humanitarian importance of the project will be captured, bearing in mind the sensitivities in sharing project results publicly due to the complex geopolitical context under which UNDP operates in DPRK. Following are some of the key actions that will be taken to improve the reporting of qualitative changes that the project is leading on the ground.

SES project to share communication material (videos/brochures…) with relevant parties including Bangkok Regional Headquarters

As the SES project will end by December 2019 according to the ProDoc, it’s the right time to start deploying its exit strategy to meaningfully consolidate the results achieved till date and ensure the sustainability of the project activities and results.

SES project shall organize a National Partners Meet to assess what was done better and what else need to be completed in fulfilling the needs of the communities.
- At this meet, SES project must encourage the local communities to make in-kind contributions in the absence of procurement activities.
- Consolidating SES project results till date

The evaluation reviewed that the M&E process at the project and programme level was very comprehensive. The UNDP DPRK Programme M&E Team showed high competency in:

conducting field monitoring visits every quarterly to assess the progress of the SES Project outputs. This included the verification of delivered items and assets through the identification of UNDP item/asset identity tags at the field sites, the onsite testing of equipment delivered by UNDP, and monitoring the use of the delivered items and assets to ensure sustainable operations.
producing high quality quarterly and annual Programme monitoring and oversight reports, as required by the UNDP DPRK ICF and UNDP DPRK CO Guidelines for Field Monitoring Visits, with key findings and analysis of progress towards results, project performance and implementation issues.
providing key recommendations and corrective actions/measures to further improving the SES Project, and monitoring the implementation of these key recommendations and corrective actions/measures until completion.
updating the M&E progresses at all PSC meetings.

The SES Project Team showed high competency in:
conducting project field monitoring visits every quarterly to assess the progress of the SES Project outputs. This included the onsite testing of equipment delivered by UNDP.
producing high quality quarterly and annual project progress reports and presenting them at all PSC meetings.
identifying key issues faced, and providing key recommendations and corrective actions/measures to address these key issues.
updating project implementation monitoring progress at all PSC meetings.

However, the M&E process at the project level could be further improved in 2 key areas:

Verification of delivered items and assets through the identification of UNDP item/asset identity tags at the field sites

In numerous occasions during the field mission trips, the project beneficiaries would show the UNDP items/assets and would also compliment the high-quality conditions of UNDP items/assets as compared to other similar items/assets delivered onsite by other organizations.

However, the Evaluator was unable to establish or fully verify whether the sighted items/assets were actually from UNDP due to a lack of identification either with an UNDP logo or a UNDP identification tag. The Evaluator would frequently rely on the M&E Specialist, Programme Analyst and DPRK Government focal point onsite for confirmation and verification.

Tagging of assets/delivered items should be more consistent to clearly distinguish UNDP’s quality items/assets, while also adhering to UNDP DPRK Guidelines for Field Monitoring Visits to ensure complete and proper physical verification and handover for the intended use/purpose.
Field data collection to measure effectiveness and impact on completed project activities

While the SES Project has consistently reported the impact through reduced coal usage and electricity consumption, there is a need for the SES Project Team to collect data to measure the effectiveness and impact on the village community beneficiaries.

For example, the installed 200 sets of solar PV systems in 170 public institutions across 15 Ris, and energy efficiency interventions in 67 public institutions and 62 community social service providers across 15 Ris such as kindergartens, nurseries, schools, hospitals and clinics, should be continuously monitored with relevant data collected to determine its positive impacts and actual benefits gained on individual, family and community well-being among the village communities.

Enabling the field data collection to measure effectiveness and impact on village communities would further strengthen the:

overall sustainability results of the SES Project pilot activities
case for future replication of the SES model in other counties/village communities (Ris) in DPRK

3.2.5 Implementing Agency

The SES Project adopted the direct implementation modality (DIM) which meant that UNDP DPRK would be the Implementing Agency with a dedicated project team based in the UNDP DPRK CO. An International Project Manager would be recruited and be responsible for the daily management of the project with assistance from recruited national project staff (comprising one National Technical Coordinator and one National Administrative Assistant). The SES Project Team would further engage International and/or National Consultants as required based on the SES Project’s technical requirements.

The SES Project also formed a Project Steering Committee (PSC) to guide the project direction and address any challenges. The PSC was co-chaired by the UNDP Deputy Resident Representative (DRR) and the National Coordinator from the DPRK National Coordinating Committee (NCC) for UNDP, with participation of representatives from the MEPI, CBS, SCoST, SAOS and other institutions as needed at the central level.

The SES Project would also work closely with Local Counterparts such as CPCs and NTDCs from:

Hoechang County, South Pyongan Province
Singye County, North Hwanghae Province
Yonsan County, North Hwanghae Province
Unsan County, North Pyongan Province
Kaechon City, South Pyongan Province
Yangdok County, South Pyongan Province

The evaluation established that there were strong working relationships between the UNDP DPRK CO, the SES Project Team and National/Local Counterparts and project beneficiaries at the county/village community (Ri) level. These working relationships were frequently tested by the slow progress of the SES Project. Key representatives of the National/Local Counterparts expressed disappointments at the prolonged delays and unsuccessful implementation of the SES Project procurement-related activities.

Many of these expressed disappointments were understandably justified as, in their views, tangible results were not delivered, especially the procurement activities for RE/EE equipment during the 2nd half of the 4.5-year project duration. Despite these procurement setbacks, the National/Local Counterparts expressed deep gratitude and appreciation on the limited but successful implementation of the SES Project interventions that has a great potential for scale up and replication.

The National/Local Counterparts expressed deep gratitude and appreciation for the SES Project Team who had done their very best, in the midst of many external factors/challenges faced, to implement the project with some significant success.

The National/Local Counterparts, while fully understanding that the external factors/challenges such as the UN Sanctions and the geo-political situation had severely affected the SES Project, highlighted their disappointment in the UNDP as an organization for not being able to deliver the results.

3.3 Achievement of Project Results

The evaluation rated the SES Project’s project results according to the evaluation ratings table listed below in Table 6.

Table 6: Evaluation Overall Results/Impact Rating

Evaluation Ratings for Overall Results/Impact, Relevance, Effectiveness, Efficiency, Basic Human Needs, Gender Equality, National Ownership

Sustainability Ratings:

6. Highly Satisfactory (HS): no shortcomings

5. Satisfactory (S): minor shortcomings

4. Moderately Satisfactory (MS): moderate shortcomings

3. Moderately Unsatisfactory (MU): significant shortcomings

2. Unsatisfactory (U): major shortcomings

1. Highly Unsatisfactory (HU): severe shortcomings

4. Likely (L): negligible risks to sustainability

3. Moderately Likely (ML): moderate risks

2. Moderately Unlikely (MU): significant risks

1. Unlikely (U): severe risks

Additional ratings where relevant:

Not Applicable (N/A)

Unable to Assess (U/A)

3.3.1 Overall Results/Impact

The evaluation rated the SES Project’s overall results/impact with reference to its 4 project outputs as per stated in the SES PRODOC. The overall results/impact are presented below in Table 7.

Table 7: Overall Results/Impact – SES Project

SES PRODOC

Achievement Rating

Comments

Outcome

Increased standards of living and sustainable livelihood

Output 1

Information about energy resources and feasible RE/EE solutions updated and made accessible to local beneficiaries.

6/6

(Highly Satisfactory)

No shortcomings

Energy resource assessments and energy demand assessments completed for 15 village communities (Ri).
24 feasibility studies integrating cost-benefit assessment methodology for RE/EE interventions completed.
20 energy audits for basic-necessity facilities, agricultural processes and public community buildings completed.
Dissemination, promotion and demonstration on the uses/applications of RE/EE solutions at village community (Ri) level through NTDCs.
Technical study tours in Serbia and China on RE/EE solutions conducted.

Output 2

Increased technical know-how of county-level personnel for energy planning and sustainable management of local renewable energy resources.

5/6 (Satisfactory)

Minor shortcomings

Comprehensive training workshops conducted by international and national consultants for decision makers, energy experts and technicians at County and community-level on energy resources, conservation, conversion, efficiency, and planning
Technical study tours in Serbia and China enabled participants in application of knowledge.
Training manuals on RE/EE solutions for improvement of rural livelihoods in DPRK developed.
Promotional and educational activities were conducted to increase awareness on the use of RE/EE solutions at village community (Ri) level.
6 counties developed and refined County Energy Management Plans with technical assistance from the National Consultants. However counties still require further technical support before they could fully “run” on their own.
There is a need to replicate the knowledge/operational capabilities and capacities of National Consultants to enhance the pool of national and local resources
Based on project reports, about 450 ha of biomass resources were sustainably managed by Ris through in-kind afforestation and reforestation of fast growing firewood trees such as Acacia, Poplar, bamboo willow (this would need to be independently verified).

Output 3

Strengthened supply chains for the delivery of appropriate RE/EE solutions for local communities in rural areas.

4/6 (Moderately Satisfactory)

Moderate shortcomings

Capacities of community and county-level workshops involved in manufacturing facilities and/or service centres for RE/EE products such as foamed cement bricks, EE stoves, double glazed windows, insulated doors, etc. improved.
Challenges in procurement due to UN Sanctions and banking channel disruption/closure to procure in-country (beyond the control of the UNDP DPRK Project Team and CO) could not fully deliver the all necessary equipment and tools for efficient manufacturing and/or assembly of RE/EE technologies. However local abilities could not be fully realized/achieved to their full potential.
County NTDCs functioning as expert centres, with roles and responsibilities, organization and management of these centres finalized for technical support, quality assurance, and performance monitoring of RE and EE applications identified. However, independent post impact evaluation study required to verify how well these expert centres are functioning and its impact on the local communities.

Output 4

Increased energy security and self-reliance of rural population through the implementation of RE/EE solutions for local communities.

6/6

(Highly Satisfactory)

No shortcomings

Renewable energy solutions identified and implemented through energy resource/demand assessments (the evaluation noted that this was a crucial/important strategic decision taken by the SES Project)

Successfully delivered and installed 200 sets of solar PV systems in 170 public institutions across 15 Ris (from project reports):

29 sets of solar PV systems in 26 productive facilities and implemented energy efficiency. interventions in 11 productive processes
171 sets of solar PV systems in 144 social service institutions.

Successfully implemented energy efficiency interventions in 67 public institutions (improved Ondol floor heating system, thermal insulation in buildings including double glazed windows and doors, retrofitting with foamed cement bricks on building envelop, EE coal stoves, EE biomass stoves, and solar PV systems) in 62 community social service providers across 15 Ris such as kindergartens, nurseries, schools, hospitals and clinics.

2/6 (Unsatisfactory)

Major shortcomings

Renewable energy solutions identified through 24 feasibility studies in targeted user groups in the pilot Counties were substantively not implemented
Challenges in procurement due to UN Sanctions and banking channel disruption/closure to procure in-country. (beyond the control of the UNDP DPRK Project Team and CO) could not fully deliver 21 out of the 24 feasibility studies.

The evaluation further noted that the SES Project Team had done its best to deliver and achieve the desired project results despite encountering significant external factors/challenges, mainly due to the 6 UN Sanctions in 2016 and 2017 and the recurring banking channel disruption/closure that prevented funds transfer into DPRK during the SES Project implementation.

3.3.2 Relevance

Relevance with national priorities

Achievement Rating: 5/6 (Satisfactory - Minor Shortcomings)

The SES Project was highly relevant and aligned with the DPRK national strategies and priorities. The SES Project was designed with humanitarian-oriented outputs and activities which were aimed to address the humanitarian needs of intended beneficiaries.

The SES Project’s relevance was further strengthened with National and Local Counterparts being involved and consulted during the project design and also during project implementation. The SES Project Team, particularly the Project Manager, also had suitable technical skillsets and competencies to deliver most of the project outputs which are technically complex and required specialised expertise and knowledge in RE and EE.

The SES Project’s relevance could be further improved if challenges in procurement due to UN Sanctions and banking channel disruption/closure severely disrupted the ability to procure internationally and in-country were appropriately resolved (which is beyond the control of the UNDP DPRK Project Team and CO) in response to the geo-political environment. Hence the SES Project’s relevance was affected as it could not fully deliver the required procurement-related activities to strengthen the energy supply chains (RE/EE tools/equipment/materials, civil works and construction).

Relevance with UNSF Outcomes and SDGs

Achievement Rating: 3/6 (Moderately Unsatisfactory - Significant Shortcomings)

The SES Project aligns closely with UNSF Outcomes 2.2, 3.2 and 4.3, MDGs 3, 4 and 7. The SES Project also contributes to the SDGs (SDG 7 on affordable and clean energy, and SDG 13 on climate action).

The UNSF (2017-2021) emphasized coherent and coordinated implementation in support of a common objective in order to achieve potential synergies among UN agencies and possibly with international organizations.

The evaluation assessed that the SES Project needed to improve its weak synergies with other UN agencies and international organizations with similar project/programme outputs and results. In particular, information sharing, communication of project results and valuable lessons learned should be further disseminated and strengthened on application, uses and impacts of RE/EE with other UN agencies and international organizations towards collectively achieving the UNSF Outcomes and also the SDGs.

3.3.3 Effectiveness

Achievement Rating: 5/6 (Satisfactory – Minor Shortcomings)

External factors/environment such as the extended banking channel disruption/closure and significant delays through the sanctions exemptions/clearance process hindered the SES Project’s implementation and severely affected UNDP’s reputation as an organization of not being able to effectively deliver (beyond the control of the UNDP project team and CO). This in-turn affected the desired Output 3 and 4 results to be fully achieved, particularly the procurement of RE/EE equipment and materials to strengthen energy supply chains.

There were significant results for the village community (Ri) beneficiaries which potentially contributed to the SDGs such as:

Hospitals/Clinics – Hospitals/clinics have reliable electricity and thermal insulation to treat patients in a conducive temperature-controlled environment and able to all year round (24 hours a day, including at night if required); Unsan County’s women hospital was able to operate all year round to further enhance the treatment of women’s well-being and health (including reproductive health disorders)
10-Day kindergartens and nurseries – children and female teacher/staff well-being and health were potentially enhanced with increased learning opportunities
Public welfare amenity facilities – cleaner air quality and improved temperature management; potential increase in work productivity and hence income-generating opportunities for employees

The above-mentioned output activities met the intended needs of the target beneficiaries at the county and village community (Ri) level. However, an independent impact evaluation study would be required as a future project output/activity component to measure the impact effectiveness, final end-line indicators and actual benefits gained.

3.3.4 Efficiency

Achievement Rating: 4/6 (Moderately Satisfactory – Moderate Shortcomings)

The project achieved the intended outcome. Out of the 4 outputs:

Output 1 considered fully achieved
Output 2 considered almost fully achieved
Outputs 3 and 4 considered partially achieved

As of 3 Dec 2019, project under-spent allocated total project funds by about 32%. This is mainly due to the inability to obtain project funds for procurement and delivery of RE/EE equipment/materials, which is caused by the delayed UN sanctions exemptions/clearance process and the extended banking channel disruption/closure.

Financial reporting processes and templates should be further strengthened for consistencies, financial accountability and transparency purposes in financial budgeting and accounting:

tracking progress of budget vs expenditure at output level for submissions of all relevant project reports (including APPRs), to demonstrate the efficient management and use of funding on project output-based activities, and align activity/output impact and results to the corresponding financial budgets
reporting these budget vs expenditure comparisons at output levels at PSC meetings

3.3.5 National Ownership

Achievement Rating: 5/6 (Satisfactory – Minor Shortcomings)

While the SES PRODOC did not include any DPRK counterparts to lead in implementing any project outputs, strong national ownership was achieved at the National/Central level through perfect attendance by DPRK counterpart representatives (NCC-UNDP, MEPI, CBS, SAOS and SCoST) of all PSC meetings.

The evaluation also found high national ownership through strong commitment and interest at the local county level with sustained results of initiation, knowledge/operational transfer and innovative creativity from the SES Project, as follow:

Increased knowledge and public awareness of the uses of RE/EE to improve rural livelihoods at the county/village community (Ri) level.
CBS benefited from the energy demand and resource assessments surveys conducted in the year 2016. Statisticians’ capacities and capabilities improved in the surveys related to energy access and related indicators.
Increased knowledge and strengthened capacities of national and local resources in RE and EE
High degree of national ownership at county and village community Ri level:

In-kind contributions through self-supplied equipment and material as well as labor work
Public buildings are now able to self-operate and maintain their own RE/EE solutions
Tailoring of training materials by National Consultants to the local context for stronger application by local counties and village communities (Ris)
Extensive utilization of National Consultants to provide on-the-ground advisory and practical hands-on to local counties and village communities (Ris)

However, the evaluation observed that county stakeholders still require continuous technical support (such as continued update of county energy management plans, conducting energy audits and conducting energy demand and resource assessments) before they could fully function on their own. Due to the incomplete procurement activities, local abilities for efficient manufacturing and/or assembly of RE/EE technologies could not yet be fully realized/achieved to their full potential.

While there are evidences of dissemination, promotion and demonstration on the uses/applications of RE/EE solutions at village community (Ri) level through the county NTDCs, there is an important need to further promote public awareness and impart knowledge to local counties/village communities (Ris) on the efficient use of electricity and better energy management practices to control demand and energy consumption at end-user level.

3.3.6 Sustainability

Sustainability Rating: 3/4 (Moderately Likely - Moderate Risks)

Risk assessments and mitigation strategies/action plans were identified and implemented during project design. However, it did not account for new external environments such as the UN sanctions and the extended banking channel disruption/closure. This resulted in unanticipated sustainability issues (incomplete procurement-related interventions for strengthening energy supply chains) emerging during project implementation and the outcome could not be fully realized/implemented.

The evaluation observed that National Consultants received extensive capacity building and knowledge in RE/EE technology and solutions. This is a commendable effort which should be continued long-term by the relevant DPRK national counterparts to conduct knowledge/operational transfer to have an extended pool of national resources for roll-out of future SES model roll-out.

The SES Project appropriately developed an exit strategy and took into account the following:

Political factors – there is strong support and commitment from the DPRK Government and CPCs to continue.
Financial factors – there is financial stability to operate on its own without further financial support.
Technical factors - skills and expertise needed were suitably assessed and with capacity building activities organized to upgrade the skillsets and competencies of the beneficiaries. However, county stakeholders still require continuous technical support before they could fully function on their own.
Environmental factors – the SES model can be replicated (in close cooperation with national and local counterparts) to other counties/Ris but this needs to be complemented with appropriate and timely procurement-related interventions for strengthening energy supply chains to maximize effectiveness and impact. It is also critically important to:
- observe and pay attention to safety measures and procedures for RE/EE equipment to minimize/prevent occupational accidents and hazards from occurring
- impart knowledge to local counties/village communities (Ris) on the efficient use of electricity and better energy management practices to control demand and energy consumption

3.3.7 Basic Human Needs / Gender Equality

Achievement Rating: 5/6 (Satisfactory – Minor Shortcomings)

The SES PRODOC did not include specific gender mainstreaming/social inclusion strategy. However the SES Project has factored these into its activities. Basic human needs and gender equality were potentially achieved based on anecdotal/proxy indicator evidence through concrete examples of:

Increased attendance of children in kindergartens and nurseries because of improved indoor environment during winter and; female teachers benefiting from cleaner air quality due to improved floor heating system.
Hospitals/clinics able to provide more reliable services to vulnerable groups such as elderly, pregnant women, children, the sick, people with disabilities.

The evaluation further noted that less women participated in training even though the SES Project prioritized/encouraged more women. This is possibly due to the technical roles in the energy and construction sectors predominantly taken up by men in DPRK.

While the reported benefits by project reports could be perceived as immense, the evaluation could not fully verify the actual benefits at ground level. This could be further realized if an impact evaluation study at project output/activity level could be externally conducted by an independent party.

Due to the lack of follow-up in monitoring and evaluating the UNDP-sponsored training courses/workshops’ impact and effectiveness, the evaluation assessed that there was insufficient data available to demonstrate how the capacity development and knowledge dissemination activities of the SES Project improved women’s employment and income generation opportunity as part of contribution to gender equality.

Future projects in DPRK should continue to prioritise gender mainstreaming activities to assess the capacity needs according to gender requirements, and capacity development activities specifically relating to enhancing gender equality and improving the women’s living and livelihood standards.

3.3.8 Synergy

Achievement Rating: 4/6 (Moderately Satisfactory – Moderate Shortcomings)

The evaluation assessed that there were strong synergy effects between the SES Project and CBDRM Project as follow:

strengthening of embankment concept from CBDRM Project was implemented with SES Project activities to enable and strengthen the implementation of RE technologies.
as part of the SES Project, public buildings such as kindergartens and schools were retrofitted with EE measures. Some of these public buildings were also selected as evacuation centers in the CBDRM Project. This would result in a positive impact to the well-being and safety of beneficiaries during emergency situations such as floods and typhoons.
the SES Project implemented EE measures to improve the indoor heating system (Ondol floor heating). This would also increase indoor thermal comfort and also increase the protection of village communities from extreme cold conditions as part of CBDRM Project interventions in disaster risk management.
EE stoves and solar PV panels were installed as part of the SES Project which helped to improve the heat insulation, improve cooking efficiency and maintain the warm indoor environment. This would result in less timber being collected by SLUG groups and used by village communities (Ris) for firewood which would be required for cooking and also for keeping the indoor environment warm during winter season. The lessened use of timber meant that more trees would be preserved on mountain slopes to strengthen prevention of soil erosion and landslides as part of CBDRM Project interventions in disaster risk management.
CBDRM project provided seeds (pinus koreansis, larix leptolepis, and castanata crenata) for improving livelihoods and saplings (aronia melanocarpa, and bamboo willow) to prevent soil erosion using soil bioengineering. Communities have further used firewood species such as poplar to enhance biomass resources availability.

The evaluation would also like to highlight another synergy effect that, in another separate UNDP DPRK SED Project, the SED Project Team learnt key lessons from the SES Project to develop innovative energy solutions such as Solar PV Panels to provide sustainable and reliable energy supply to the Spirulina and Pistia Centres in Unryul and Unchon Counties (South Hwanghae Province) for the UNDP DPRK SED Project.

As previously mentioned, the evaluation assessed that the SES Project needed to improve on its weak synergies with other UN agencies and international organizations with similar project/programme outputs and results, which need to be improved. As previously mentioned in Section 3.3.2, information sharing, communication of project results and valuable lessons learned should be further disseminated and strengthened on application, uses and impacts of RE/EE with other UN agencies and international organizations towards collectively achieving the UNSF Outcomes and also the SDGs.

The evaluation also observed that synergy effect between the SES Project and CBDRM Project have undesirable implications such as:
village communities (Ri) who were not the beneficiaries of both SES and CBDRM Projects would perceive as receiving less “benefits”.
unhealthy comparisons and competitions between the projects village communities (Ris) observed. For example:
- some Ris received such as tree seeds and saplings while other Ris did not receive these items. Proposed “compensation” with more project interventions were not realized.
- selected Ris were able to successfully procure RE assets, while other Ris were not successful in procuring similar RE assets with a lack of a sound explanation from UNDP on why this happened.
  
  The evaluation further noted that the SES Project Team justified its response to the needs on ground based on the project objective whereby:

certain interventions, mainly RE, are based on resource availability.
the SES and CBDRM projects have responded to the needs on ground considering availability of budget, prioritisation at the community level in order to balance its overall support.
procurement plans 2018 and 2019 were not materialised under SES and CBDRM. This is beyond the SES Project Team’s control.

Recommendations
1	Strengthen financial reporting process
2	Extensive review and update of country office policies and procedures with long-term scenario planning
3	Consistently monitor and report of assets/delivered items
4	Manage UNDP CO’s reputational risks and stakeholders’ expectation
5	Rollout/replication of the SES Project in DPRK at county/village community (Ri) level
6	Communication of project results
7	Implementation of safety measures and procedures on RE/EE equipment

1. Recommendation:

Strengthen financial reporting process

Management Response: [Added: 2020/05/22]

Accepted

Key Actions:

Key Action	Responsible	DueDate	Status	Comments	Documents
1.1 For improved financial accountability and transparency purposes, UNDP DPRK project financial reporting processes and templates should track and report progress of consistent financial figures i.e. budget and actual expenditure for consistent comparisons between budget and actual expenditure, as per project outputs, based on project CDRs, for submissions of all relevant project reports (including annual project progress reports), to demonstrate the efficient use of funding on project output-based activities. [Added: 2020/05/22]	Project team	2020/01	No Longer Applicable	[Justification: Project closed]

2. Recommendation:

Extensive review and update of country office policies and procedures with long-term scenario planning

Management Response: [Added: 2020/05/22]

Accepted

Key Actions:

Key Action	Responsible	DueDate	Status	Comments	Documents
2.1 UNDP DPRK CO should work with UNDP BRH to extensively review and update all operational, procurement and financial policies and procedures to account for all that happened within the 2015-2019 period and appropriately mitigate any future constraints. [Added: 2020/05/22]	CO, BRH	2020/12	Overdue-Not Initiated	New program/project yet to be formulated
2.2 UNDP DPRK CO should incorporate extensive long-term scenario planning processes with appropriate and specific risk assessments and counter-measures. [Added: 2020/05/22]	CO	2020/12	Overdue-Not Initiated	New program/project yet to be formulated

3. Recommendation:

Consistently monitor and report of assets/delivered items

Management Response: [Added: 2020/05/22]

Agreed

Key Actions:

Key Action	Responsible	DueDate	Status	Comments
3.1 UNDP DPRK CO should ensure that procurement of any equipment/materials strictly complies to relevant UNDP Policies and Procedures, with the monitoring process/procedure stringently following UNDP DPRK Guidelines for Field Monitoring Visits. [Added: 2020/05/22]	CO, Project team	2020/01	No Longer Applicable	[Justification: Project closed]
3.2 The project team should register any assets/items in the asset/delivered items list and physically monitor them, regardless of how they are procured given the DPRK special context working environment. [Added: 2020/05/22]	CO, Project team	2020/01	No Longer Applicable	[Justification: Project closed]
3.3 The project team continuation of monitoring and reporting on the use of the assets/delivered items after handover to project beneficiaries, at minimum during project implementation, should be adhered to. [Added: 2020/05/22]	CO< project team	2020/01	No Longer Applicable	[Justification: Project closed]

4. Recommendation:

Manage UNDP CO’s reputational risks and stakeholders’ expectation

Management Response: [Added: 2020/05/22]

Agreed

Key Actions:

Key Action	Responsible	DueDate	Status	Comments
4.2 UNDP DPRK CO should strengthen its relationship management processes with project beneficiaries such as continued field visits, as practical and as relevant as required during the project implementation period, to better manage stakeholder expectations. By doing so, the CO would avoid/minimize potential economic and productivity losses to counties/village communities (Ris). [Added: 2020/05/22]	CO, Program and project team	2020/01	No Longer Applicable	[Justification: Project closed]
4.3 UNDP DPRK CO minimizes and/or avoid unequal distribution of delivered assets/items to avoid unhealthy comparisons between project beneficiaries and across any projects that have synergies. [Added: 2020/05/22]	CO, Program and project team	2020/01	No Longer Applicable	[Justification: Project closed]
4.1 UNDP DPRK CO should set conditions and mechanisms to implement “Force Majeure” or early termination of projects if need to. [Added: 2020/05/22]	CO, Program and project team	2020/12	Overdue-Not Initiated	New program/project yet to be formulated

5. Recommendation:

Rollout/replication of the SES Project in DPRK at county/village community (Ri) level

Management Response: [Added: 2020/05/22]

Agreed

Key Actions:

Key Action	Responsible	DueDate	Status	Comments
5.1 Facilitate knowledge/operational transfer of the SES Project’s procedural, operational and hands-on training manuals, guidelines, SOPs, CEMPs and other related SES equipment/materials [Added: 2020/05/22]	Project team	2020/01	No Longer Applicable	[Justification: Project closed]
5.2 Organize study tours, in other countries of similar context and/or culture to DPRK, for increased exposure to acquiring knowledge/application of best practices in RE/EE [Added: 2020/05/22]	Project team	2020/01	No Longer Applicable	[Justification: Project closed]
5.3 Conduct a base-line study to establish the starting indicators of current energy consumption and socio-economic development in local village communities (Ris) [Added: 2020/05/22]	Project team	2020/01	No Longer Applicable	[Justification: Project closed]
5.4 Conduct an independent impact evaluation study, as a future project output/activity component, to measure the impact effectiveness, final end-line indicators and actual benefits gained [Added: 2020/05/22]	Project team	2020/01	No Longer Applicable	[Justification: Project closed]

6. Recommendation:

Communication of project results

Management Response: [Added: 2020/05/22]

Agreed

Key Actions:

Key Action	Responsible	DueDate	Status	Comments	Documents
6.1 Future SES-related projects should strengthen its communication/sharing platforms to engage in closer collaboration/synergies with international organizations/agencies on SES-related activities [Added: 2020/05/22]	Project team	2020/01	No Longer Applicable	[Justification: Project closed]
6.2 Current SES project sites and future SES-related interventions should display stronger on-the-ground visibility of UNDP’s unique contributions at the county/village community (Ri) level through the consistent placing of UNDP logos, nameplates and/or asset/delivered item tags [Added: 2020/05/22]	Project team	2020/01	No Longer Applicable	[Justification: Project closed]

7. Recommendation:

Implementation of safety measures and procedures on RE/EE equipment

Management Response: [Added: 2020/05/22]

Agreed

Key Actions:

Key Action	Responsible	DueDate	Status	Comments
7.1 Installing protective covering over live equipment for insulation from any electrical shocks [Added: 2020/05/22]	Project team	2020/01	No Longer Applicable	[Justification: Project closed]
7.2. Creating risk-free and secured access to any sites housing the RE/EE equipment to minimize/prevent any potential workplace accidents [Added: 2020/05/22]	Project team	2020/01	No Longer Applicable	[Justification: Project closed]
7.3. Developing safety procedures/manuals when operating, cleaning and/or maintaining any RE/EE equipment. [Added: 2020/05/22]	Project team	2020/01	No Longer Applicable	[Justification: Project closed]

End of Project Evaluation: Sustainable Energy Solutions for Rural Livelihoods in DPRK (SES)

Share

5.1 Conclusions and Lessons Learned

3. findings

3.1 Project Design

3.2 Project Implementation

3.3 Achievement of Project Results

Management Response: [Added: 2020/05/22]

Key Actions:

Management Response: [Added: 2020/05/22]

Key Actions:

Management Response: [Added: 2020/05/22]

Key Actions:

Management Response: [Added: 2020/05/22]

Key Actions:

Management Response: [Added: 2020/05/22]

Key Actions:

Management Response: [Added: 2020/05/22]

Key Actions:

Management Response: [Added: 2020/05/22]

Key Actions:

Evaluations by Unit

Latest Evaluations

Evaluation Snap Shot

Contact us