3.1.1 Project Document (PRODOC) Formulation
The SES PRODOC indicated that the earliest commencement of the SES Project formulation was a UNDP fact-finding mission to conduct a rural energy survey in June 2014. A detailed assessment indicated to implement local-level energy solutions in rural areas through an approach that entails:
- the establishment of delivery models enabling the sustainable supply and operation of energy solutions in rural areas
- the introduction of renewable energy technologies (RE), and solutions for more efficient energy use (EE) and energy conservation (EC)
- the increase of county-level energy self-reliance by enhanced ownership and technical and managerial competencies for the sustainable use of local renewable energy resources.
The SES PRODOC developed TORs to recruit suitable project team members to implement and manage the SES Project. The evaluation determined that the SES Project Team (comprising one International Project Manager, one National Technical Coordinator and One National Administrative Assistant) had the project management expertise and suitable technical expertise to deliver the project outputs which are technically complex and required specialised expertise and knowledge in RE/EE.
3.1.2 Analysis of Results and Resources Framework (Project Logic/Strategy and Indicators)
In reviewing the effectiveness and efficiency of the SES Project in meeting its outcome, the evaluation reviewed the SES Project’s Results and Resources Framework in relation to the UNDP DPRK CPD (2011 to 2015) and UNSF (2011 to 2016, 2017 to 2021) on the strategic priorities, outcomes, outputs and the primary applicable key environment and sustainable development key result areas (KRAs). The evaluation assessment also addressed the SES Project’s strategy, indicators, baseline, end of project target, source of verification, and risk and assumptions.
The evaluation reviewed that the SES Project’s Results and Resources Framework design has taken careful consideration of the UNDP DPRK CPD and UNSF outcomes and was aligned to the key environment and sustainable development KRAs. Furthermore, the SES Project’s Results and Resources Framework was prepared with in-depth thinking, accurately described the end of project goals, listed the sources of verification, and appropriately identified the risks and the assumptions.
The Results and Resources Framework was clearly described with the indicative activities and end of project targets. There were 15 indicators in total which reflected against outputs and activities.
The SES Project took extensive consideration to stakeholder participation in project design, decision making, planning, implementation and monitoring. For example, the National Counterparts and Local Counterparts were invited to contribute to designing of project interventions and technical discussions on the output activities. This translated to an increase in confidence and ownership of project activities in the SES Project implementation.
The SES Project’s outcome and outputs were consistent with the DRPK Government’s national priorities. A consultative approach with the National and Local Counterparts was followed in the development and design of project outputs and activities, resulting in strong project ownership and commitment.
The SES Project’s proposed outcome and outputs of the Project individually addressed specific needs identified and collectively presented a comprehensive solution to strengthen local capacity for improved nutrition and food security.
The SES Project also aligned with local county development plans and reinforced stakeholders’ engagement and supported their achievement of priorities. The SES Project design was also strategically aligned and consistent with the UN MDGs and subsequent UN SDGs.
The evaluation further noted that the SES Project’s expected results in the SES PRODOC are more output-oriented (WHAT IS BEING PRODUCED - EFFICIENCY) than outcome-oriented (WHAT IS THE VALUE/BENEFIT/ CHANGE/IMPACT - EFFECTIVENESS). While this is not an assessment of the SES Project Team’s performance, the evaluation is of a view that future PRODOC design should consider a balance of expected results with outcome-oriented targets/indicators to determine the effectiveness.
3.1.3 Risks and Assumptions
The SES PRODOC had appropriate risk assessments with impact and probability ratings, and prepared corresponding counter-measures/management responses which were appropriate at that point of time and for the project duration (2015 to 2019). The SES Project identified a total of 15 risks:
- 1 governance risk
- 3 operational risks
- 5 strategic risks
- 3 financial/fiduciary risks
- 3 sustainability risks
However, the evaluation reviewed that the risk assessments could be further extended to be part of the Results and Resources Framework to identify the key risks and appropriate counter-measures/management response for each of the 4 SES Project outputs. Many of these activities would have governance, operational risks, strategic risks, financial/fiduciary and/or sustainability risks that would require appropriate counter-measures/management responses.
The evaluation also determined that the SES PRODOC’s risk analysis did not account for scenarios of extreme UN sanction measures and the extended banking channel disruption/closure. Furthermore, the implementation of the SES PRODOC’s counter-measures/management responses did not appropriately resolve the significant change of events caused by the UN Sanction measures and the extended banking channel disruption/closure over the project duration.
3.1.4 Lessons from Other Relevant Projects Incorporated into Project Design
The evaluation observed that the SES Project Team took opportunity to align the SES Project with the CBDRM Project to maximize the synergy effects (more details found in Section 3.3.8).
The SES Project was also built from the experience and lessons learned from two previous UNDP DPRK projects:
- Small Wind Energy Development Programme Rural Areas (SWEDPRA) Project
- Sustainable Rural Energy Development (SRED) Project
For example:
- the SES Project focused on the attainment of effective and sustainable local energy solutions that generate positive impact among rural beneficiaries, rather than involving in technology development.
- the SES Project focused on implementing RE/EE technologies and solutions at community-level (such as public institutions , buildings and facilities) rather than household level (individual homes). This would be through area-based clustering or crowding-in approach to maximize demonstration effects and awareness raising.
3.1.5 Planned Stakeholder Participation
The SES Project generated strong stakeholder interest, especially at the DPRK national/central government ministries and Local Counterparts such as CPCs and NTDCs in Kaechon City and Hoechang, Singye, Unsan, Yonsan, and Yangdok Counties.
In terms of project design, the proxy indicators would be the number of stakeholders involved in planning and attendance during the project formulation/planning meetings. The evaluation interviews with National and Local Counterparts indicated sufficient evidence of direct involvement based on detailed accounts of the project outputs.
The minutes of the PSC meetings recorded perfect attendance and representations from the National Counterparts. The proxy indicators from M&E Field Monitoring Visits for participation at the project implementation stage indicated high project output ownership, perfect attendance at project field site meetings, capacity development/knowledge dissemination activities, and the visible evidence of construction/installation taking place. During the evaluation interviews, there were high levels of project output-ownership as the Local Counterparts and beneficiaries were able to provide extensive technical details of their project outputs.
3.1.6 Replication Approach
Replication and up-scaling are fundamental to the SES Project as it provides the opportunity to build on best practices and lessons learned, and expand the reach and impact of its project outputs. As such, UNDP, government agencies, international agencies/organizations and the private sector would utilize these given opportunities to support the replication and up-scaling of the most successful projects and practices through their networks and contacts.
The SES Project has the potential for replication in other provinces/counties in DPRK through:
- Methodologies and approaches for energy resource assessments, energy demand assessments, energy audits and feasibility studies including cost-benefit analysis
- Development and implementation of County Energy Management Plans (CEMPs)
- Dissemination, promotion and demonstration on the uses/applications of RE/EE solutions at village community (Ri) level through NTDCs
- Roll-out implementation of suitable RE/EE interventions at public buildings, institutions and facilities
- Establishment of institutional frameworks and energy supply chains at county level to continue the sustainability of RE/EE equipment and interventions.
3.1.7 Management Arrangements
Execution Modality: In accordance with the SES PRODOC, the SES Project modality was Direct Implementation Modality (DIM) which meant the project execution and implementation would be undertaken directly by UNDP DPRK in accordance with UNDP Programme and Operations Policies and Procedures (POPP). The overall decision, including financial accountability would rest with UNDP DPRK and the SES Project was to be executed in coordination with relevant partners, including at the local county level, with a view to ensuring that effective assistance flowed directly to targeted beneficiaries.
Project Steering Committee (PSC): The PSC was established to provide high-level oversight and to steer the SES Project. The PSC is responsible for high-level management decisions and policy guidance required for implementation of the project, including recommendations and approval of project plans, budget and revision. The PSC membership comprised the following key stakeholders:
The SES Project Team travelled to the respective county locations to hold regular and quarterly meetings with the project partners to review the project progress and initiate early corrective actions.
The SES Project FMV reports indicated effective discussions to resolve project management and coordination issues, and also contained details of reviews and actions taken. The Programme FMV, led by the M&E Specialist and CO Management, validated the results achieved. All recommended actions were consistently followed up and presented by the M&E Specialist at PSC meetings and captured in the quarterly programme and oversight FMV reports. These reports were subsequently sent to the UNDP Regional HQ Bureau as required by the UNDP DPRK ICF. The evaluation reviewed that there was a focus on results and activity scheduling across activities and outputs. Progress was reviewed against the objectives and targets set in the SES PRODOC’s Results and Resources Framework. The Project and Programme FMV reports were written to reflect the progress achieved against targets.
Project Management Unit (PMU): Being a DIM agency, the UNDP formed a PMU comprising one International Project Manager, one National Technical Coordinator and one National Administrative Assistant.
The PMU would be fully responsible for the coordination of National/Local Counterparts for project execution in a timely manner and within budget. The PMU facilitated effective project planning, that included preparation of annual work plans and project monitoring and reporting. The PMU was charged with coordinating and facilitating the procurements. As a curator, the evaluation reviewed that the PMU had effectively and efficiently held all the records, publications and minutes of meetings pertaining to the SES Project.
3.2.1 Adaptive Management
The SES Project was formally signed off on 26 August 2015. However, there were prolonged delays at the start of the project due to the:
- extended period of banking channel disruption/closure for funds transfer to the UNDP DPRK CO
Due to the early UN Sanctions on DPRK (UN Resolutions #2087 and #2094), the UNDP DPRK CO had to implement prolonged periods of organizational cash conservation mode due to the lack of funds being transferred into DPRK. Hence, there were minimal funds to implement any project activities and eventually slow progress in delivering project results.
- Late recruitment of the SES Project Team
The extended period of banking channel disruption/closure created uncertainties for the UNDP DPRK CO and possibly resulted in the late recruitment of the SES Project Team. The Project Manager, National Technical Coordinator and Project Administrative Assistant were eventually on board in the 1st quarter of 2016.
Despite the early and recurring setbacks, the evaluation reviewed that the SES Project Team displayed good project management abilities and effectively utilised appropriate project management tools to implement the SES Project to the best of their abilities.
The project implementation was delayed by 7 months from August 2015 until March 2016, with the first PSC Meeting involving the SES Project Team on board held on 21 April 2016. The SES Project Team effectively applied adaptive management in planning by having to reschedule the timelines for activities in order to accomplish the project outputs, with activities starting in 2016.
The UN Security Council imposed two UN Sanctions (UN Resolutions #2270 and #2321) in 2016 and another four UN Sanctions (UN Resolutions #2356, #2371, #2375 and #2397) in 2017 were imposed on DPRK which included (among many measures) import, financial and economic restrictions.
Table 1 below showed the implementation status of each SES Project output as assessed by the evaluation. The evaluation noted that the SES Project would have produced a significantly different implementation status if there were no UN Sanctions imposed on DPRK and there was no banking channel disruption/closure issue to deal with.
Table 1: SES Project Implementation Status
|
SES PRODOC
|
Implementation Status1
|
|
Outcome
Increased standards of living and sustainable livelihood
|
|
Output 1
Information about energy resources and feasible RE/EE solutions updated and made accessible to local beneficiaries.
|
Fully Achieved
|
|
Output 2
Increased technical know-how of county-level personnel for energy planning and sustainable management of local renewable energy resources.
|
Almost Fully Achieved
|
|
Output 3
Strengthened supply chains for the delivery of appropriate RE/EE solutions for local communities in rural areas.
|
Partially Achieved
|
|
Output 4
Increased energy security and self-reliance of rural population through the implementation of RE/EE solutions for local communities.
|
Partially Achieved
|
Note:
- The implementation status is purely based on the desired results of the SES PRODOC. It has not been moderated based on the implications and resultant consequences attributed to the 6 UN Sanctions imposed on DPRK in 2016 and 2017, and the extended period of banking channel disruption/closure which severely disrupted funds being transferred into DPRK to implement project activities.
In the case of Outputs 3 and 4, the evaluation reviewed that these implications and resultant consequences were beyond the control of the SES Project Team and the UNDP DPRK CO, and there were minimal or no alternative adaptive management measures that could have produced a better outcome.
The evaluation assessed and made the following observations relating to safety and durability:
- Communications – enhanced visibility and communication of project results
Even though facing significant setbacks on procurement-related activities to deliver RE/EE equipment and materials, the SES Project did make significant impacts at village community (Ri) level such as the successful delivery and implementation of:
- 200 sets of solar PV systems in 170 public institutions across 15 Ris (from project reports):
- 29 sets of solar PV systems in 26 productive facilities and implemented energy efficiency interventions in 11 productive processes.
- 171 sets of solar PV systems in 144 social service institutions.
- energy efficiency interventions in 67 public institutions (improved Ondol floor heating system, thermal insulation in buildings including double glazed windows and doors, retrofitting with foamed cement bricks on building envelop, EE coal stoves, EE biomass stoves, and solar PV systems) and 62 community social service providers across 15 Ris such as kindergartens, nurseries, schools, hospitals and clinics.
However, during the field visit to project sites, the evaluation noted many items were confirmed as funded by the UNDP SES Project but missing UNDP-registered name tags, name plates and/or logos that would enable local counties and village communities to recognize and acknowledge UNDP’s contributions to improving livelihoods. The evaluation determines that UNDP’s unique contributions should be given more on-the-ground recognition through stronger visibility and communication of project results at project sites such as UNDP logos, nameplates, asset/delivered item tags.
The evaluation also observed that a number of items procured/provided under the SES Project were not updated accordingly into the asset/delivered items list. This could potentially lead to be externally viewed/perceived as not physically accepted at time of delivery by the SES Project Team and/or not physically verified by the M&E Specialist, as required in the UNDP DPRK Guidelines for Field Monitoring Visits.
- Communications – Management of expectations and reputational risk
There is a need to manage village community (Ri) expectations on (1) UNDP’s “inconsistent” delivery of items (such as RE/EE equipment) to different Ris to minimize the occurrence of unhealthy comparisons and unhealthy competitions between project Ris, and (2) prolonged delays in UNDP interventions to minimize/avoid potential economic and productivity losses to Counties/Ris.
UNDP DPRK has gained a reputation among national and local counterparts as an organization that failed to deliver on its promises. Restoring UNDP’s reputation as an organization that can effectively deliver would be a key priority. The evaluation would therefore find it beneficial for UNDP DPRK by:
- prescribing conditions and mechanisms to implement “Force Majeure” or early termination of projects if need to; and
- continuing field visits, as practical and as relevant as required during the project implementation period, to maintain relationships and communications with village communities (Ris).
3.2.2 Partnership Arrangements
The SES Project generated strong stakeholder interest and participation from National/Local Counterparts in DPRK. The stakeholders at the National/Central level were:
There was evidence of strong interest and commitment at the local county level through stakeholder contributions (both financial and in-kind), roles and responsibilities to implement the SES Project activities.
Despite the external factors/challenges that were beyond the control of the UNDP DPRK CO, the partnership arrangement between SES Project Team and the National and Local Counterparts endured the challenging 4.5-year project period, and demonstrated great patience, understanding and resilience to overcome the difficulties faced. The fruits of this partnership agreement in challenging circumstances were the successful completion of many SES Project interventions as follow:
- Energy resource assessments and energy demand assessments completed for 15 village communities (Ri).
- 24 feasibility studies integrating cost-benefit assessment methodology for RE/EE interventions completed.
- 20 energy audits for basic-necessity facilities, agricultural processes and public community buildings completed.
- Dissemination, promotion and demonstration on the uses/applications of RE/EE solutions at village community (Ri) level through NTDCs.
- Successfully delivered and installed 200 sets of solar PV systems in 170 public institutions across 15 Ris (from project reports):
- 29 sets of solar PV systems in 26 productive facilities and implemented energy efficiency interventions in 11 productive processes.
- 171 sets of solar PV systems in 144 social service institutions.
- Successfully implemented energy efficiency interventions in 67 public institutions (improved Ondol floor heating system, RE hybrid system units, thermal insulation in buildings including double glazed windows and doors, retrofitting with foamed cement bricks on building envelop, EE coal stoves, EE biomass stoves, and solar PV systems) and 62 community social service providers across 15 Ris such as kindergartens, nurseries, schools, hospitals and clinics.
3.2.3 Project Finance
The SES Project had a duration of about 4.5 years (August 2015 – December 2019) with an approved funding of US$6,117,572. The details of the planned financing allocation based on the SES PRODOC are as follow:
Table 2: SES Project – Original Planned Budget as per SES PRODOC
|
SES Project
|
2015 and 2016
(US$)
|
2017
(US$)
|
2018
(US$)
|
2019
(US$)
|
|
Output 1
|
543,500
|
354,500
|
162,500
|
122,500
|
|
Output 2
|
141,750
|
539,250
|
199,250
|
156,750
|
|
Output 3
|
251,750
|
468,750
|
292,750
|
207,750
|
|
Output 4
|
142,750
|
1,168,322
|
827,750
|
537,750
|
|
Total
|
1,079,750
|
2,530,822
|
1,482,250
|
1,024,750
|
While the SES PRODOC did not include any co-financing from National/Local Counterparts, the evaluation reviewed that the Local Counterparts provided in-kind contributions (labor and construction materials) to assist the timely completion of SES Project activities.
The budget and actual expenditure of the SES Project is provided below in Table 3.
Table 3: Summary of Budget and Actual Expenditure (SES Project)
|
SES Project
|
2015 and 2016
(US$)
|
2017
(US$)
|
2018
(US$)
|
2019
(US$)
|
|
Operational Expenses (Actual)
|
1,573.41
|
323,321.58
|
4,878.37
|
4120.92
|
|
Output 1 (Actual)
|
190,077.29
|
480,374.36
|
174,522.36
|
82,493.57
|
|
Output 2 (Actual)
|
33,795.07
|
313,228.68
|
244,935.66
|
105,478.13
|
|
Output 3 Actual)
|
34,144.28
|
79,916.14
|
75,458.84
|
464,993.32
|
|
Output 4 (Actual)
|
68,803.31
|
1,157,608.30
|
166,697.68
|
162,968.91
|
|
Total (Actual)1
|
328,233.98
|
2,354,449.06
|
666,492.91
|
820,054.85
|
|
Utilization Rate
(Actual/PRODOC Budget)
|
30%
|
93%
|
45%
|
80%
|
Note:
- Actual figures are based on financial system extracts provided by the UNDP DPRK CO
The evaluation noted that the SES Project under-spent its allocated total project funds by about 32% and its utilization with an average of 62%. This was due to the banking channel disruption/closure, caused by the UN Sanctions, which disrupted funds from being transferred into DPRK. This further resulted in the SES Project’s inability to obtain funds to implement the SES Project procurement-related activities.
In considering the UN sanction measures together with recurring and extended banking channel disruption/closure which led to the UNDP DPRK CO activating the cash conservation mode to sustain the office operations, the SES Project Team displayed sound financial management processes to implement the relevant SES Project activities which were not affected by the UN sanction measures.
However, the evaluation reviewed that there were inconsistencies (and inconsistent templates) in the SES Project Team’s financial reporting processes due to different reporting requirements given.
- Different reporting of SES Project budget figures
The budget figures in the SES Project Annual Work Plans were different from that of the SES Project Annual Progress Reports. This was due to the different submission timelines required by different reports which led to different budget figures being reported, e.g. due to budget revisions made during the year.
- Inconsistent reporting of SES Project actual expenditure figures
The actual expenditure provided to the evaluation was based on actual expenditure according to project outputs. However, the actual expenditure in the SES Project Annual Progress Reports were based on actual expenditure, as per financial reporting templates being provided by UNDP DPRK CO, according to the categories of Project Activity, Management and Staff, General Operations Expenditure, and/or Common Services.
3. Inconsistent reporting on comparison of SES Project budget versus actual expenditure figures
The SES Project Team did not provide budget and actual expenditure figures in PSC meetings. However, the SES Project Annual Progress Reports report these comparisons for the calendar year period but not at output levels as the SES Project Team followed the financial reporting templates being provided by UNDP DPRK CO. The evaluation further noted that only the SES Project’s CDR run was attached at the time of the report submission.
For improved financial accountability and transparency purposes as part of demonstrating the efficient use of funding on project output-based activities, future financial reporting processes and templates of UNDP DPRK projects should:
- track and report consistent financial figures (budget and actual expenditure)
- have consistent comparisons between budget and actual expenditure, as per project outputs, based on project CDRs, for submissions of all relevant project reports (including annual progress reports and submissions to PSC meetings)
3.2.4 Monitoring and Evaluation (M&E) Activities Used for Adaptive Management
The M&E framework consisted of local monitoring and reporting as well as international independent evaluations. Both the SES Project Manager and the UNDP DPRK Programme M&E Specialist were responsible for the preparation and submission of the M&E reports and evaluations at project and programme levels respectively, as stated in the SES PRODOC. Table 4 below summarizes the achievement of monitoring actions as required by the SES PRODOC.
Table 4: M&E Plan and Completion Status
|
Type of
M&E Activity/Report
|
Frequency/ Timing
|
Status
|
Comments
|
|
Detailed Quarterly Workplan
|
Every beginning of the quarter
|
Completed
|
Detailed workplans for 2016, 2017, 2018 and 2019 completed
|
|
Annual Workplan and Budget
|
Beginning of each year
|
Completed
|
Detailed workplans with budget for 2016, 2017, 2018 and 2019 completed
|
|
Quarterly Progress Report
|
Quarterly
|
Completed
|
Reports completed every quarter in 2016, 2017, 2018 and 2019
|
|
Annual Progress Report
|
End of year
|
Completed
|
Reports completed in 2016, 2017 and 2018. The Annual Progress Report for 2019 is to be completed by the SES Project Manager in December 2019
|
|
Mid-Term Review
|
End of Year 2
|
Completed
|
This M&E activity was delayed with one MTR report completed by an independent evaluator in August 2018
|
|
Terminal Report
|
End of Project (end of Year 4)
|
In progress
|
One Terminal Report to be completed by the SES Project Manager in December 2019
|
|
Terminal Evaluation and Audit
|
End of Project (end of Year 4)
|
In progress
|
One Terminal Evaluation report to be completed by an independent evaluator in January 2020
|
|
Mission reports
|
After each mission
|
Completed
|
Mission reports by individual experts (International and National) completed
|
|
Other Reports and Deliverables
|
After each TA or sub-contract
|
Completed
|
Reports and deliverable by individual experts (International and National) completed
|
|
Monitoring Reports
|
After every field visits
|
Completed
|
Field Monitoring reports by SES Project Team and UNDP DPRK Programme M&E Team completed
|
|
Financial records & reporting
|
Continuous
|
Completed
|
Financial records and reporting completed
|
The UNDP DPRK CO and the SES Project Team proactively responded with specific adaptive management measures to recommendations from MTR as shown below in Table 5:
Table 5: Management Response to SES Project MTR Recommendations
|
SES Project MTR Recommendation
|
Management Response
|
|
SES project is experiencing significant delays in procurement of RE and EE systems that need to be piloted. Therefore, the project shall develop a contingency plan to refocus on wider implementation of EE installations and consolidating knowledge gains among engineers and decision makers at the provincial and county level in order to prepare for (eventual) scale-up of the initiatives by the government.
|
- Conduct in-country study tours (bringing together national partners) including NTDCs to discuss and share lessons and areas for intervention
- National consultants select most qualified experts at the NTDCs, Counties and Ris with the support of SES project team.
- Such visits could be expanded beyond the project and to the places such as Jangchon Co-operative farm, Pyongyang City and Natural Energy Institute under the State Academy of Science.
- Produce a report following in-country study tours and commitment received from its participants in scaling up of the EE interventions.
- Establish a regular communication with communities in the implementation of EE technologies with the ownership of MEPI by sustaining momentum in activity implementation.
|
|
Project team works closely with county-level stakeholders (i.e. managers of public buildings and national consultants) to improve their data collection on how many people (disaggregated by sex, age, disability) access services, and the impacts that the RE and EE improvements have had on particular humanitarian outcomes, particularly health.
|
- Firm up of the methodology of data collection from local stakeholders
- Rely on a good expertise of national consultants hired under SES
- Train them on the data collection on how many people (disaggregated by sex, age, disability) access public services.
|
|
It is critical that the project team monitor results of capacity building at the output level, beyond demonstrating the successful implementation of capacity building activities. Below proposed output indicators, and targets, which aim to allow for the collection of data which can be used to analyze the meaningful change in capacity and quality effected by the project to date. These suggestions aim to identify possible entry points for the project, conscious of data access limitations.
|
- In consultation with MEPI following indicators will be monitored by SES project. These will be reported in the Quarterly Progress Reports:
- Extent to which County Energy Management Plans receive budgetary support for implementation from the county governments (Target: County Energy Management Plans receive at least 75% of necessary funding)
- Extent to which NTDCs are operational (Target: plans and budgets for 3 NTDCs approved)
- % change in improved attendance of 5-7-year-old children in target kindergartens between November-March (Target: at least a 50% increase)
|
|
It is recommended that instead of having joint reports following field visits, whether it is for implementation and/or monitoring purposes, team members should submit individual BTORs separately for project and programme.
|
- Projects and programme team will submit separate BTORs upon field missions.
- Quarterly Progress Reports (QPRs) should be completed by the project team (lead by the Project Manager), with quality assurance of the data and analysis undertaken.
- Report qualitative changes of the projects in ROAR through captured qualitative results from projects.
|
|
With the inclusion of more qualitative indicators at the output level, it is hoped that more meaningful analysis of the humanitarian importance of the project will be captured, bearing in mind the sensitivities in sharing project results publicly due to the complex geopolitical context under which UNDP operates in DPRK. Following are some of the key actions that will be taken to improve the reporting of qualitative changes that the project is leading on the ground.
|
- SES project to share communication material (videos/brochures…) with relevant parties including Bangkok Regional Headquarters
|
|
As the SES project will end by December 2019 according to the ProDoc, it’s the right time to start deploying its exit strategy to meaningfully consolidate the results achieved till date and ensure the sustainability of the project activities and results.
|
- SES project shall organize a National Partners Meet to assess what was done better and what else need to be completed in fulfilling the needs of the communities.
- At this meet, SES project must encourage the local communities to make in-kind contributions in the absence of procurement activities.
- Consolidating SES project results till date
|
The evaluation reviewed that the M&E process at the project and programme level was very comprehensive. The UNDP DPRK Programme M&E Team showed high competency in:
- conducting field monitoring visits every quarterly to assess the progress of the SES Project outputs. This included the verification of delivered items and assets through the identification of UNDP item/asset identity tags at the field sites, the onsite testing of equipment delivered by UNDP, and monitoring the use of the delivered items and assets to ensure sustainable operations.
- producing high quality quarterly and annual Programme monitoring and oversight reports, as required by the UNDP DPRK ICF and UNDP DPRK CO Guidelines for Field Monitoring Visits, with key findings and analysis of progress towards results, project performance and implementation issues.
- providing key recommendations and corrective actions/measures to further improving the SES Project, and monitoring the implementation of these key recommendations and corrective actions/measures until completion.
- updating the M&E progresses at all PSC meetings.
The SES Project Team showed high competency in:
- conducting project field monitoring visits every quarterly to assess the progress of the SES Project outputs. This included the onsite testing of equipment delivered by UNDP.
- producing high quality quarterly and annual project progress reports and presenting them at all PSC meetings.
- identifying key issues faced, and providing key recommendations and corrective actions/measures to address these key issues.
- updating project implementation monitoring progress at all PSC meetings.
However, the M&E process at the project level could be further improved in 2 key areas:
- Verification of delivered items and assets through the identification of UNDP item/asset identity tags at the field sites
In numerous occasions during the field mission trips, the project beneficiaries would show the UNDP items/assets and would also compliment the high-quality conditions of UNDP items/assets as compared to other similar items/assets delivered onsite by other organizations.
However, the Evaluator was unable to establish or fully verify whether the sighted items/assets were actually from UNDP due to a lack of identification either with an UNDP logo or a UNDP identification tag. The Evaluator would frequently rely on the M&E Specialist, Programme Analyst and DPRK Government focal point onsite for confirmation and verification.
Tagging of assets/delivered items should be more consistent to clearly distinguish UNDP’s quality items/assets, while also adhering to UNDP DPRK Guidelines for Field Monitoring Visits to ensure complete and proper physical verification and handover for the intended use/purpose.
- Field data collection to measure effectiveness and impact on completed project activities
While the SES Project has consistently reported the impact through reduced coal usage and electricity consumption, there is a need for the SES Project Team to collect data to measure the effectiveness and impact on the village community beneficiaries.
For example, the installed 200 sets of solar PV systems in 170 public institutions across 15 Ris, and energy efficiency interventions in 67 public institutions and 62 community social service providers across 15 Ris such as kindergartens, nurseries, schools, hospitals and clinics, should be continuously monitored with relevant data collected to determine its positive impacts and actual benefits gained on individual, family and community well-being among the village communities.
Enabling the field data collection to measure effectiveness and impact on village communities would further strengthen the:
- overall sustainability results of the SES Project pilot activities
- case for future replication of the SES model in other counties/village communities (Ris) in DPRK
3.2.5 Implementing Agency
The SES Project adopted the direct implementation modality (DIM) which meant that UNDP DPRK would be the Implementing Agency with a dedicated project team based in the UNDP DPRK CO. An International Project Manager would be recruited and be responsible for the daily management of the project with assistance from recruited national project staff (comprising one National Technical Coordinator and one National Administrative Assistant). The SES Project Team would further engage International and/or National Consultants as required based on the SES Project’s technical requirements.
The SES Project also formed a Project Steering Committee (PSC) to guide the project direction and address any challenges. The PSC was co-chaired by the UNDP Deputy Resident Representative (DRR) and the National Coordinator from the DPRK National Coordinating Committee (NCC) for UNDP, with participation of representatives from the MEPI, CBS, SCoST, SAOS and other institutions as needed at the central level.
The SES Project would also work closely with Local Counterparts such as CPCs and NTDCs from:
- Hoechang County, South Pyongan Province
- Singye County, North Hwanghae Province
- Yonsan County, North Hwanghae Province
- Unsan County, North Pyongan Province
- Kaechon City, South Pyongan Province
- Yangdok County, South Pyongan Province
The evaluation established that there were strong working relationships between the UNDP DPRK CO, the SES Project Team and National/Local Counterparts and project beneficiaries at the county/village community (Ri) level. These working relationships were frequently tested by the slow progress of the SES Project. Key representatives of the National/Local Counterparts expressed disappointments at the prolonged delays and unsuccessful implementation of the SES Project procurement-related activities.
Many of these expressed disappointments were understandably justified as, in their views, tangible results were not delivered, especially the procurement activities for RE/EE equipment during the 2nd half of the 4.5-year project duration. Despite these procurement setbacks, the National/Local Counterparts expressed deep gratitude and appreciation on the limited but successful implementation of the SES Project interventions that has a great potential for scale up and replication.
The National/Local Counterparts expressed deep gratitude and appreciation for the SES Project Team who had done their very best, in the midst of many external factors/challenges faced, to implement the project with some significant success.
The National/Local Counterparts, while fully understanding that the external factors/challenges such as the UN Sanctions and the geo-political situation had severely affected the SES Project, highlighted their disappointment in the UNDP as an organization for not being able to deliver the results.
The evaluation rated the SES Project’s project results according to the evaluation ratings table listed below in Table 6.
Table 6: Evaluation Overall Results/Impact Rating
|
Evaluation Ratings for Overall Results/Impact, Relevance, Effectiveness, Efficiency, Basic Human Needs, Gender Equality, National Ownership
|
Sustainability Ratings:
|
|
6. Highly Satisfactory (HS): no shortcomings
5. Satisfactory (S): minor shortcomings
4. Moderately Satisfactory (MS): moderate shortcomings
3. Moderately Unsatisfactory (MU): significant shortcomings
2. Unsatisfactory (U): major shortcomings
1. Highly Unsatisfactory (HU): severe shortcomings
|
4. Likely (L): negligible risks to sustainability
3. Moderately Likely (ML): moderate risks
2. Moderately Unlikely (MU): significant risks
1. Unlikely (U): severe risks
|
|
Additional ratings where relevant:
Not Applicable (N/A)
Unable to Assess (U/A)
|
3.3.1 Overall Results/Impact
The evaluation rated the SES Project’s overall results/impact with reference to its 4 project outputs as per stated in the SES PRODOC. The overall results/impact are presented below in Table 7.
Table 7: Overall Results/Impact – SES Project
|
SES PRODOC
|
Achievement Rating
|
Comments
|
|
Outcome
Increased standards of living and sustainable livelihood
|
|
Output 1
Information about energy resources and feasible RE/EE solutions updated and made accessible to local beneficiaries.
|
6/6
(Highly Satisfactory)
No shortcomings
|
- Energy resource assessments and energy demand assessments completed for 15 village communities (Ri).
- 24 feasibility studies integrating cost-benefit assessment methodology for RE/EE interventions completed.
- 20 energy audits for basic-necessity facilities, agricultural processes and public community buildings completed.
- Dissemination, promotion and demonstration on the uses/applications of RE/EE solutions at village community (Ri) level through NTDCs.
- Technical study tours in Serbia and China on RE/EE solutions conducted.
|
|
Output 2
Increased technical know-how of county-level personnel for energy planning and sustainable management of local renewable energy resources.
|
5/6 (Satisfactory)
Minor shortcomings
|
- Comprehensive training workshops conducted by international and national consultants for decision makers, energy experts and technicians at County and community-level on energy resources, conservation, conversion, efficiency, and planning
- Technical study tours in Serbia and China enabled participants in application of knowledge.
- Training manuals on RE/EE solutions for improvement of rural livelihoods in DPRK developed.
- Promotional and educational activities were conducted to increase awareness on the use of RE/EE solutions at village community (Ri) level.
- 6 counties developed and refined County Energy Management Plans with technical assistance from the National Consultants. However counties still require further technical support before they could fully “run” on their own.
- There is a need to replicate the knowledge/operational capabilities and capacities of National Consultants to enhance the pool of national and local resources
- Based on project reports, about 450 ha of biomass resources were sustainably managed by Ris through in-kind afforestation and reforestation of fast growing firewood trees such as Acacia, Poplar, bamboo willow (this would need to be independently verified).
|
|
Output 3
Strengthened supply chains for the delivery of appropriate RE/EE solutions for local communities in rural areas.
|
4/6 (Moderately Satisfactory)
Moderate shortcomings
|
- Capacities of community and county-level workshops involved in manufacturing facilities and/or service centres for RE/EE products such as foamed cement bricks, EE stoves, double glazed windows, insulated doors, etc. improved.
- Challenges in procurement due to UN Sanctions and banking channel disruption/closure to procure in-country (beyond the control of the UNDP DPRK Project Team and CO) could not fully deliver the all necessary equipment and tools for efficient manufacturing and/or assembly of RE/EE technologies. However local abilities could not be fully realized/achieved to their full potential.
- County NTDCs functioning as expert centres, with roles and responsibilities, organization and management of these centres finalized for technical support, quality assurance, and performance monitoring of RE and EE applications identified. However, independent post impact evaluation study required to verify how well these expert centres are functioning and its impact on the local communities.
|
|
Output 4
Increased energy security and self-reliance of rural population through the implementation of RE/EE solutions for local communities.
|
6/6
(Highly Satisfactory)
No shortcomings
|
Renewable energy solutions identified and implemented through energy resource/demand assessments (the evaluation noted that this was a crucial/important strategic decision taken by the SES Project)
- Successfully delivered and installed 200 sets of solar PV systems in 170 public institutions across 15 Ris (from project reports):
- 29 sets of solar PV systems in 26 productive facilities and implemented energy efficiency. interventions in 11 productive processes
- 171 sets of solar PV systems in 144 social service institutions.
- Successfully implemented energy efficiency interventions in 67 public institutions (improved Ondol floor heating system, thermal insulation in buildings including double glazed windows and doors, retrofitting with foamed cement bricks on building envelop, EE coal stoves, EE biomass stoves, and solar PV systems) in 62 community social service providers across 15 Ris such as kindergartens, nurseries, schools, hospitals and clinics.
|
|
2/6 (Unsatisfactory)
Major shortcomings
|
- Renewable energy solutions identified through 24 feasibility studies in targeted user groups in the pilot Counties were substantively not implemented
- Challenges in procurement due to UN Sanctions and banking channel disruption/closure to procure in-country. (beyond the control of the UNDP DPRK Project Team and CO) could not fully deliver 21 out of the 24 feasibility studies.
|
The evaluation further noted that the SES Project Team had done its best to deliver and achieve the desired project results despite encountering significant external factors/challenges, mainly due to the 6 UN Sanctions in 2016 and 2017 and the recurring banking channel disruption/closure that prevented funds transfer into DPRK during the SES Project implementation.
3.3.2 Relevance
Relevance with national priorities
Achievement Rating: 5/6 (Satisfactory - Minor Shortcomings)
The SES Project was highly relevant and aligned with the DPRK national strategies and priorities. The SES Project was designed with humanitarian-oriented outputs and activities which were aimed to address the humanitarian needs of intended beneficiaries.
The SES Project’s relevance was further strengthened with National and Local Counterparts being involved and consulted during the project design and also during project implementation. The SES Project Team, particularly the Project Manager, also had suitable technical skillsets and competencies to deliver most of the project outputs which are technically complex and required specialised expertise and knowledge in RE and EE.
The SES Project’s relevance could be further improved if challenges in procurement due to UN Sanctions and banking channel disruption/closure severely disrupted the ability to procure internationally and in-country were appropriately resolved (which is beyond the control of the UNDP DPRK Project Team and CO) in response to the geo-political environment. Hence the SES Project’s relevance was affected as it could not fully deliver the required procurement-related activities to strengthen the energy supply chains (RE/EE tools/equipment/materials, civil works and construction).
Relevance with UNSF Outcomes and SDGs
Achievement Rating: 3/6 (Moderately Unsatisfactory - Significant Shortcomings)
The SES Project aligns closely with UNSF Outcomes 2.2, 3.2 and 4.3, MDGs 3, 4 and 7. The SES Project also contributes to the SDGs (SDG 7 on affordable and clean energy, and SDG 13 on climate action).
The UNSF (2017-2021) emphasized coherent and coordinated implementation in support of a common objective in order to achieve potential synergies among UN agencies and possibly with international organizations.
The evaluation assessed that the SES Project needed to improve its weak synergies with other UN agencies and international organizations with similar project/programme outputs and results. In particular, information sharing, communication of project results and valuable lessons learned should be further disseminated and strengthened on application, uses and impacts of RE/EE with other UN agencies and international organizations towards collectively achieving the UNSF Outcomes and also the SDGs.
3.3.3 Effectiveness
Achievement Rating: 5/6 (Satisfactory – Minor Shortcomings)
External factors/environment such as the extended banking channel disruption/closure and significant delays through the sanctions exemptions/clearance process hindered the SES Project’s implementation and severely affected UNDP’s reputation as an organization of not being able to effectively deliver (beyond the control of the UNDP project team and CO). This in-turn affected the desired Output 3 and 4 results to be fully achieved, particularly the procurement of RE/EE equipment and materials to strengthen energy supply chains.
There were significant results for the village community (Ri) beneficiaries which potentially contributed to the SDGs such as:
- Hospitals/Clinics – Hospitals/clinics have reliable electricity and thermal insulation to treat patients in a conducive temperature-controlled environment and able to all year round (24 hours a day, including at night if required); Unsan County’s women hospital was able to operate all year round to further enhance the treatment of women’s well-being and health (including reproductive health disorders)
- 10-Day kindergartens and nurseries – children and female teacher/staff well-being and health were potentially enhanced with increased learning opportunities
- Public welfare amenity facilities – cleaner air quality and improved temperature management; potential increase in work productivity and hence income-generating opportunities for employees
The above-mentioned output activities met the intended needs of the target beneficiaries at the county and village community (Ri) level. However, an independent impact evaluation study would be required as a future project output/activity component to measure the impact effectiveness, final end-line indicators and actual benefits gained.
3.3.4 Efficiency
Achievement Rating: 4/6 (Moderately Satisfactory – Moderate Shortcomings)
The project achieved the intended outcome. Out of the 4 outputs:
- Output 1 considered fully achieved
- Output 2 considered almost fully achieved
- Outputs 3 and 4 considered partially achieved
As of 3 Dec 2019, project under-spent allocated total project funds by about 32%. This is mainly due to the inability to obtain project funds for procurement and delivery of RE/EE equipment/materials, which is caused by the delayed UN sanctions exemptions/clearance process and the extended banking channel disruption/closure.
Financial reporting processes and templates should be further strengthened for consistencies, financial accountability and transparency purposes in financial budgeting and accounting:
- tracking progress of budget vs expenditure at output level for submissions of all relevant project reports (including APPRs), to demonstrate the efficient management and use of funding on project output-based activities, and align activity/output impact and results to the corresponding financial budgets
- reporting these budget vs expenditure comparisons at output levels at PSC meetings
3.3.5 National Ownership
Achievement Rating: 5/6 (Satisfactory – Minor Shortcomings)
While the SES PRODOC did not include any DPRK counterparts to lead in implementing any project outputs, strong national ownership was achieved at the National/Central level through perfect attendance by DPRK counterpart representatives (NCC-UNDP, MEPI, CBS, SAOS and SCoST) of all PSC meetings.
The evaluation also found high national ownership through strong commitment and interest at the local county level with sustained results of initiation, knowledge/operational transfer and innovative creativity from the SES Project, as follow:
- Increased knowledge and public awareness of the uses of RE/EE to improve rural livelihoods at the county/village community (Ri) level.
- CBS benefited from the energy demand and resource assessments surveys conducted in the year 2016. Statisticians’ capacities and capabilities improved in the surveys related to energy access and related indicators.
- Increased knowledge and strengthened capacities of national and local resources in RE and EE
- High degree of national ownership at county and village community Ri level:
- In-kind contributions through self-supplied equipment and material as well as labor work
- Public buildings are now able to self-operate and maintain their own RE/EE solutions
- Tailoring of training materials by National Consultants to the local context for stronger application by local counties and village communities (Ris)
- Extensive utilization of National Consultants to provide on-the-ground advisory and practical hands-on to local counties and village communities (Ris)
However, the evaluation observed that county stakeholders still require continuous technical support (such as continued update of county energy management plans, conducting energy audits and conducting energy demand and resource assessments) before they could fully function on their own. Due to the incomplete procurement activities, local abilities for efficient manufacturing and/or assembly of RE/EE technologies could not yet be fully realized/achieved to their full potential.
While there are evidences of dissemination, promotion and demonstration on the uses/applications of RE/EE solutions at village community (Ri) level through the county NTDCs, there is an important need to further promote public awareness and impart knowledge to local counties/village communities (Ris) on the efficient use of electricity and better energy management practices to control demand and energy consumption at end-user level.
3.3.6 Sustainability
Sustainability Rating: 3/4 (Moderately Likely - Moderate Risks)
Risk assessments and mitigation strategies/action plans were identified and implemented during project design. However, it did not account for new external environments such as the UN sanctions and the extended banking channel disruption/closure. This resulted in unanticipated sustainability issues (incomplete procurement-related interventions for strengthening energy supply chains) emerging during project implementation and the outcome could not be fully realized/implemented.
The evaluation observed that National Consultants received extensive capacity building and knowledge in RE/EE technology and solutions. This is a commendable effort which should be continued long-term by the relevant DPRK national counterparts to conduct knowledge/operational transfer to have an extended pool of national resources for roll-out of future SES model roll-out.
The SES Project appropriately developed an exit strategy and took into account the following:
- Political factors – there is strong support and commitment from the DPRK Government and CPCs to continue.
- Financial factors – there is financial stability to operate on its own without further financial support.
- Technical factors - skills and expertise needed were suitably assessed and with capacity building activities organized to upgrade the skillsets and competencies of the beneficiaries. However, county stakeholders still require continuous technical support before they could fully function on their own.
- Environmental factors – the SES model can be replicated (in close cooperation with national and local counterparts) to other counties/Ris but this needs to be complemented with appropriate and timely procurement-related interventions for strengthening energy supply chains to maximize effectiveness and impact. It is also critically important to:
- observe and pay attention to safety measures and procedures for RE/EE equipment to minimize/prevent occupational accidents and hazards from occurring
- impart knowledge to local counties/village communities (Ris) on the efficient use of electricity and better energy management practices to control demand and energy consumption
3.3.7 Basic Human Needs / Gender Equality
Achievement Rating: 5/6 (Satisfactory – Minor Shortcomings)
The SES PRODOC did not include specific gender mainstreaming/social inclusion strategy. However the SES Project has factored these into its activities. Basic human needs and gender equality were potentially achieved based on anecdotal/proxy indicator evidence through concrete examples of:
- Increased attendance of children in kindergartens and nurseries because of improved indoor environment during winter and; female teachers benefiting from cleaner air quality due to improved floor heating system.
- Hospitals/clinics able to provide more reliable services to vulnerable groups such as elderly, pregnant women, children, the sick, people with disabilities.
The evaluation further noted that less women participated in training even though the SES Project prioritized/encouraged more women. This is possibly due to the technical roles in the energy and construction sectors predominantly taken up by men in DPRK.
While the reported benefits by project reports could be perceived as immense, the evaluation could not fully verify the actual benefits at ground level. This could be further realized if an impact evaluation study at project output/activity level could be externally conducted by an independent party.
Due to the lack of follow-up in monitoring and evaluating the UNDP-sponsored training courses/workshops’ impact and effectiveness, the evaluation assessed that there was insufficient data available to demonstrate how the capacity development and knowledge dissemination activities of the SES Project improved women’s employment and income generation opportunity as part of contribution to gender equality.
Future projects in DPRK should continue to prioritise gender mainstreaming activities to assess the capacity needs according to gender requirements, and capacity development activities specifically relating to enhancing gender equality and improving the women’s living and livelihood standards.
3.3.8 Synergy
Achievement Rating: 4/6 (Moderately Satisfactory – Moderate Shortcomings)
The evaluation assessed that there were strong synergy effects between the SES Project and CBDRM Project as follow:
- strengthening of embankment concept from CBDRM Project was implemented with SES Project activities to enable and strengthen the implementation of RE technologies.
- as part of the SES Project, public buildings such as kindergartens and schools were retrofitted with EE measures. Some of these public buildings were also selected as evacuation centers in the CBDRM Project. This would result in a positive impact to the well-being and safety of beneficiaries during emergency situations such as floods and typhoons.
- the SES Project implemented EE measures to improve the indoor heating system (Ondol floor heating). This would also increase indoor thermal comfort and also increase the protection of village communities from extreme cold conditions as part of CBDRM Project interventions in disaster risk management.
- EE stoves and solar PV panels were installed as part of the SES Project which helped to improve the heat insulation, improve cooking efficiency and maintain the warm indoor environment. This would result in less timber being collected by SLUG groups and used by village communities (Ris) for firewood which would be required for cooking and also for keeping the indoor environment warm during winter season. The lessened use of timber meant that more trees would be preserved on mountain slopes to strengthen prevention of soil erosion and landslides as part of CBDRM Project interventions in disaster risk management.
- CBDRM project provided seeds (pinus koreansis, larix leptolepis, and castanata crenata) for improving livelihoods and saplings (aronia melanocarpa, and bamboo willow) to prevent soil erosion using soil bioengineering. Communities have further used firewood species such as poplar to enhance biomass resources availability.
The evaluation would also like to highlight another synergy effect that, in another separate UNDP DPRK SED Project, the SED Project Team learnt key lessons from the SES Project to develop innovative energy solutions such as Solar PV Panels to provide sustainable and reliable energy supply to the Spirulina and Pistia Centres in Unryul and Unchon Counties (South Hwanghae Province) for the UNDP DPRK SED Project.
As previously mentioned, the evaluation assessed that the SES Project needed to improve on its weak synergies with other UN agencies and international organizations with similar project/programme outputs and results, which need to be improved. As previously mentioned in Section 3.3.2, information sharing, communication of project results and valuable lessons learned should be further disseminated and strengthened on application, uses and impacts of RE/EE with other UN agencies and international organizations towards collectively achieving the UNSF Outcomes and also the SDGs.
The evaluation also observed that synergy effect between the SES Project and CBDRM Project have undesirable implications such as:
- village communities (Ri) who were not the beneficiaries of both SES and CBDRM Projects would perceive as receiving less “benefits”.
- unhealthy comparisons and competitions between the projects village communities (Ris) observed. For example:
- certain interventions, mainly RE, are based on resource availability.
- the SES and CBDRM projects have responded to the needs on ground considering availability of budget, prioritisation at the community level in order to balance its overall support.
- procurement plans 2018 and 2019 were not materialised under SES and CBDRM. This is beyond the SES Project Team’s control.
IC_ToR_SES_19Sept2019_Final_Ad.pdf
Table 9: Timeline of External Factors/Challenges Faced by UNDP DPRK CO