Building Energy Efficiency in North West of Russia

Report Cover Image
Evaluation Plan:
2011-2017, Russian Federation
Evaluation Type:
Final Project
Planned End Date:
03/2018
Completion Date:
05/2018
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
40,000

Share

Document Type Language Size Status Downloads
Download document 4131 NW Buildings TE TOR International Consultant.docx tor English 40.92 KB Posted 1252
Download document 4131 North-West Buildings TE TOR National Consultant.doc tor English 97.50 KB Posted 1126
Download document 4131_NW Buildings_TE_Final_report.docx report English 1125.61 KB Posted 1307
Title Building Energy Efficiency in North West of Russia
Atlas Project Number: 00059438
Evaluation Plan: 2011-2017, Russian Federation
Evaluation Type: Final Project
Status: Completed
Completion Date: 05/2018
Planned End Date: 03/2018
Management Response: Yes
Focus Area:
  • 1. Others
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 1.5. Inclusive and sustainable solutions adopted to achieve increased energy efficiency and universal modern energy access (especially off-grid sources of renewable energy)
Evaluation Budget(US $): 40,000
Source of Funding: GEF
Evaluation Expenditure(US $): 31,000
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Nationality
Mr. Alexey Zakharov National Evaluation Expert RUSSIAN FEDERATION
Mr. Paata Janelidze FE International Team Leader GEORGIA, REPUBLIC OF
GEF Evaluation: Yes
GEF Project Title: Building Energy Efficiency in North West of Russia
Evaluation Type: Terminal Evaluation
Focal Area: Climate Change
Project Type: FSP
GEF Phase: GEF-4
GEF Project ID: 3659
PIMS Number: 4131
Key Stakeholders: Russian Energy Agency
Countries: RUSSIAN FEDERATION
Lessons
Findings
1.

3. Findings

(As requested by the ToR, in addition to a descriptive assessment, all criteria marked with (*) must be rated) 

3.1 Project Design / Formulation

As recommended by the Guidance for Conducting Terminal Evaluations of UNDP-Supported, GEFFinanced Projects the findings of this chapter are based on the analysis whether or not: - The BEENWR Project objectives and components were clear, practicable and feasible within its time frame - The capacities of the Executing Agency and its counterparts were properly considered when the project was designed - Lessons from other relevant projects (if any) were properly incorporated in the project design - The partnership arrangements were properly identified, and roles and responsibilities negotiated prior to project approval - Counterpart resources (funding, staff, and facilities), enabling legislation, and adequate project management arrangements were in place at project entry - The project assumptions and risks were well-articulated in the ProDoc 

An additional important point in terms of project formulation is to consider whether the planned outcomes were "SMART" (S - Specific: Outcomes must use change language, describing a specific future condition; M - Measurable: Results, whether quantitative or qualitative, must have measurable indicators, making it possible to assess whether they were achieved or not; A - Achievable: Results must be within the capacity of the partners to achieve; R - Relevant: Results must make a contribution to selected priorities of the national development framework; T - Time- bound: Results are never open-ended. There should be an expected date of accomplishment).


Tag: Energy Effectiveness Relevance Programme/Project Design Results-Based Management

2.

3.1 Project Design / Formulation(continuation)

Relevance of the problem addressed The need in building local capacity and demonstration local solutions to improved EE in buildings in Pskov, Vologda and Arkhangelsk Oblasts, is convincingly justified in the ProDoc. Based on the housing sector development trends (dynamic construction sector) and state of energy performance of buildings in the baseline scenario, it is demonstrated that: (i) there is a high likelihood of scaling up construction (and renovation as well) of residential buildings in those Oblasts of the North West Federal Okrug of Russia; (ii) there is high GHG reduction potential due to the high potential for energy savings (compared with baseline) in newly constructed and renovated buildings. In addition, as provided in the ProDoc, discussions with stakeholders during project identification and preparation, indicated that there is a strong commitment of the regional governments to EE programmes. Continuation of the construction of residential buildings at about the same magnitude as during the pre-project period, has been actually demonstrated during the BEENWR Project implementation. Most of the residential buildings built in 1980-1990-ies, suffer from outdated design and inefficient heating systems and the renovations include typically only improvement of their appearance, but not EE measures.


Tag: Energy Effectiveness Relevance Programme/Project Design Results-Based Management

3.

3.1.1 Analysis of Logical Framework (Project logic /strategy; Indicators)

The Logical Framework (LogFrame) is a key basis for planning of detailed activities under the implementation framework that was defined in the ProDoc. The LogFrame shall in principle serve to monitor and evaluate the overall project achievements – based on defined targets and indicators to measure these targets. The original LogFrame, at the certain level, is lacking internal logic, clarity and consistency; planned activities are not always appropriate and/or adequate towards the identified targets. For instance: 

 


Tag: Effectiveness Efficiency Relevance Programme/Project Design Project and Programme management Results-Based Management Theory of Change

4.

3.1.1 Analysis of Logical Framework (Project logic /strategy; Indicators)(continuation)

- Activity 1.1.a (to achieve Output 1.1; indicator - Operational oblast-level legal and regulatory framework for enforcing and monitoring building codes in Vologda oblast) considers analysis of enforcement models in other CIS countries. The ProDoc doesn’t further provide whether there are any such enforcement models. None of the UNDP-supported GEF-financed projects on energy efficient buildings in the CIS countries (Armenia, Belarus, Kazakhstan, Kyrgyzstan, Uzbekistan, Turkmenistan) was completed before the BEENWR Project start. The earliest project (Promoting Energy Efficiency in Public Buildings in Uzbekistan) started in 2009 and thus, no lessons learned was available while designing this BEENWR Project. Therefore, more logical would be to consider enforcement models in EU states, especially in Eastern European countries. 


Tag: Effectiveness Efficiency Programme/Project Design Project and Programme management Results-Based Management Risk Management Theory of Change

5.

3.1.1 Analysis of Logical Framework (Project logic /strategy; Indicators)(continuation)

The sites for the demo projects are not specified in the LogFrame. This is appropriate. But on p. 16 the exact location of the demo project is provided: “under the Activity the pilot construction site is located in Nifantov, a settlement in the Shekninskii Municipal District that is not far from Vologda City” - Indicator for achievement of Output 3.3 is “Energy audit program in place for public and residential buildings when they are commissioned”. The corresponding activities among others include the reconstruction of six residential buildings and providing energy performance certificates both for these buildings and for more than 570 buildings that will be audited. The ProDoc doesn’t provide why for demonstration of economic benefits of EE measures, auditing of 570 buildings is necessary and whether this can be considered as an effective use of the resources of the BEENWR Project


Tag: Energy Effectiveness Efficiency Programme/Project Design Project and Programme management Quality Assurance Results-Based Management Risk Management

6.

3.1.2 Assumptions and Risks

Assumptions: Assumptions and risks are outlined in the Project Results Framework for each project indicator and target and built around the continued commitment of all BEENWR Project Partners: - Oblast-level, okrug-level, and municipal-level support for the project will remain strong, and legislation and supporting regulations will be accepted - More efficient codes prepared under the BEENWR Project will enter into effect and be enforced - Municipalities will have sufficient interest and awareness in the EE programs for heat and hot water supply - Willingness and availability for training by project stakeholders - Students and practicing professionals will have strong incentives to apply the techniques that they have learned through training

However, assumptions are not always robust. In particular: - It is assumed that the existing financing model/schemes would ensure the allocations of necessary funds for EE construction and renovation. This might be problematic for the residential buildings owned by the municipalities, especially for the Oblasts with budgetary constraints. This was proved during the BEENWR Project implementation when e.g. in Pskov Oblast, under the Business-as-Usual (BAU) scenario, the priority is being given to the less costly constructions/renovations - It is assumed that practically all EE measures in buildings are cost-effective and thus attractive for investors. Therefore, no cost-benefit analysis of applied EE measures is foreseen in the ProDoc. During the actual implementation cost effectiveness of all EE measures were analyzed and only financially feasible options were recommended for implementation in the demo projects - The planned activities among others include establishment of Inter-Regional Professional Training Center with branches across the North West Federal Region, with corresponding staffing and budget allocation. It was assumed that these resources would be available. Actually, the concept of the Inter-Regional Professional Training Center was developed only in October 2015 and the Center itself was established in 2016.


Tag: Effectiveness Efficiency Programme/Project Design Results-Based Management Risk Management

7.

3.1.2 Assumptions and Risks (continuation)

As it is seen from the table, not all the potential risks were identified in the ProDoc, and the identified ones were underestimated. Unfortunately, risks weren’t monitored carefully; risks were updated in ATLAS only three times (19/08/2013; 21/05/2015, 12/10/2017) but not all risks were entered ATLAS. For instance, it is stated in PIR 2013 that there is a critical financial risk (without description), which is not included into the risk log. 


Tag: Efficiency Programme/Project Design Project and Programme management Risk Management

8.

3.1.3 Lessons from other relevant projects incorporated into project design

Before the BEENWR Project start certain lessons were learned from the UNDP-supported GEFfinanced projects: - Capacity Building to Reduce Key Barriers to Energy Efficiency in Russian Residential Buildings and Heat Supply (1998-2004), which addressed three key activities for energy efficiency improvement and energy saving in heat supply and consumption in the city of Vladimir: o Institutional changes that create incentives for greater EE and energy conservation in existing distribution and end-use infrastructure o Greater penetration of autonomous sources of heat; and o Capacity to analyze the financial and economic aspects and feasibility of capital investment projects for commercial and multi-lateral financing 

- Cost Effective Energy Efficiency Measures in the Russian Educational Sector (2002-2006), the overall objective of which was to contribute to the GHG abatement by improving the EE of Russian educational facilities. This objective was planned to be achieved through awareness raising, training and capacity building, demonstration program and development of schemes and tools including models for sustainable administrative and financial solutions. The NorthWest Russia (Tver, Arkhangelsk, Karelia) was selected as a pilot region for that project. The project composed of three components including Component 2: Demonstration Program, which was aimed at: o Implementation of cost-efficient energy conservation measures to mitigate GHG emissions o Demonstration of a sustainable combined program of EE improvements using revolving funds for replication o Verification of technical and cost performance of the selected EE measures o Identification of major obstacles and lessons learned in order to incorporate these into the management tools and guidelines for planning and implementation of replication projects o Awareness and visibility for the EE measures in order to promote their wider replication


Tag: Energy Effectiveness Efficiency Relevance Knowledge management Project and Programme management Results-Based Management

9.

3.1.3 Lessons from other relevant projects incorporated into project design (continuation)

Since November 2010 the implementation of EBRD/GEF project “Improving Energy Efficiency in Public Buildings in the Russian Federation” started, the objective of which was “To reduce greenhouse gas emissions in Russia by improving efficient use of energy in Public Buildings such as kindergartens, schools, hospitals, and public offices. By combining technical assistance with financing, the EBRD aims to help local authorities overcome common obstacles to financing energy efficiency (EE) improvements such as allocation of resources for energy audits and project preparation, tendering procedures, and management of larger-scale programs that may need additional dedicated resources”. This experience and lessons learned of this project, with similar objectives, could be very helpful for the design of the BEENWR Project. However, this was not possible because implementations of these two projects started practically simultaneously.


Tag: Relevance Programme Synergy Programme/Project Design Results-Based Management

10.

3.1.4 Planned stakeholder participation

Key stakeholders of the BEENWR Project, including governmental agencies and ministries, and first of all, the Office of Plenipotentiary Representative of the President of the RF in the North-West Federal Okrug as the Executing Agency of the project, have been actively involved during the design phase.

The Office was an executive partner for the BEENWR Project preparation (Project Preparation Grant: US$ 140,000 from GEF; Duration: January 2009 - December 2010). Planning of the stakeholder participation has started from the early stages of the BEENWR Project development. The planning was based on clear understanding of the features of Russian Federation regarding the housing sector, namely roles played by key institutions. Among them:

The plenipotentiary of the President of the RF in the North West Federal District (Okrug) – overall coordination - Line ministries (Ministry of Regional Development; Ministry of Energy; Ministry of Natural Resources and Ecology) – Legal aspects - Authorities responsible for enforcement of building codes and related legislation at the Oblast level – corresponding legal and regulatory provisions at Oblast level - Municipalities - legal and regulatory provisions at municipal level - Educational institutions - training programs, academic plans and curricula - Building management companies - Energy producers and utilities Based on the abovementioned text it is the Consultant’s opinion that stakeholder participation has been planned adequately.


Tag: Efficiency Sustainability Partnership Project and Programme management Results-Based Management

11.

3.1.6 UNDP comparative advantage In general UNDP comparative advantage lies in its experience in integrated policy development. UNDP’s assistance in designing and implementing activities is consistent with both the GEF mandate and national sustainable development plans. UNDP has implemented a number of EE in buildings projects in Central and Eastern Europe and in the CIS region. UNDP Russia by the start date of the BEENWR Project had the adequate administrative capacity for implementation of it. Comparative advantage of UNDP is proven by the fact that UNDP PSO is responsible for the overall communication and coordination between programme partners and projects under the Umbrella Programme in Russia.

3.1.7 Linkages between project and other interventions within the sector As mentioned above, there are a number of projects in Russia, with the similar objectives, either already implemented or being under the implementation. BEENWR Project design calls for cooperation with similar activities in the sector, to consider their lessons and avoid overlapping/duplication of activities. However, the linkages with them are fragmental. The Consultant agrees with the MTR, according to which the ProDoc “does not really provide many other details about the linkages and co-operation opportunities with other interventions within the sector nor does it explain to what extent the project can build on and how it complements the results of the earlier projects”. This looks not logical because the importance of these linkages was recognized in the ProDoc. Indeed, it is stated in Chapter “Design principles and strategic considerations” that “a number of energy saving projects have been implemented in the North Western Federal Okrug … These projects remained mainly uncoordinated”. Nevertheless, the ProDoc is shifting the establishment of linkages (and analysis of LL as well) to the implementation phase rather than address co-operation opportunities and elaborate corresponding plans and arrangements. 


Tag: Programme Synergy Project and Programme management Results-Based Management Risk Management Strategic Positioning

12.

3.1.8 Management arrangements(continuation)

- Project Board / Project Steering Committee (PSC) - overall management of the project by making management decisions and playing critical role in quality assurance of monitoring and evaluation. Permanent secretary of PSC: NPD - Project Manager (PM) - responsible for day-to-day run the project on behalf of the Implementing Partner to ensure that the project produces the results specified in the ProDoc - Regional Team Leaders (three regional team leaders in each region) - day-to-day coordination of project activities at the oblast level - Project Support (Project Assistant) - project administration, management and technical support to the Project Manager - Unlike similar UNDP/GEF building EE projects in CIS countries, BEENWR Project team was not supported by the long-term International Chief Technical Advisor (CTA). Involvement of short-term International consultants was limited and took place only in the pre-MTR period of implementation where the international advisor on energy management, worked only for a short period for the project. On the other hand, as correctly stated in the MTR report, “the capacity of the project management ultimately determines the success or failure of the project. Several nationally executed GEF funded projects in different countries have also evidenced problems in attracting experienced enough local project managers that can manage the project entirely on their own without external expert support and advice, who can bring state of the art knowledge and experience from similar projects implemented elsewhere and monitor the progress and substantive impact of the project on an ongoing basis starting from the project inception. Therefore, and especially for the project of this size and complexity, close monitoring of the project progress and inclusion of an experienced international project adviser into the project management structure with demonstrated capacity and adequate experience from similar projects implemented in other countries to serve that function together with adaptive management advise should have been considered already at the project design stage”. The Consultant fully agrees on this statement. 


Tag: Energy Effectiveness Efficiency Project and Programme management Results-Based Management

13.

3.1.8 Management arrangements(continuation)

These management arrangements in general, with some exceptions, were followed during the BEENWR Project implementation:

In June 2013 the Office of Plenipotentiary Representative of the President of the RF in the North-West Federal Okrug informed UNDP PSO that it wouldn’t be able to carry out its supervisory and coordinating role for the BEENWR Project after 2013. The second PSC meeting held on 16.12.2013 recommended replacement of the Plenipotentiary Office by the Russian Energy Agency and appointed Deputy Director General of the Russian Energy Agency (REA) as a new NPD. Officially the REA took responsibility of the Executing Agency since March 2014 - National Responsible Party also was changed. Since mid-2014 International Sustainable Energy Development Centre under the auspices of UNESCO is serving as a Responsible Party - From June 2011 until August 2013 the project has been managed (full-time) by Mr. Grigory Markin, from August 2013 until December 2014 – by Mr. Andrei Karpus, and since January 2015 – by Mr. Vitaly Bekker - During the inception phase the management structure has been slightly changed by introducing a position of National Deputy Director (NDD). Formally this is acceptable but actually, NDD was playing double role, of: (i) NDD itself; and at the same time of (ii) Director of the North-West branch of the “Energy Forecasting Agency” (Responsible Party). In addition, in some cases she was taking responsibilities of the Project manager. This was a problem because, according to the MTR, there was a “fundamental lack of clarity on who is actually managing the project” 10, NDD, PM or Project Implementation Coordinator (former PM continued his work at this position). NDD was providing all the substantive advice and leadership for project implementation and, among others, was managing all the discussions with the project partners on any financing related matters.  


Tag: Energy Efficiency Policies & Procedures Project and Programme management Results-Based Management

14.

3.2 Project Implementation

As recommended by the Guidance for Conducting Terminal Evaluations of UNDP-Supported, GEFFinanced Projects, TE findings in this chapter are based on assessment of implementation approach, whether or not: (i) The logical framework is used during implementation as a management and M&E tool; (ii) Effective partnerships arrangements are established for implementation of the project with relevant stakeholders involved; (iii) Lessons from other relevant projects are incorporated into project implementation; and (iv) Feedback from M&E activities used for adaptive management. 

3.2.1 Adaptive management (changes to the project design and project outputs during implementation)

The implementation of the BEENWR Project can be divided into two periods of engagement of two different executing Agencies, and a short transition period between them. In the first period (until 2014) the Management Team (i.e. NDD) was in a position that all planned activities of ProDoc are logical and feasible, probably because the Office of Plenipotentiary Representative of the President of the RF in the North-West Federal Okrug was an executive partner during the BEENWR Project design (under the GEF PPG) and was familiar with the arguments for designing the project in that particular way. PMs with irrelevant competence (implicitly stated in the MTR report and confirmed by the TE Team through the interviews) also were not always able to flag the critical issues and apply adaptive management for their adequate addressing. Project Implementation Reviews (PIRs) were not focused on measuring the progress towards the Outcomes and how the achieved results (or in progress) have contributed (or would contribute) to this in practice. In addition, the “PSC has not really been able to fulfil its envisaged role in effectively monitoring and guiding the project implementation” (MTR). Therefore, the application of the adaptive management was negligible. Partly, this can be blamed by inadequate LogFrame with not always right indicators and targets, but the BEENWR Project has done not much to correct the situation during the implementation either. 


Tag: Effectiveness Efficiency Human and Financial resources Monitoring and Evaluation Project and Programme management Quality Assurance Results-Based Management Risk Management Theory of Change

15.

3.2 Project implementation - 3.2.1 Adaptive management (changes to the project design and project outputs during implementation)(continuation)

 In accordance with the amended EE Law, it was not possibility for regional administration to introduce any new EE construction Oblast norms in addition to the Federal ones. Therefore, the activities aimed at development of new regional construction norms (Two international experts in EE municipal norms were involved) were not continued and the already developed materials for Vologda region submitted to the National Housing Refurbishment Fund and Ministry of Construction o Indicator 1.3 and corresponding target (see Table 1 above) became more specific; reference is made to the specific software, Energy Management and Information System (EMIS), developed by the UNDP project in Croatia, and its application for municipalities in the Pskov region along with appointment of the first energy managers in Pskov was planned (and actually implemented in Pskov and Vologda oblasts) o Five indicators and targets for Outcome 2 were consolidated into one, simple to monitor and verify, indicator and two targets.


Tag: Effectiveness Efficiency Sustainability Business Model Implementation Modality Project and Programme management Results-Based Management

16.

3.2 Project implementation - 3.2.1 Adaptive management (changes to the project design and project outputs during implementation)(continuation)

The BEENWR Project management has applied adaptive management and implemented alternative projects: o For new constructions: Since 2013, the program for the relocation of Russian citizens (Resettlement Program) from dilapidated and emergency housing is being implemented (https://www.reformagkh.ru/relocation). The program is financed by the Fund to Promote Reform in the Residential and Communal Services Sector and Budgetary Resources of the Regions. As a rule, only those basic EE measures are supported that are considered by the existing rules. Often only minimum requirements for the thermal protection of buildings do not ensure the achievement of a normative levels of energy consumption. The BEENWR Project used these circumstances and managed to identify two residential buildings in Porkhov (Pskov Oblast) and Parfino (Novgorod Oblast), which were implemented under the above program and the Project has provided co-financing of the EE measures. o For renovations (capital repair): in cooperation with Arkhangelsk regional administration BEENWR Project has short-listed 12 buildings that were included in the renovation program of the city of Arkhangelsk for 2015, out of which only one was selected for demo implementation. Therefore, the BEENWR Project looked for the alternative choices in other Oblasts including public buildings. Finally, two renovation projects were implemented in Pskov and Vologda Oblasts (EE retrofit of kindergarten «Raduga» in the Ostrovsky district, Pskov oblast and EE retrofit of the main building of Vologda Health Clinic No. 3). More details are presented in Chapter 3.3.1 below

 Scope of Activity 3.3.d.iv (Measure energy performance of the renovated buildings and provide buildings with energy certificates) was extended and energy savings were calculated based on actual monitoring. For this purpose, energy performance of the reference buildings, also were monitored (in order to calculate baseline energy demand)


Tag: Efficiency Monitoring and Evaluation Project and Programme management Results-Based Management Risk Management Theory of Change

17.

3.2 Project implementation - 3.2.1 Adaptive management (changes to the project design and project outputs during implementation)(continuation)

The necessity of the development of EMIS was underlined by the International consultants from very beginning of the BEENWR Project implementation. In the report “Best Practice methodology for Energy Management in Municipalities. Best practice methodology, recommendations and examples”, prepared by Mr. Zoran Morvaj in June 2012, the importance of EMIS (to continuously record, archiveand analyze energy consumption data and thus, a central tool in establishing Energy Management and introduction of buildings certification scheme) and its detailed description with best practices of its application were presented. EMIS was one of the main focuses also of another report prepared by Mr. Morvaj “Project strategy and adaptive management. For achieving results and sustainability” (November 2012). 

Unfortunately, these ideas reflected in those reports weren’t followed. The related activities were focused on elements of the monitoring system, such as development of a prototype of EMIS for the Arkhangelsk Oblast (under Outcome 3.2); creation of a municipal information system providing quick data collection and coordination of energy consumption in public buildings (objects of municipal education); development of an accounting model of energy by using geographical information systems (GIS) in the Pskov Oblast, etc. However, the progress was insufficient and not cost-effective (e.g. USD 150,000 for separate studies and software development in the Arkhangelsk region). Therefore, the issue of EMIS was a focus of the third PSC meeting held on December 4, 2014. This PSC meeting was the first one chaired by new NPD representing a new Executing Agency (REA). This PSC meeting took a crucial decision, which largely determined the further implementation strategy of the BEENWR Project and therefore, more details are provided below.At the meeting, the regional coordinator in the Pskov Oblast presented the status of a prototype of EMIS being under the development in the Ostrovsky and Nevelsky districts. The PSC noted that the creation of an EE management system at the municipal level, based on the information system created, would be one of the most important results of the BEENWR Project.


Tag: Effectiveness Efficiency Project and Programme management Results-Based Management

18.

3.2 Project implementation - 3.2.1 Adaptive management (changes to the project design and project outputs during implementation)(continuation)

Another International Consultant, who prepared Recommendations for Updated Project Strategy, just after the third PSC Meeting, also recommended: (i) to ensure that the necessary backstopping (incl. financing, appointment of energy managers etc.) is provided; (ii) to explore the replication potential of the EMIS developed in the Pskov region in Arkhangelsk and Vologda and to clarify the linkages to the already concluded GEF funded activities on EMIS development in Arkhangelsk; and (iii) to take stock on the UNDP supported EMIS activities in Croatia. As a result, the BEENWR Project Management Team took a right decision and undertook relevant steps for application of EMIS: 


Tag: Energy Effectiveness Efficiency Monitoring and Evaluation Partnership Project and Programme management Results-Based Management Risk Management Theory of Change

19.

3.2.2 Partnership arrangements (with relevant stakeholders involved in the country/region)

Ensuring the partnership with the stakeholders has a crucial importance for such a complex project as the BEENWR Project is, especially if the activities are either not precisely described in the ProDoc or due to the changing environment of the project implementation, some risks are becoming critical and adaptive management is to be applied. The BEENWR Project from the beginning was trying to establish effective partnership with the key stakeholders by engaging them into the PSC, organizing permanent information exchange meetings and discussions on the matters related to the project planning and implementation. However, these attempts were not always successful and as a result, there were significant delays by the MTR at least, in implementation of two components of the BEENWR Project (aimed at achievement of Outcomes 1 and 3). The risks of further delays related to Outcome 3 remain critical also after the MTR and the BEENWR Project experienced major and repeated delays in its demo projects including due to the unexpected slowness of necessary approvals and authorizations. Therefore, efforts to ensure that all the stakeholders, within their mandate, facilitate the progress of achievement of the planned results, have been intensified. 


Tag: Energy Effectiveness Efficiency Partnership Project and Programme management Results-Based Management

20.

3.2.2 Partnership arrangements (with relevant stakeholders involved in the country/region) (continuation)

Through the representatives of the partners, engaged in the PSC. The PSC, for the purposes of efficient decision-making and work coordination, includes representatives in charge from various ministries and other project partners, including Ministry of Industry and Trade of the Russian Federation, Ministry of Economic Development of the Russian Federation, Ministry of Energy of the Russian Federation, Ministry of Natural Resources and Environment of the Russian Federation, Ministry of Education and Science of the Russian Federation, UNDP Project Support Office in Russia as well as other Project partners. - Meetings, discussions followed by the informal agreements at the working level. BEENWR Project web (http://www.undp-eeb.ru/en/news.html), PIRs, technical reports, provide many evidences on that; this was confirmed by the interviewees during the TE mission. Among them:


Tag: Energy Effectiveness Efficiency Knowledge management Partnership Project and Programme management Results-Based Management

21.

3.2.3 Feedback from M&E activities used for adaptive management

In accordance with the ProDoc, the BEENWR Project should regularly use feedback from M&E to appropriately and adequately address any new challenges (issues) and thereby ensure the achievement of established targets. The M&E plan includes the LogFrame, Inception Report and MidTerm Review and thus, changes in the LogFrame, in accordance with the recommendations of the Mid-Term Review, also were used as a basis for adaptive management. Other activities under the M&E plan, which were used for the adaptive management, were decisions of the PSC meetings and findings and recommendations of technical reports prepared by the experts and consultants engaged in the BEENWR Project implementation. The changes in the LogFrame are already discussed in the Chapter 3.2.1 above. Other feedbacks from M&E activities used for adaptive management are as follows:

- Strengthening the management structure – From the very beginning of the implementation, along with the NPD, a position of NDD was introduced; in the middle of 2013, when the significant delays were observed, the new PM was engaged while former PM continued working as the Project Implementation Coordinator. Nevertheless, these changes haven’t been resulted in acceleration progress. The real progress was observed only after the changing of the Implementing Partner (Executing Agency) followed by the replacement of the (second) PM - After the evaluation of results of the development of Energy Monitoring System in Arkhangelsk and assessment of accompanied risks, the BEENWR Project has opted EMIS developed under the UNDP/GEF project in Croatia. After the successful adaptation/adjustment of EMIS to the Russian reality, the focus of Component 1 was shifted to the further application of EMIS. Positive experience with application of the EMIS in Pskov Oblast was extended to the Vologda Oblast 


Tag: Energy Effectiveness Efficiency Monitoring and Evaluation Project and Programme management Results-Based Management

22.

3.2.3 Feedback from M&E activities used for adaptive management (continuation)

- Recommendation 1 suggested reducing the number and re-establishing of indicators and targets in order to be focused on the main targets to be achieved under each outcome considering qualitative and sustainability aspects. This recommendation was accepted and actually followed during the post-MTR period. This really helped in strengthening the focus on main targets

- It was stated in Recommendation 2 that there was a need in serious reorganization and strengthening of the BEENWR Project management. It was recommended to find experienced enough local project managers to run the project entirely on their own, the recruitment of an experienced international project management and technical advisor (with knowledge of the Russian language/Russian speaking staff and previousworking experience in the Russian building sector). In response to this recommendation the new PM was recruited and Chief Technical Advisor (Mr. Gorshkov) engaged additionally (Mr. Moskalyov, CTA on implementation of educational component, was already engaged by the MTR)

- Recommendation 3 suggested that for leveraging funding for demo projects, the BEENWR Project should intensify its efforts and tie its activities more closely to the already available financing sources instead of performing research work, energy audits, energy certificates and preparation of feasibility studies, which would be useless, if not leading to concrete EE investments. Following this recommendation, the BEENWR Project has established close relationships with the Regional administrations to identify budget lines that could be attributed to project activities in this direction


Tag: Energy Effectiveness Efficiency Monitoring and Evaluation Project and Programme management Results-Based Management

23.

3.2.4 Project Finance

For the evaluation of BEENWR Project finance, the key financial aspects of the actual costs and leveraged and financing have been assessed. Differences between planned and actual expenditures also were assessed and explained. Findings of the financial audits also were considered. The following has been observed:  

In the ProDoc the BEENWR Project total resources were USD 33,340,000, including a GEF grant (USD 5,840,000), co-financing from the Government (USD 11,490,000) and privatesector (USD 13,470,000) and in-kind contribution from the Government (USD 2,540,000). The planned and actual co-financing are presented in Table 4. - According to the ProDoc “Federal and regional co-financing will be invested into design and construction of infrastructure and public buildings. The partner institution or institutions will finance residential, cultural, and commercial buildings”. By the date of MTR, no implementation of demo projects was even started. Therefore, it is the Consultants opinion that co-financing confirmed by the MTR (details are presented in Annex 8 of the MTR report) cannot be accounted as an actual co-financing because those costs were spent under the Business-as-Usual scenario and should be deducted from the co-financing. In addition, co-financing for the demo project in Parfino, Novgorod Oblast (RUB 54.5 million or USD 0.97 million), is missing in this table. Revised co-financing data are presented in Table 4.1 


Tag: Energy Efficiency Global Environment Facility fund Government Cost-sharing

24.

3.2.5 Monitoring and evaluation: design at entry and implementation (*)

M&E Design at Entry The Request for CEO Endorsement/Approval for the BEENWR Project among other includes description of the budgeted Monitoring and Evaluation (M&E) plan with identified responsible parties for M&E activities, allocated indicative budget, and specified time frame for each M&E activity. According to M&E plan, M&E should be conducted in accordance with established UNDP and GEF procedures. Monitoring Framework and Evaluation was further substantiated in the ProDoc. The indicative M&E budget was USD 285,000 or about 4.9% of the total GEF grant.

M&E activities among others include measurement of means of verification for project indicators and measurement of means of verification for project progress and performance (measured on an annual basis), etc. Standard M&E tools include LogFrame (contains performance and impact indicators as well as means of verification), Inception Report, Mid-Term Review, Terminal Evaluation as well as standard UNDP and GEF project progress reports – Annual Project Reviews (APR) and Project Implementation Reviews (PIR).

At the same time, as mentioned above in Chapter 3.1.1, the original LogFrame, at the certain level, was lacking internal logic and consistency and therefore, some indicators and targets have been redefined to better and more specifically reflect project outputs. Risk analysis was inadequate. Naturally, the M&E design at project start was not very specific towards measurement of means of verification for progress in achieving of those outputs. Based on the above mentioned the M&E design at project start up is rated as Moderately Satisfactory (MS).


Tag: Energy Efficiency Monitoring and Evaluation Programme/Project Design Project and Programme management

25.

Implementation of M&E(continuation)

Project implementation has been regularly reviewed by the PSC, chaired by the NPD. AWPs have been regularly developed and submitted for approval to the PSC. The PSC plays a critical role in M&E by quality assurance of the activities and outputs. It ensures that required resources are committed and negotiates solutions to any problems with external parties. Nine meetings of the PSC were organized in total (No.1: 21.06.2011; No.2: 16.12.2013; No.3: 04.12.2014; No.4 (online): 11.08.2015; No.5: 18.11.2015; No.6 (on-line): 28.04.2016; No.7 (on-line): 30.09.2016; No.8: 22.11.2016; No.9 (on-line): 26.06.2017). At each PSC meeting the achievements to date were discussed, assessed and agreed, and also guidance for the future implementation provided. - Inception Workshop was held on June 20, 2011 (Inception Report has been finalized in January 2012) with participation of UNDP CO, UNDP RTA, National Implementing Partner and key stakeholders. Among others it recommended revisions to the LogFrame, identified “potential adversary circumstances that may affect project implementation” (but all the risks were rated as low)

The MTR mission was conducted in November-December 2013, MTR report delivered in January 2014. Overall Project Achievement and Impact was rated as Moderately Unsatisfactory (MU). Rating for project design was Moderately Satisfactory (MS); Project implementation (Efficiency) – MU; Outcome 1 – MU; Outcome 2 – S; Outcome 3 – MU; Relevance – Highly Satisfactory (HS); Country Ownership – Satisfactory (S); Stakeholder Participation – S; Replication Approach – MS; Sustainability – S; Linkages between project and other interventions within the sector – MU; Management Arrangements – MU; Project Implementation – MU (based on: Project implementation and management approach and arrangements – U; Project administration and financial management – MS; Monitoring, evaluation and adaptive management - MU; Contribution of Implementing and Executing Agencies – MU; Stakeholder involvement, outreach and co-ordination with other related initiatives - MU; Identification and management of risks – U)The BEENWR Project was also subject to external financial audit. All financial audits had “no comments or observations” and provided overall satisfactory ratings - The revised LogFrame (after the MTR) and M&E plan in the ProDoc served as a source of annual targets for the project. As for the baselines for the established indicators, some of them were appropriately and adequately based on the actual monitoring, e.g. energy consumptions in reference buildings (newly constructed or renovated without additional EE measures); for someother indicators the Status Quo scenario was used similarly to ProDoc, which is not always appropriate. For instance, baseline emissions for buildings including in EMIS were estimated with high level of uncertainty (see details in Chapter 3.3.1 below). 


Tag: Energy Efficiency Monitoring and Evaluation Project and Programme management Quality Assurance Results-Based Management Risk Management

26.

3.2.6 UNDP and Implementing Partner implementation / execution coordination, and operational issues (*)

UNDP (Implementing Agency) implementation 

This BEENWR Project is being implemented under the National Implementation Modality (NIM) and thus the role in the implementation is not as high as under the Direct Implementation Modality (DIM). Nevertheless, a number of direct responsibilities of the UNDP was described in the ProDoc (the exact roles of UNDP CO in the project organization structure should be assigned at the Inception Workshop). Among them: - The Implementing Partner and the NPD should be in charge of planning, coordination, administration and financial management of the BEENWR Project with support by the UNDP CO - PM should work under the direct supervision of the NPD and UNDP CO Head of Environment & Energy UnitIn case consensus cannot be reached within the PSC, the final decision should rest with the UNDP PM, to ensure UNDP’s ultimate accountability for the BEENWR Project results - An integrated management model should be designed for a rural residential construction site, that would address institutional, regulatory and information management issues throughout design and construction process focusing on EE, and based on a partnership approach between the UNDP, regional authorities, and investment institutions operating in the region - UNDP CO and the UNDP Regional Coordination Unit (RCU) would conduct visits to project sites - In addition, the BEENWR Project was considered under the Umbrella Programme “Improvement of Energy Efficiency in the Russian Federation” implemented jointly by UNDP, EBRD and UNIDO with the participation of key federal branch departments and regional authorities. As suggested in the Umbrella Programme, UNDP CO should be responsible for the overall communication and coordination between programme partners and projects.


Tag: Efficiency Human and Financial resources Knowledge management Policies & Procedures Project and Programme management Results-Based Management

27.

3.2.6 UNDP and Implementing Partner implementation / execution coordination, and operational issues (*) -UNDP (Implementing Agency) implementation (continuation)

At the same time, project management team before the MTR was lacking capacity. As mentioned above, the MTR identified a management risk and inadequate capacity for adaptive management. In the ProDoc very simple requirements were considered for the Project Manager position (the ToR is included also in the Inception report, which was prepared after the selection of the PM), namely: experience as a project manager of not less than 3-5 years. No other specific experience was required that seems illogical for such complex project and not in line with the establishedprocurement rules. However, during the selection process specific requirements were added (http://www.undp.ru/index.php?iso=RU&lid=2&cmd=vacancies&id=926 ). Namely:

Higher education in technical sciences or relevant field. An advanced degree is an advantage; - Work experience in implementing projects on energy savings and energy efficiency - Work experience in managing large projects at enterprises - Participation in international projects is an advantage - Excellent knowledge of peculiarities of pilot sites in project’s pilot regions.It must be noted that length of the experience wasn’t specified at all. This was resulted in the selection of Mr. Grigory Markin (born 04.05.1988), who would be 23 years old when selected as a project manager. It is very unlikely that he could meet the requested criteria (even without criteria of 5-year experience). 


Tag: Effectiveness Efficiency Human and Financial resources Implementation Modality Project and Programme management Results-Based Management

28.

3.2.6 UNDP and Implementing Partner implementation / execution coordination, and operational issues (*) -UNDP (Implementing Agency) implementation (continuation)

Executing Agency implementation

It must be noted that situation was greatly improved after the changing the Project Management Team in 2015. Since the BEENWR Project was implemented under the NIM modality, the UNDP didn’t play a leading role in achievement of the planned results. Nevertheless, UNDP probably could be more active in monitoring the established indicators, especially related to the Objective (CO2 emissions from energy use in new and renovated buildings in the 3 participating oblasts). According to the ProDoc, in achieving of a target of direct GHG reductions of 48,050 tCO2e, main role was given to the demo projects in Arkhangelsk and their exclusion from the final list led to the sharp decrease of the emission reduction. Fortunately, due to the implementation of EMIS, additional (not foreseen in the ProDoc) GHG reductions were achieved, and overall target was achieved at the level of 50.23%. Based on the abovementioned the UNDP implementation is rated as Moderately Satisfactory (MS).

As mentioned above, Executive Agency was changed in 2014. Initially the BEENWR Project was executed by the Office of Plenipotentiary of the President of the Russian Federation in the North West Okrug, and since 2014 - the Russian Energy Agency (REA) under the Ministry of Energy. The TE Team agrees with the MTR, in which the BEENWR Project implementation is rated as Moderately Unsatisfactory (MU). Before the MTR the Executing Agency (and Project Team as well) couldn’t identify main/strategic objectives of the implementation and corresponding step-by-step approach. Because of this a significant time and funds were spent on activities, which not always were continued after the MTR (more details are presented in Chapter 3.3.3). The situation was improved after the REA became an Executive Agency. The newly appointed NPD personally conducted various meetings and discussions with all parties involved and formulated realistic implementation strategy, which among others included replacement of the Project Team with more experienced and energetic one, engagement of CTA and other consultants as it was appropriate, and establishment of closer cooperation with the UNDP PSO and UNDP RTA. The REA was effectively implementing its both roles, aimed at providing management inputs as well as ensuring the high level of country ownership. 


Tag: Effectiveness Efficiency Human and Financial resources Implementation Modality Project and Programme management Results-Based Management Country Government

29.

3.3 Project Results

In this chapter BEENWR Project results including direct project outputs, short- to medium-term outcomes, and longer-term impact including global environmental benefits, replication effects and other local effects are evaluated. For better understanding of the logic of evaluation the detailed milestones (actions) as well as list of publications and video materials prepared by the EERRB Project, are presented in below tables.

3.3.1 Overall results (attainment of objectives)

(*) In this Chapter, the achievements of expected results are evaluated in terms of attainment of overall objective as well as identified outcomes and outputs. For this the performance by components is analyzed by looking at: (i) general progress towards the established baseline level of the indicators; (ii) actual values of indicators by the end of the BEENWR Project vs. designed ones; (iii) evidences of relevance, effectiveness and efficiency of the results as well as how these evidences were documented. The summary of evaluation of attainment of Objective and Outcomes of the BEENWR Project are presented in Table 7.


Tag: Emission Reduction Energy Effectiveness Sustainability

30.

3.3.1 Overall results (attainment of objectives) (continuation)

In calculations AF Consult used the same values for emission factors of natural gas (is used for heat production) and electricity grid, namely 0.202 t CO2/MWh. It must be noted that in the ProDoc higher value of power grid emission factor is used, namely 0.725 t CO2/MWh. In the EBRD report “Dynamics of the development of carbon emission factors from production of Electricity in Russia” (2010), combined margin emission factor for North-West power grid for 2017 is predicted to equal to EFcm = 0.392 t CO2/MWh. This means that the emission reductions are calculated conservatively.

Emission reductions due to the application of EMIS in Pskow and Vologda oblasts are calculated on the basis of data provided by the regional energy managers. Energy savings were achieved due to the responsive actions when the EMIS was showing either the inconsistency between the supplied (by the utility) and consumed (metered by EMIS) heat energies (e.g. due to leakage) or overheating of the buildings. Energy savings due to the EMIS were estimated based on the 3-year data (covering the period of EMIS usage) with corresponding adjustments (by Heating Degree Days Index). It must be noted, that (i) the accuracy of such calculations cannot be very high; and (ii) application of static baseline is questionable (unlikely the energy consumption would remain as high as now, for the next 20 years). Nevertheless, the calculated emission reductions due to the EMIS can be verified because likely the number of public and residential buildings entered in EMIS, will be increased in the future and correspondingly, the post-project emission reductions due to the EMIS will be higher than calculated. Based on the abovementioned it is the TE Team opinion that the energy savings and GHG reductions achieved due to the implementation of demo projects – construction of new and renovation of existing buildings, as well as putting in operation of EMIS, are reliable and can be verified. The established target, direct reductions of 48,050 tCO2e as compared to the baseline, has been achieved by 50.23%.


Tag: Emission Reduction Energy Effectiveness Project and Programme management

31.

Objective. Target 2: Indirect reductions of 599,000 tCO2 emitted due to space heating in new and renovated buildings during their lifetime (assuming a 20-year lifetime) (continuation)

The Table 9 clearly shows that consequential emission reductions were underestimated by the BEENWR Project. Another reason for underestimation of consequential emission reductions is that replication effects of EMIS were not considered at all. On the other hand, scaling-up of use of EMIS in Pskov and Vologda oblasts is observed and considering that Russian Energy Agency plans extension of the EMIS use in other oblasts and that in the future not only public buildings can be entered into the system, but residential buildings as well, the replicability factor might reach high value. In 2016 an analysis of the current state of the building stock in the Russian Federation and its changes over the past 10 and 20 years in terms of energy efficiency was conducted, according to which:

The policy of revitalization activities in the field of energy efficiency in buildings has produced results - The share of buildings after capital repairs in Russia is significantly lower than in the EU, i.e. there is a need in more renovations - Buildings in Russia is the sector with the largest potential for energy savings. - In 2016, a "road map" was introduced to improve the energy efficiency of buildings. According to the roadmap, the share of apartment buildings, as well as public and administrative buildings of the highest energy efficiency class, should reach 10% by 2018; 20% by 2020; and 30% by 2030. The above-mentioned could be also considered in estimations of consequential emission reductions because the BEENWR Project facilitated the penetration of EE principles, designs and technologies in the building sector. If the consequential emission reductions are estimated in this way, they might be doubled-tripled and if so, even reach the established target. Finally, consequential emission reductions associated with the BEENWR Project’s contribution to the creation of regulatory framework (if any) were not assessed at all. Based on the abovementioned achievement of the Objective is rated as Moderately Satisfactory (MS).


Tag: Emission Reduction Energy Effectiveness Local Governance

32.

Outcome 1: Enabling environment and strengthened enforcement capacities for improved EE at the provincial and local levels (continuation)

In the pre-MTR period of implementation, the efforts were focused at improving legal framework at the federal level through the participation in various working groups and among them: - Scientific advisory council of the Federation Council Working Group for monitoring of implementation of Federal Law No. 261-FZ "On energy saving and energy efficiency improvements ..." (2009) - State Duma Energy Committee Work Group on energy savings and EE - Expert council of the National Energy Savings Union - Expert council of the Russian Energy Agency 

At those meetings suggestions on regulatory framework improving, adopting of new regulations and requirements in construction, maintenance and metering in buildings, were being discussed. Nevertheless, the BEENWR Project impact on improving Federal legislation during the first half of the implementation was negligible. Main activities under this output included:

Development of regional methodic instructions for application in Vologda Oblast in order to implement a new monitoring model on EE buildings at the regional level. - Analysis of the best practices of legal and regulatory frameworks on energy saving and energy efficiency at regional and municipal levels. As a result, a plan for establishment of regional and local regulations in Pskov Oblast for 2012-2013, was developed. - Development of Regional methodical instructions for: (i) implementation of a new model of monitoring of building EE at the regional level; (ii) collection, analysis, and systematization of data on current energy expenditures; (iii) energy surveys in Pskov Oblast (108 apartment buildings and public houses in Ostrov and Nevel Districts). In implementation of these activities, two international experts in EE and development of EE municipal norms, were involved.The development of a GIS-based model for accounting of energy resources to enable data collection for calculation of economic effects from the implementation of EE measures in Pskov Oblast. - Preparation of a catalogue of EE solutions for reconstruction of typical buildings In Arkhangelsk - passed to the Department of Housing and Communal Services of Arkhangelsk Oblast


Tag: Energy Effectiveness Local Governance Project and Programme management Results-Based Management

33.

Outcome 1: Enabling environment and strengthened enforcement capacities for improved EE at the provincial and local levels (continuation)

Achievements - Conducted analysis of energy consumption in the Russian Federation (by entities) - - Analysis of energy saving potential in the Russian Federation (by subjects) - Analysis of the current state of the building stock in the Russian Federation (including residential buildings, apartment buildings) in terms of energy efficiency - Analysis of the legislation of the Russian Federation and the subjects of the Russian Federation in the field of energy conservation - Analysis of existing incentives/mechanisms aimed at attraction of non-budgetary investments (including loan financing) for the implementation of EE projects - Recommendations for attracting investments for the implementation of EE measures (projects) in buildings (construction, renovation) by entities financed from the budgets (federal, regional, municipal) and at the expense of owners These achievements were welcomed by the Ministry of Construction, Housing and Utilities of the Russian Federation. Nevertheless, it must be noted that they are not fully relevant and adequate to the targets

The greatest progress has been observed in implementation of activities aimed at adoption and operationalization of energy management and information system (EMIS). The introduction of EMIS in Arkhangelsk oblast was included in the initial project design as an Output 3.3. The BEENWR Project was carrying out studies and developing software in 2012-2013 but after two years of the BEENWR Project implementation, works were not completed, and as stated in Chapter 3.2.1 above, the PSC took decision to opt another system, namely the EMIS developed by the UNDP/GEF project in Croatia. The following results were achieved in this direction: - Installation and deployment of the EMIS system (a server with software) in Pskov oblast (at Pskov Communal Systems): o Development of a list of public buildings of the Pskov oblast (485 buildings). o Primary technical inventory of buildings o The basic energy profiles of 28 buildings: the required (standardized) specific energy consumptions calculated, the analysis of the obtained data on consumption of fuel and energy resources o Proposals for updating EMIS in relation to the conditions of Russia o Detailed o Detailed information on demo buildings entered o Training conducted  


Tag: Energy Effectiveness Local Governance

34.

Outcome 2: Improved local capacities to leverage and manage investments into EE

The BEENWR Project successfully has implemented activities under the Component 2 and expected Outcome 2 has been achieved. First of all, it must be noted that after the revision of LogFrame, the Outcome 2 became more clear and focused, compared with the original one. The revised Component 2 consists of only one output with targets, which are easy to monitor. The successful implementation of activities under the Component 2 was greatly conditioned by the fact that CTA on educational component was engaged in the implementation from the very early stages. The following has been achieved:

- Creation and implementation of the system of continuous education in the field of EE o Implementation of the module, for the elementary school program on electricity saving. For pupils of primary grades 3000 copies of workbook "How to save electricity in the house" as well as a manual for the teacher was published. Published 3,500 copies of workbook "Sustainable energy use" (grades 5-9), methodical materials for project activities (grade 10) o Introduction of the optional module on the topic of industrial and household energy efficiency for secondary schools and institutions of secondary vocational education o A set of manuals "Fundamentals of energy saving and energy efficiency" for secondary vocational education and higher education (bachelor's degree). o The complex of work programs "Energy management" in the direction "Management" (23 programs) was developed - Creation of the scientific-educational energy management laboratory at the Educational and Scientific Center for Energy Innovation of the Nordic (Arctic) Federal Lomonosow University (NARFU). - Effective work of the Interregional Center for Vocational Training in Energy Saving (established at NARFU) and its Regional branches: o Vologda - based in Vologda State University o Pskov – based in Pskov State University o St. Petersburg – based in St. Petersburg State University of Industrial Technologies and Design The Council of the Interregional Center for Vocational Training and Retraining of Personnel on Energy Conservation from the representatives of universities is working. 4 meetings were held. Representatives of the center participated in the development of training programs. Training started under the distance learning system. 


Tag: Energy Effectiveness Local Governance Project and Programme management

35.

Outcome 3: Reduction of GHG emissions demonstrated: 45-76% reduction in energy consumption in construction and maintenance sectors; 10-20% reduction in energy losses in energy networks

Achievement of this Outcome was one of the main challenges of the BEENWR Project. In total 11 projects demonstrating EE design and technologies were planned in the ProDoc including construction of 4 cottages and 1 residential house in Vologda and renovation of 6 residential buildings in Arkhangelsk. Unfortunately, none of them have been implemented despite the significant resources allocated (especially in Arkhangelsk) before the adaptive management applied and new demonstration projects were chosen. 

Before the MTR the detailed design documents of demo buildings in Vologda municipality had been finalized and reviewed by an international expert. However, it appeared that despite the signing of an agreement with Vologda Oblast Administration on construction of three apartment buildings with total footage of 21,500 m 2 , the Administration was not able to finance and construct the buildings by covering their baseline costs. To cope with the situation, the Project Team “considered an option to abandon the previous site and design and restart the work by a tender requesting offers from private construction companies for the integration of specific EE measures and technologies into the construction and thereby avoid the risk of investing more resources into the design of a building that might never be constructed” 22 . Detailed design documents for renovation of 6 demo buildings in Arkhangelsk also were finalized before the MTR. However, no practical steps were undertaken to secure financing. Therefore, based on the recommendations of MTR and “Recommendations for Updated Project Strategy” the targets for Outcome 3 were revised in the following way:


Tag: Emission Reduction Energy Effectiveness Results-Based Management

36.

Outcome 3: Reduction of GHG emissions demonstrated: 45-76% reduction in energy consumption in construction and maintenance sectors; 10-20% reduction in energy losses in energy networks(continuation)

Construction of demo buildings

A) New construction Initially, due to the technical constraints (?) the construction site for the implementation of a demoproject in Vologda was changed. The designs were revised and supplemented with recommendations of the international experts. The designs envisaged 50% reduction of energy consumption and 15% decrease in network losses. However, because the financing of those demo projects wasn’t secured, the BEENWR Project was looking for other options. Since 2013, the Russian Federation is implementing a Resettlement Program from obsolete and dilapidated housing. The program is financed by the State Corporative Fund for the Reform of Housing and Communal Services as well as the regional budgetary funds. Subject to the allocated financing, as a rule, a set of mandatory energy-saving measures is limited to the minimum requirements for thermal protection. About 40 buildings, included in the Resettlement Program, were pre-selected in Pskov, Arkhangelsk, and Vologda Oblasts. One of the main criteria for selection was the completion of construction works latest in early 2016 in order the perform at least, one-year monitoring on energy performance and GHG reductions during 2016-2017. In the end, two buildings, presented on Figure 1, were selected for the implementation in Pskov and Novgorod Oblasts. Cooperation Agreement with the authorized representatives of those oblasts were signed in 2015.


Tag: Energy Effectiveness Sustainability Results-Based Management

37.

Outcome 3: Reduction of GHG emissions demonstrated: 45-76% reduction in energy consumption in construction and maintenance sectors; 10-20% reduction in energy losses in energy networks(continuation)

B) Renovation: Similarly to the new constructions, renovation demo sites also were changed for the same reason of non-financing. At the first stage, based on the assessment of availability of financing, the list of potential demo projects in Arkhangelsk oblast (ProDoc considered implementation of demo retrofits only in Arkhangelsk oblast) has been limited by municipal buildings. Then potential demo renovation projects were identified also in other oblasts. The feasibility studies were carried out for four municipal objects (one in Arkhangelsk and three in Pskov oblasts) were short-listed for demo renovation (EE retrofit) after detailed screening. Later the demo projects in Arkhangelsk oblast became unfeasible to implement (according to PIR 2017 the Arkhangelsk Oblast could not guarantee the financing of heat insulation of building sides that was the most expensive part of the retrofit design). Finally, two demo retrofits were implemented in the kindergarten "Raduga" in Ostrov town of Pskov Oblast and Vologda Health Clinic No. 3; they are presented on Figure 2. The selected buildings were connected to the Energy Management Information Systems for future analysis of energy consumption. In short, the BEENWR Project couldn’t succeed with the demo projects in Arkhangelsk, main pilot oblast for renovations. International consulting company AF Consult has estimated the energy savings in the implemented demo buildings. Kindergarten Raduga (total costs of EE measures 8,478,233 RUB or about USD 150,000; payback period: 8 years): the following additional EE measures were implemented: - Replacement of wooden windows with EE double-glazed windows - Installing supply valves - Warming of extensions to the main building - Restoring ITP - Insulation of the main building facades - Insulation of the roofing of extensions - Insulation of the attic floor of the main building - Insulation of the basement of the building - Installation of EE lighting devices 


Tag: Energy Effectiveness Project and Programme management Results-Based Management

38.

3.3.2 Relevance (*)

Relevance of the problem addressed by the BEENWR Project is already demonstrated above in introduction to Chapter 2 and Chapter 3.1. During the TE mission the Consultants obtained evidence that achieved results are also relevant to the priorities of both, the Government of Russian Federation and UNDP. Interviewed stakeholders underlined that the BEENWR Project was highly relevant to the country. In fact, one of the changes attributed to the BEENWR Project was the treatment of energy efficiency more broadly as a “hot topic” at the highest levels of government. Achieved results, especiallyimprovements in energy management, are fully relevant to the country’s climate change strategy priorities. The project has also been highly relevant to UNDP activities in Russia. Based on the abovementioned the Relevance is rated as Relevant (R).


Tag: Energy Effectiveness Relevance Project and Programme management Results-Based Management

39.

Efficiency

The BEENWR Project leveraged necessary financial resources and support (from BEENWR Project Partners) for implementation of planned activities. From one hand, the efficiency of the financial management of the BEENWR Project was evidenced by its ability to meet all of the procurement needs not only for initially planned activities (original ProDoc) but also additional ones within the original budget (GEF resources were used as planned). The breakdown on the spending of the GEF resources are presented in Table 10.

The analysis of Table 10 shows that in the first half of the project (2011-2014), the significant funds were inefficiently spent. For instance, USD 600,000 was spent for Outcome 1 and USD 880,000 for Outcome 3 (mostly for the preparation of technical reports, which haven’t been used during the implementation of demo projects). It must be noted that in the second half of the implementation 65% of the total budget was spent with less (compared with the first half) management costs. This clearly indicates on the inefficiency during the first half of the BEENWR Project. On the other hand, the BEENWR Project has been implemented within the initially planned budget. This shows that decision on extension of the duration of BEENWR Project without cost extension, was appropriate. However, not all the planned activities were completed (e.g. development of EMIS in Arkhangelsk oblast) despite the resources spent by the BEENWR Project. 


Tag: Energy Efficiency Human and Financial resources Operational Efficiency Policies & Procedures Procurement Project and Programme management

40.

3.3.4 Country ownership: Country Ownership was modest during the first half of the project implementation, while was much stronger during the second half, when the Russian Energy Agency took over. In the ProDoc a main role for the implementation was given to the Implementing Partner. Although REA was successful as an Executing Agency of the BEENWR Project, the state policy in the building sector is the responsibility of the Ministry of Construction, Housing and Utilities. Therefore, the success of the BEENWR Project was depending on support from this ministry. Country ownership for this BEENWR Project was largely dependent on whether or not the project concept was in line with country priorities. Country ownership was actually confirmed by adoption of more stringent building codes (in 2012), support in obtaining permits for implementation of demo projects, etc. 

3.3.5 Mainstreaming: The BEENWR Project is successfully mainstreaming other UNDP priorities. In particular: o The BEENWR Project helped in job creation (Energy managers were appointed, EE measures were implemented by the local contractors by using local materials) o The policy framework has been improved o BEENWR Project delivered education and raised capacity of aspiring and practicing professionals, as well as decision makers, with regard to EE improvement in the building sector o Impact on environment has been released (less GHG, less air pollutants due to the energy savings) o Gender issues - while gender issues were not taken directly into account in ProDoc, BEENWR Project staffing was balanced; trainings involved representative numbers of women and men


Tag: Sustainability Gender Mainstreaming Integration Ownership Project and Programme management Results-Based Management

41.

3.3.6 Sustainability

Socio-economic risks Question: Are there social or political risks that may threaten the sustainability of project outcomes? What is the risk for instance that the level of stakeholder ownership (including ownership by governments and other key stakeholders) will be insufficient to allow for the project outcomes/benefits to be sustained? Answer: The social risk is identified neither by the BEENWR Project nor the Consultant. Only the political risk identified in the beginning of the BEENWR Project, was related to energy price subsidies in residential and public sectors, leading to an insufficient incentive for efficient products. This risk is not critical at present. Question: Do the various key stakeholders see that it is in their interest that the project benefits continue to flow? Is there sufficient public/stakeholder awareness in support of the project’s long-term objectives? Answer: Certainly yes, stakeholders are interested in EE in public and residential sectors. Based on the above-mentioned the Socio-economic Risks are negligible and the sustainability is rated as Likely (L)


Tag: Energy Environmental impact assessment Effectiveness Efficiency Sustainability Project and Programme management Inclusive economic growth

42.

3.3.7 Impact

The BEENWR project has made certain advances in promoting EE in the public and residential building sectors of the North West of Russia, especially considering the starting point and thebaseline scenario, in which EE was minimally reflected in national policy, investment, educational curricula, and design practice. Some outputs of the BEENWR Project were achieved in Russia for the first time – the first regional energy managers appointed; EMIS put in operation in Pskov and Vologda oblasts. These outcomes along with created local capacity created a foundation for real changes in practice in the country and the most important change is the increased national-level and agency-level ownership of energy efficiency as an issue. The BEENWR Project managed to change people’s thinking and perception of energy savings. 


Tag: Energy Impact Human and Financial resources Project and Programme management

Recommendations
1

4.1 Corrective actions for the design, implementation, monitoring and evaluation of the project

Design No Corrective Action Request (CAR) is raised 

Monitoring and Evaluation CAR 1: Develop a Final Project Report with lessons learned One of the target to be achieved under the Outcome 3 is that “Best practices and lessons learned (should be) shared across the NW federal region”. Neither the final report nor lessons learned was provided to the TE team. Therefore, the project manager is requested to develop, prepare and release a final report including lessons learned study, which outlines the main lessons learned from this project.

4.2 Actions to follow up or reinforce initial benefits from the project

R1: To update the estimation of GHG reductions due to the application of the EMIS in Pskov and Vologda Oblasts.

In November 2017, international company AF Consult revised GHG emissions reduction calculations, and presented updated figures to the TE Team. However, new data on energy consumptions in buildings, which might be entered after November 2017, would lead to additional GHG reductions, because it is obvious that as many buildings are entered as higher reduction is achieved due to the management of the energy consumption in the buildings. 

2

4.3 Proposals for future directions underlining main objectives

R2. To launch the process of certification of EMIS in accordance with the existing procedures and to launch EMIS for the entire Russian Federation, coordinated by the Russian Energy Agency, including securing funds from the Federal Budget for EMIS scale up and roll out.

One of the possibilities for scaling up EMIS could be a development of a proposal for the UNDP Russia Trust Fund to implement and roll out EMIS in countries in Central Asia and in Belarus.

1. Recommendation:

4.1 Corrective actions for the design, implementation, monitoring and evaluation of the project

Design No Corrective Action Request (CAR) is raised 

Monitoring and Evaluation CAR 1: Develop a Final Project Report with lessons learned One of the target to be achieved under the Outcome 3 is that “Best practices and lessons learned (should be) shared across the NW federal region”. Neither the final report nor lessons learned was provided to the TE team. Therefore, the project manager is requested to develop, prepare and release a final report including lessons learned study, which outlines the main lessons learned from this project.

4.2 Actions to follow up or reinforce initial benefits from the project

R1: To update the estimation of GHG reductions due to the application of the EMIS in Pskov and Vologda Oblasts.

In November 2017, international company AF Consult revised GHG emissions reduction calculations, and presented updated figures to the TE Team. However, new data on energy consumptions in buildings, which might be entered after November 2017, would lead to additional GHG reductions, because it is obvious that as many buildings are entered as higher reduction is achieved due to the management of the energy consumption in the buildings. 

Management Response: [Added: 2018/05/11] [Last Updated: 2021/01/21]

The project management team agrees that the calculations reported in the PIR 2016-2017 were not final because by July 2017 works on the establishment of EMIS and pilots were not completed. However, in November 2017 the revised GHG emissions reduction calculations were presented to the Final Evaluation Team. The Project reported the achievement of around 60% from the initial target (including the impact from EMIS) and is planning to exceed this figure due to EMIS operation in 2018-2019. During the Final Steering Committee meeting in October 2017 both Vologda and Pskov Regional Administrations officially confirmed that EMIS operation is included into the regional budget financing and will be supported by relevant departments (Regional EE Center in Vologda and Pskov Utilities Company in Pskov)

Key Actions:

Key Action Responsible DueDate Status Comments Documents
For the project manager in partnership with the Russian Energy Agency and Regional Administrations to keep track of energy savings due to EMIS operation for its further promotion in other parts of the country
[Added: 2018/05/11] [Last Updated: 2019/01/14]
Project Manager 2018/04 Completed History
2. Recommendation:

4.3 Proposals for future directions underlining main objectives

R2. To launch the process of certification of EMIS in accordance with the existing procedures and to launch EMIS for the entire Russian Federation, coordinated by the Russian Energy Agency, including securing funds from the Federal Budget for EMIS scale up and roll out.

One of the possibilities for scaling up EMIS could be a development of a proposal for the UNDP Russia Trust Fund to implement and roll out EMIS in countries in Central Asia and in Belarus.

Management Response: [Added: 2018/05/11] [Last Updated: 2021/01/21]

The project management team fully agrees with this recommendation. The Ministry of Economic Development of the Russian Federation (MED) and the Russian Energy Agency share this opinion too. During the Final Steering Committee meeting in October 2017 a representative of MED confirmed that the Project initiatives on using EMIS as a state energy monitoring and verification tool are interesting for the Russian Federation. The new national EE Plan includes a special section regarding further dissemination of Pskov and Vologda Energy Management experience across Russia. In October 2017 the second annual Regional Energy Managers meeting was held where the representatives of 12 Russian regions declared their readiness to implement the Project experience in EMIS operation.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
For the project manager in partnership with the Russian Energy Agency to promote EMIS for its integration into the existing State Information System GIS, as a mandatory part of the national energy reporting.
[Added: 2018/05/11] [Last Updated: 2019/01/14]
Project Manager 2018/12 Completed The recommendation was being implemented until the closure of the project and the UNDP Project Support Office in the Russian Federation. It is expected that the Russian Energy Agency will continue to promote EMIS. History

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org