INDEPENDENT FINAL EVALUATION: PROJECT ON “DEVELOPING AN INTEGRATED APPROACH TO A STEPPED-UP ENERGY SAVING PROGRAMME IN BELARUS”

Report Cover Image
Evaluation Plan:
2016-2020, Belarus
Evaluation Type:
Final Project
Planned End Date:
04/2017
Completion Date:
08/2017
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
10,000

Share

Document Type Language Size Status Downloads
Download document Final Evaluation report EE in Schools.docx report English 2619.71 KB Posted 623
Download document Final Evaluation report UNDP-EU Belarus EE.docx report English 2665.92 KB Posted 655
Title INDEPENDENT FINAL EVALUATION: PROJECT ON “DEVELOPING AN INTEGRATED APPROACH TO A STEPPED-UP ENERGY SAVING PROGRAMME IN BELARUS”
Atlas Project Number: 00067546
Evaluation Plan: 2016-2020, Belarus
Evaluation Type: Final Project
Status: Completed
Completion Date: 08/2017
Planned End Date: 04/2017
Management Response: Yes
Focus Area:
  • 1. Environment & Sustainable Development
  • 2. Cross-cutting Development Issue
  • 3. Others
Corporate Outcome and Output (UNDP Strategic Plan 2014-2017)
  • 1. Output 1.5. Inclusive and sustainable solutions adopted to achieve increased energy efficiency and universal modern energy access (especially off-grid sources of renewable energy)
SDG Target
  • 4.7 By 2030, ensure that all learners acquire the knowledge and skills needed to promote sustainable development, including, among others, through education for sustainable development and sustainable lifestyles, human rights, gender equality, promotion of a culture of peace and non-violence, global citizenship and appreciation of cultural diversity and of culture’s contribution to sustainable development
  • 7.2 By 2030, increase substantially the share of renewable energy in the global energy mix
  • 7.3 By 2030, double the global rate of improvement in energy efficiency
Evaluation Budget(US $): 10,000
Source of Funding: Project Budget
Evaluation Expenditure(US $): 10,000
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
GEF Evaluation: No
Key Stakeholders:
Countries: BELARUS, REPUBLIC OF
Lessons
1.

The project delivered almost all of its planned deliverables, and even more for some of the activities, e.g. training more people and completing the pilots not in 3 but 4 educational institutions. This is commendable given the large scope and complexity of the project compared to the short project duration and lean staffing, and the fact that it suffered delays caused by the lengthy process of hiring the staff. The pilots in their technical part were ahead of the peers in the country leading to significant interest in the innovation technologies that were piloted. Whether large scale replication will be forthcoming or not will depend on many external factors.

The educational institutions had the chance to develop creative approaches in teaching energy saving especially to the lower and middle grades. The quality of the information products, including teaching aids is appreciated by the stakeholders, but – and this is a lesson learnt- in similar projects in the future it is important to secure more active participation of the Ministry of Education. The sheer quantity and variety of the information products developed under this project, as a sum of both streams (produced by the pilots and the project team) was huge; while this has advantages, the project could have had a stronger coordinating role and interface with the Ministry of Education. This is another lesson learnt, and particularly important when there are teaching aids/tools involved which require approval front eh Ministry of Education. And finally, in the part concerning the information campaign, there should have been mechanisms to capture the feedback of the recipients of the products (also a lesson learnt): timely analysis and learning from this analysis and acting upon this learning would have increased the effectiveness of the information component

The PABs proved to be effective mechanisms for public monitoring of the pilot projects at the local level, but expecting that these would have an influence on the planning/investment at the local government level in EE was perhaps overly optimistic. The only activity where the project has hardly succeeded is related to the development of local EE strategies at the pilot locations, whereby, in a nutshell there was no apparent interest on their behalf in having such plans; the fact that there was no budget allocated for that activity did not help.

The survey among the 23 schools, including schoolchildren, parents and teachers shows that while there is some increase in the awareness (more in the pilot schools) compared to the baseline, some are more effective than the others (and again this is more notable in the pilot schools). Awareness was raised more in relation to the potential of using solar energy than on importance of saving energy, and for the latter more in theory and not in the application to, say own household expenses. This shows that awareness raising must be well targeted and timely to affect the level of awareness and change behaviour (e.g. coincide with reforms). Another lesson is that apart from and in parallel to implementing awareness raising activities there is a need to institute mechanisms to assess the effectiveness and create feedback loops.


Findings
1.

3. 1 Project formulation: relevance

3.1.1.Relevance of the Project focus

The buildings’ sector has a large potential of energy savings for Belarus: it consumes 40 percent of heat used in the country, and as such represents an important potential source of greenhouse gas (GHG) emission reductions through fuel savings. More than 80 percent of the country’s residential stock, and about 95 percent of the public building stock was built before 1996. Building thermal protection standards were significantly strengthened in 1993 and updated in 2010. Pre-1996 buildings consume, on average, nearly twice as much energy per square meter as buildings constructed in the last four years, and this applies to the educational institutions also (see Figure 3). More than 90 percent of public buildings in Belarus were built before 1996, including 95 percent of kindergartens and secondary schools: 2,479 (about 94 percent) secondary schools and 2,236 kindergartens (about 96 percent) were built before 1996.


Tag: Emission Reduction Energy Rule of law Technology Institutional Strengthening

2.

The project is also relevant given the ongoing energy tariff reforms aimed at removing of subsidies and hence expected increases of the tariffs in the residential sector and removing cross subsidies in the public and industrial sectors, which requires better awareness of the benefits of higher EE by the population. Linked to this is the relevance of choosing schools as targets, given that they act as agents of change with children often bringing the knowledge they gain to the families, i.e. spreading the knowledge. To sum up (and also according to the previous experience in similar projects), institutional buildings, and in particular schools, kindergartens, boarding schools and vocational-technical schools are natural points of entry for EE activities at a local level for the following reasons:

  • high visibility;
  • a natural setting for educating students and their families;
  • vast geographic coverage;
  • significant source of potential energy savings (and associated savings in operational costs);
  • the use of design prototypes for institutional buildings with a potential for replication in renovation initiatives; and
  • institutional buildings’ retrofits producing important improvements in the learning environment (comfort, visibility) that contributes to enhancing the quality of education

Tag: Energy Relevance MDGs Partnership International Financial Institutions UN Agencies SDG Integration

3.

3.1.2 Implementation approach

Barriers to EE in Public Buildings

Barriers to EE in public buildings in Belarus fall into the following categories: legal, regulatory and institutional, incentive-related, and financing. The barriers are described in more detail below

  • Legal Regulatory, and Institutional
    • Inability to reallocate expenditure between line items. This means that the amount set aside in a budgetary organization’s budget to pay energy bills (measured, for example, in cost per litre, cubic meter, ton or kWh of purchased energy) cannot be used for other purposes (for example, a reserve fund to pay for investments in EE).
    • Restrictions on multi-year obligations. Article 138 of the Budget Code prohibits any commitments beyond the approved annual appropriations. The restriction on multiyear obligations stifle the evolution of organizations like ESCOs, which could help public organizations save on energy consumption. To get around this restriction, administrators of budget funds typically try to ensure that their medium- to long-term projects are supported by an act of the Council of Ministers or the Presidential Decree.
    • Highly fragmented responsibilities for certain sectors making coordination of bundled procurement and investment in EE improvements in the public sector more complex. For example, expenditures of post graduate schools are financed by the Republic budget while expenditures for lower level educational institutions such as pre-schools might be financed by oblast and base tier budgets.

Tag: Rule of law Human and Financial resources

4.

Incentive-related

  • Inability to retain savings on energy. Budgetary organizations in Belarus use incremental, line-item budgeting that tend to limit budgetary organizations’ incentives to save energy (leading it to a “use it or lose it” mentality), so that their budgets are not reduced in the next planning period.
  • “Mutual settlement” and other inter-governmental transfers. The system of transfers between levels of government creates little incentive for subnational governments (SNGs) to reduce operating expenditures and generate permanent fiscal savings. Approximately 35 percent of the budget for SNGs is sourced by transfers from the central government, three-fourths of which being general purpose grants.  

Tag: Energy Challenges Relevance

5.
  • Financing barriers
    • Article 79 of the Budget Code forbids public entities from borrowing in any form. SNGs may issue securities on the domestic market or may take intergovernmental loans to implement investment projects for example. Public enterprises may borrow from commercial banks, using their assets as collateral, but such permissions are granted by the Government (owner of public enterprise) only on a limited basis.

The project pursued the promotion of energy savings in schools in participating communities with three- pronged approach, described in Figure 5.

ProDoc


Tag: Human and Financial resources

6.

a)   Public awareness and training in energy efficiency for local stakeholders in target districts.

The project has a multifaced information component, including: thematic presentations for local authorities and communities; Quarterly information newsletter, thematic project web-site; Training of trainers (ToT) on EE issues; aim at introducing Energy Efficiency Extracurricular Subject in educational institutions; Information stands in pilot buildings; Information campaign, etc. This allowed to cover different segments of the society and tackle the task of increased awareness from various angles and with varying level of complexity.


Tag: Energy Communication Awareness raising

7.

b)   Energy efficiency pilot projects implemented via a competitive grants scheme

The project included:

  •  New approaches to increase the impact of standard retrofit measures. Measures, such as the replacement of old windows with new energy-efficient ones, thermal insulation for exterior walls and roofs, automatic controls for heat, and efficient lighting systems with controlled luminance have previously been used in institutional buildings in Belarus. However, this project implements these measures according to the new energy-saving regulations adopted in 2010 in the construction sector, which require much higher thermal resistance in roofs, walls and windows compared to the previous ones;  

Tag: Energy Challenges Effectiveness Technology

8.

C) The use of Area Based Development (ABD) approach within the framework of sustainable energy management at the local level

According to the Description of Action of the EU-UNDP Financing Agreement “…for the purposes of this project ABD is defined as an approach to development, which implies targeting specific geographical areas in a country, characterized a particular complex development problem, through an integrated.  inclusive, participatory and flexible approach (UNDP, 2003)”. Specifically, the project was expected to:


Tag: Relevance Local Governance Knowledge management Policies & Procedures Programme/Project Design Capacity Building

9.

3.1.3 Stakeholder participation/ownership

The EED was part of the developing the project concept and this can be observed from the well drafted Component 2. However, in some aspects, especially in relation to some of the activities under Component 3 (Local EE strategies, as discussed) the design process required consultations at various level of the local government (region, oblast, city) to assess the level of interest before including this as part of the project proposal. Similarly, in the part of Component 1, a more active role of the Ministry of Education at the stage of the writing the proposal (as well as implementation, as will be discussed in Section 3.2.1) should have been assured to clarify the modes of participation and clarify the role.


Tag: Ownership

10.

3.1.4 Cost-effectiveness

The estimated economic performance, the low level of technological risk of the planned technical rehabilitation of the pilots and the fact that some of the main components were planned to be locally produced imply that cost effectiveness was treated as an important factor in the project design. Ideally concerns about the cost effectiveness of the awareness raising work under Component 1 should have prompted incorporation of measures that would have allowed to assess the effectiveness of various modes of work robustly. 

The project implementation Unit (PIU) had only 4 staff members. This has saved on administrative costs but arguably at the expense of efficiency and effectiveness to some – even if not large - extent. This is discussed more in Chapters 3.2 and 3.4). Similarly, the 3 years’ duration was overly optimistic given the ambitious goals of the project.


Tag: Effectiveness Efficiency

11.

3.1.5 Replication approach and sustainability design

The choice of the new technologies was informed by the desire to make them more likely to replicate. These were doomed to be easier to replicate given their estimated economic performance and them carrying low level of technological risk (important given the high visibility). In this way, the pilot buildings were meant to serve as important blueprints for other local governments and population, and provide a visible demonstration of the fuel-saving and money-savings results of the new higher standards. Piloting of other forms of EE technologies, e.g. heat recovery (a novelty, in essence), and the use of solar energy (also a relative novelty in Belarus) was supposed to trigger the interest and replication as well. To support the replication of the best practices, a series of in-country study tours and roundtable discussions for the EE pilot communities and potential stakeholders from other oblasts not involved in pilot renovations were planned (see the discussion in Section 3.3.1 and Recommendations). Thus, the design was adequate to trigger interest in replication, but hardly sufficient to bring about changes in the investment making policy in EE by the regional and local governments as was expected – perhaps too ambitiously- according to the Description of Action of the EU-UNDP Financing agreement, as discussed.


Tag: Technology Policies & Procedures Education Local Governance Knowledge management

12.

3.1.6 Results framework

The results framework (RF) of the project could have been better formulated. In particular,

a) there are many targets in the text of the ProDoc but not in the RF, and, moreover, there are concerns related to their formulation. This list is presented below, but also in the text of the report later:

  • 80 percent of trained educators will report of their willingness and readiness to facilitate inclusion of EE training into schools in the form of extracurricular or elective classes” for Activity 1.4. It is in the ProDoc page 10 para 1, but not in the RF;
  • By the end of the project, 100 percent of target project regions will establish school-based EE elective curriculum” for the Activity 1.5. It is an unclear what is meant under the word “Region”. It is in the ProDoc, page 10, para 3, but not in the RF;

Tag: Results-Based Management Awareness raising Capacity Building

13.

3.1.6 Results framework

The results framework (RF) of the project could have been better formulated. In particular,

b) Plus, the outcome level results are mentioned as tasks in the ProDoc (page 9, para 2) but they are not all featured in the RF. This applies to Outcome 3, whereby the expected outcome level result, i.e. “Active participation of local governments, educational institutions and local population in energy-saving pilot initiatives at the local level, along with demonstration of best practices and replication ensured” is hard to capture with quantifiable evidence, and the Indicators from the RF (No. of representatives of local stakeholders trained in ABD participatory practices; and No. of public hearings conducted in every target region/territory) do not capture it well. Additionally, these 3 “tasks” represent actually different level of results: while “To raise awareness and build capacity within communities to carry out energy saving measures at local level that improve energy efficiency and utilize renewable energy sources” is a good outcome level expected result (for Component 1), the expected result for Component 2, namely, “To establish pilot sites in local communities demonstrating application of innovative energy efficiency technologies in school buildings” is actually an output level result.


Tag: Local Governance Results-Based Management Capacity Building National Institutions

14.

3.1.7 Linkages between project and other interventions within the sector

Apart from the related activities of the EU and UNDP discussed earlier, the project was also complementary to activities of some other international organizations and financing institutions which were ongoing or happened before/or approximately at the same time as the current project. In particular:


Tag: Clean Energy Energy Environment Policy Partnership International Financial Institutions Awareness raising

15.

3.1.8 Management arrangements

The EED was charged with the responsibility to ensure the successful implementation of the project activities, sustainability of the achieved results, as well as reporting to the Belarusian state authorities. It was expected to closely cooperate with UNDP to ensure that all the project activities are planned and implemented in a manner appropriate to the project’s goals and objectives. The EED was expected to assign a senior official as a National Project Coordinator responsible for project implementation on behalf of the EED. The Project Organization Structure (presented below in Figure 6) includes the Project Steering Committee (PSC), Project Assurance, and the Project Team.


Tag: Coordination Oversight Project and Programme management Quality Assurance

16.

3.2 Results: attainment of outputs, outcomes and objectives

3.2.1 Outcome 1: Public awareness and training in energy efficiency in target districts.

78 participants took part in the Thematic presentations for local authorities and communities in October 2014 covering various aspects related to EE in buildings (including the Belarusian experience in EE, including national policies in this area). Thematic materials were developed, disseminated and placed on the project web-site There is no hard evidence to claim that at least 80 percent of the participants have improved their knowledge of all potential benefits of energy-saving technologies and the use of renewable energy - a target from the text of the ProDoc- as the feedback forms were collected but not systematically analysed. However, the interviews for this evaluation indicated satisfaction with the quality of the presentations, with a reflection that these helped them to increase their awareness related to EE measures in educational institutions.  

By the end of the project implementation, 10 issues of the information newsletter were published with 299 copies each and distributed to project stakeholders in target districts. 50 percent of the newsletter’s content presented by local stakeholders (local authorities, enterprises, local mass media, NGO, local population)- a target from the ProDoc - was taken as a guiding principle. No formal feedback mechanism was put in place (e.g. in the form of inserts, feedback box on the website, etc.) but the interviewed stakeholders for this evaluation commented positively on the usefulness of these in their work and, in general, for their awareness of EE.


Tag: Communication Knowledge management Awareness raising

17.

The Educational tools, which are covered by the list above deserve a more detailed discussion

For the additional educational programme (outside the formal curricula, i.e. for coteries, elective or extracurricular classes etc.) the requirements are as follows to ensure that the certain educational aids could be used:

  • The teachers could use their own materials at this classes but they should be approved by educational-methodical unit in frame of the District Educational Departments.
  • For kindergartens, all materials used in educational programme have to approved by Ministry of Education.

The educational aids under this project were produced both by the pilot educational institutions and by the project team:  

  • by the pilot educational institutions: In this case, the educational tools were sent for review to the respective Educational Departments of the Regional Executive Councils, and even if there is no formal letter of approval of the Ministry of Education, the process is sufficient;

Tag: Energy Knowledge management Education Awareness raising National Institutions

18.

Achievement of expected Outcome: increased awareness  

The analysis of the results of the Energy Efficiency Awareness Survey (EEAS) 2014/2016 indicate that the awareness raising activities carried out by the Project for the period 2014-2016 did contribute to an increased level of awareness in 23 educational institutions that took part in this survey, but that the increase varied across the topics covered. In particular:

  • The share of the respondents indicating that they possessed enough information on the topic of energy saving was slightly more in 2016 compared to 2014: 80 percent compared to 78 percent;
  • In 2016, at the High School No. 4 in Dziarzhynsk 14 percent more compared to 4 percent in the region answered positively to the question on “whether they think about the burden of the payments on energy bills”; in Hrodna kindergarten 20 percent more compared to 10 percent in the region;
  • In 2016, the share of those not being able to answer the question about the share of the payments on heating and electricity in the household budgets was by third less at the High School No. 4 in Dziarzhynsk compared to regional average; in Viciebsk State Vocational Technical College it was 4 percent compared to 6 percent of the region;
  • In 2016, at the High School No. 4 in Dziarzhynsk 26 percent more answered positively to the question on the sufficiency of information on energy saving, compared to the 3 percent increase in the region; and
  • The increased awareness about RES is more pronounced than in the case of differences observed for other indicators (listed above, although the difference is more visible in the level, quality and depth of answers to open questions) and this is much more the case for the pilot schools. At the high School No. 4 in Dziarzhynsk 47 percent more respondents answered positively about the opportunities of using solar energy at home, summer house, work, school compared to 4 percent in the region. For the Kindergarten No. 6 in Ashmiany this comparison was 21percent to 4 percent; for the Hrodna kindergarten – 35 percent to 4 percent, and for the Viciebsk college -15 percent more than in the region. Also, at all the pilots more respondents knew about the use of biomass and wind energy (difference was less than in the case of solar power).

The formal indicator from the RF, namely “percent of pilot schools acknowledge improved awareness of pupils and their parents on the benefits of rational use of energy and energy efficient measures” is met.


Tag: Education National Institutions

19.

3.2 Results: attainment of outputs, outcomes and objectives

3.2.2 Outcome 2: Energy efficiency pilot projects implemented via a competitive grants scheme.

Outputs:

Component 2 followed through the following steps:

1) Selection mechanism was developed. With Viciebsk, Hrodna and Minsk regions preselected, the following were the main principles of the selection mechanism, spelled out in the “Instructions to the applicants”:

    • Utilization of the heat of the vented air;
    • Use of solar energy for hot water supply/heating;
    • Thermal rehabilitation of building envelopes;
    • Introduction of energy efficient windows
    • Reconstruction of the roofs using energy-efficient heat-insulating materials;
    • Energy-efficient lighting;
    • Information and education;
    • Strengthening the capacity of local executive and administrative bodies;
    • Increasing the level of involvement of the local population and public organizations in activities related to energy conservation;
  • The funding requested for the implementation of the Technical part
    • determined based on the results of the energy audit of the educational institution (in its absence of it – based on calculations);
    • Component of the ABD - in the range from 1 to 50 thousand US dollars;
    • Minimum 40 percent contribution from local sources;
  • Other things being equal, preference was to be given to:
    • winners and prize winners of the Republican contest "Energomarathon" in 2008-2014;
    • those with actively functioning parents’ committees of the educational institutions (boards of trustees);
    • level of co-financing from other sources; and
    • social significance and importance to other institutions of education.

Tag: Energy Education Capacity Building

20.

3) Training on the procedure for receiving grants. 68 representatives of schools, lyceums, vocational schools (27 participants from 18 education entities from Hrodna region, 15 participants from 8 education entities from Viciebsk region, 26 participants from 18 education entities from Minsk region) were trained on the procedure for receiving grants within the project (November 2015). A full package of methodological materials necessary for potential applicants (local governments or the educational institutions’ authorities) were developed. Project experts provided guidance (group, individual and on-line) to the applicants and issued clarifications on the application process, were necessary and on request (1 workshop in each target community was organized).


Tag: Small Grants Programme Jobs and Livelihoods National Institutions

21.

5) Necessary design documentations were developed (a feasibility study, basic design, and detailed construction documents in accordance with local regulations) for the pilot projects.

6) Identification of co-funding opportunities. The Project Team supported the local governments in searching for additional co-funding sources for pilot initiatives, including in the form of preparation of necessary documents. It was expected that at least 40 percent of total pilot project costs will be covered by means mobilization of local resources in cash and in-kind (work, services, goods and etc.). This criterion was even surpassed in reality (see Table 10)

7) Implementation of EE increasing measures. The following EE measures were implemented at all four pilot educational institutions (see Figure 8):

  • Installation of solar collectors, which were expected to reduce the consumption of thermal energy for hot water supply in educational institutions by 50-70 percent. This technology has not yet become widespread in residential and administrative buildings in Belarus. Pilot educational institutions were to become demonstration sites for the promotion and dissemination of this technology and, as the interviews indicated, this expectation has materialized;

Installation of the ventilation systems with heat recovery expected to allow heat loss reduction during air exchange by 28-30 percent. The pilots were expected to demonstrate the economic efficiency of this technology. This technology is a novelty in Belarus and the evidence suggests that even the energy experts visit the sites to see this in operation (in Viciebsk in particular);

 


Tag: Energy Effectiveness Resource mobilization Local Governance National Institutions

22.

9) Training of the local authorities and pilot projects implementers in the target districts in both (a) the rules of grants administration and good practices in project management (1 workshop in each target community was organized) and (b) Regular project monitoring. Site owners and local authorities were made responsible for monitoring EE and energy savings at pilot sites as a contribution to project implementation. Site owners collected and recorded day to day measurements of heat and electricity consumption, inside and outside temperature, fixing data in written form in special journals, further used for analysis by the Project Team. Monitoring relied on heat and electricity meters. The project EE expert trained 1 staff member from each pilot institution to be in charge of this. During the project implementation, regular site visits were made to pilot projects sites to check on the operation of the equipment installed and to undertake additional measurements needed. Monitoring reports on the pilot projects were made available on the project web-site.

All 4 institutions had signed contracts with the suppliers as well as companies to provide services connected with maintenance and repair.


Tag: Local Governance Education Capacity Building National Institutions Energy

23.

Outcome Pilot sites demonstrating application of innovative energy efficiency technologies in social buildings are established.

Four pilot sites demonstrating the application of innovative EE technologies in educational institutions of different types are now established and operational. The monitoring results demonstrate savings in both heat energy and electricity (see Table 8). For electricity, the achieved savings are more or close to the expected levels. As for the heating the achieved savings are somewhat less than expected, mostly due to pre-existing technical reasons, e.g. the heating systems in all four buildings not being well balanced and hence the need to overheat some of the rooms: based on the results of heat consumption monitoring recommendations on heating system balancing were provided to the management of these institutions.  Also, it should be noted however, that the expected heat saving includes savings due to the use of heat recovery ventilation: taking into account, that actually old ventilation system was not used, this amount of saving was only expected, but not actual.


Tag: Energy Innovation Technology

24.

 

3.2.3.Outcome 3: The use of an area-based development (ABD) approach within the framework of sustainable energy management at the local level

Outputs

UNDP and the Regional Executive Committees signed Memorandums of Understanding (MoU) at the start of the projects. These MOUs were supposed to witness the willingness and readiness to cooperate and provide mutual support within the framework of EU-UNDP project implementation. MoUs were rather generic: had they specified the roles with more level of detail and covering all activities, the lack of clarity in relation to the roles pertaining to the local EE strategies would have been clarified and sooner in the process. Also, Partnership Agreements were signed with the pilot institutions (2 each) spelling out UNDP and implementing entity roles during the pilot initiatives implementation process using the ABD approach as well as covering the responsibilities under the National Implementation Modality (NIM).


Tag: Implementation Modality Results-Based Management

25.

Creation and facilitation of EE Public Advisory Boards (PABs) operating under local authorities. The 4 public meetings aimed at raising awareness of the Project’s goal, objectives, activities and expected results were also used as platforms to promote the creation of the PABs: since there were representatives of the same organizations present, the project had rightly used the opportunity. Participants were invited to participate actively in the relevant project activities to be organized at the pilot sites, including PAB meetings and PR events. According to the Progress reports of the project, it took time, however, to agree with the pilot regional authorities on the responsible party (institution) to host the respective PABs: the ProDoc specifies 2 options, at the level of the Regional Executive Committees or at the level of the pilot institutions. It was finally agreed that PABs will be established on the basis of the pilot educational institutions (kindergartens in Ashmiany and Hrodna, college – in Viciebsk and secondary school – in Dziarzhynsk). Additional discussions were required to reach an agreement among all the stakeholders and this led to the delay in the formation of the PABs and lesser number of meetings (quarterly). By the time of the evaluation the PABs were present with 15 members at each pilot (13 in Ashmiany). The Project provided technical assistance in the process of formation and functioning of the PABs as well as ensured the availability of relevant information regarding the project implementation process, including with:


Tag: Civic Engagement Local Governance Oversight Education Awareness raising Technical Support Civil Societies and NGOs National Institutions

26.

Assistance in developing local EE strategies. Working meetings were supposed to be held with the representatives of local executive and administrative authorities and stakeholder groups with the (a) analysis of the needs of the target districts in the EE in education sphere and (b) determination of the main priorities of the districts and education sphere for reducing energy consumption in education establishments. This was supposed to be followed up next by the preparation of recommendations and proposals for the development of the local EE strategies in education sphere.

This is perhaps the one subcomponent of the project which was the least successful and the reasons are multiple:

  • Neither the ProDoc nor the Description of Action of the EU-UNDP Financing Agreement specified the level of the government where there was a proven need and interest in such strategies (this was insufficiently analysed during the drafting of these document, as was argued in the Section 3.1.2). Actively probing into this was left until quite late by the project team with the argument that until the pilots were identified it was not possible to know which would be the counterpart level, i.e. regional or city administration. Arguably, such probing could have been done much earlier, right at the start at both levels, had the project had sufficient resources and funding specifically for that;
  •  

Tag: Energy Challenges Results-Based Management

27.

Outcome: Active participation of local governments, educational institutions and local population in energy-saving pilot initiatives at the local level, along with demonstration of best practices and replication ensured

The outcome specified in the Project Document (as one of the 3 Tasks on page 9), namely “Active participation of local governments, educational institutions and local population in energy-saving pilot initiatives at the local level, along with demonstration of best practices and replication ensured” is hard to capture with quantifiable but given the scope of the activities of the project, and the due to the existence of the PAGs, it could be said that this was achieved.

Against the baseline ofLocal population is not actively involved into sustainable local development planning”, the RF of the ProDoc had 2 indicators (NB: these do not capture the specified Outcome).

  • No. of representatives of local stakeholders trained in ABD participatory practices. 536 representatives of local stakeholders trained in ABD participatory practices in Viciebsk, Hrodna and Minsk region; and
  • No. of public hearings conducted in every target region/territory. Town Hall Meetings are conducted in target districts on a regular basis. 8 project public meetings were held in target districts of Viciebsk, Hrodna and Minsk regions, with 373 people participating, including the parents and the representatives of NGOs and local community. These meetings were also used as platforms for discussing the progress of pilot initiatives and finding creative solutions to complex challenges.

It was expected that the project will also work with local governments to improve their capacity to undertake energy saving investments: this is not in the ProDoc, but in the Description of Action of the EU-UNDP Financing Agreement: this would be hard to claim, given the status and level of interest in the local EE strategies.

 


Tag: Civil Societies and NGOs National Institutions

28.

3.3 Prospects for sustainability and replication

3.3.1.Rehabilitation with EE technologies

Sustainability

The high co-financing figures for the pilots (see Table 11) indicate strong local ownership and high likelihood that the provided infrastructure will be looked after. This is also supported by the fact that the 4 institutions have contracts with the supplier companies in case there is a need in spare parts as well as contracts for maintenance. One person from each institution was trained by the Project Team in carrying out routine daily monitoring function.


Tag: Sustainability Ownership National Institutions

29.

Replication

The pilots serve as examples of an integrated approach to EE rehabilitation of public buildings: a need to shift to this approach rather than category- specific renovations (e.g. replacing the windows in certain number of buildings, then lighting, etc.) has been recognized by the EED.  There is evidence that the interest on behalf of educational institutions in the regions and beyond is high in similar rehabilitation. The pilots also demonstrated that there is interest in co-funding such initiatives resulting in “pooled” funding model: with central government (EED), local government, enterprises- sponsors, the beneficiaries themselves (e.g. parents). In other words, securing half of the necessary funding from a dedicated funding (EU in this case, but it could be WB, EBRD, etc.,) may well attract other funders contributing to funding of the remaining amount. But the barriers in the form of the financial mechanisms, as well as of legislative and institutional nature that were discussed in Section 3.1.2  are significant, and therefore the potential for large scale replication with government sources is not high. There is one more barrier characteristic for the whole country: lack of technical specialists able to provide routine technical maintenance services, but there are already ideas for 2 regions on the establishment of specialized colleges.


Tag: Efficiency Sustainability Human and Financial resources Reconstruction Technology

30.

3.3.2. EE as an extracurricular subject

While there is no certainty, it is possible that at least for the near future the 4 educational institutions might continue in some form the creative educational activities involving other schools in the regions- provided that there is an interest on behalf of the Educational Departments of the Regional Executive Committees and financial reward for extra activities (which was present in the current project).

As for teaching EE as a formal extracurricular subject, there is a good chance that the Manual for teaching EE in the school curricula will be used in the educational institutions since it has been already approved by the Ministry of Education. This was developed with the help of the experts from the Association for the Education for Sustainable Development (AESD), who have now, at the project end also expressed a keen interest in becoming the “home” for educational materials developed under the project. They have also expressed a commitment to ensure that all the products are reviewed by the Ministry of Education. This will also make it more likely that the best ones will be used more widely in Belarus (see Recommendations)

The AESD unites juridical entities which are involved in the promotion of education in sustainable development. It is a non-for profit entity with the mission to coordinate the activities of the members vis a vis non-members (and in particular, the Ministry of Education). 3 of the pilots of the project are now members of the AESD (see Table 12). All 3, as well as the state educational institution "Kindergarten No. 6” in Ashmiany have already developed proposals to strengthen their role as the designated leaning centres: these proposals will need to be approved by the Educational Departments of the Reginal Executive bodies as well as the Ministry of Education (expected in August 2017). Staff of the Belarusian State Pedagogical University, the Academy of Postgraduate Education and the Republican Institute of Vocational Education are consultants of these innovative projects


Tag: Education National Institutions

31.

3.3.3.Active participation of local population in sustainable development  

It would be perhaps too speculative to believe that the PAGs will continue to exist beyond the project. There are, however, interesting developments worth mentioning, In Hrodna for example, the director of the kindergarten plans to include some of the members of the PAG in the membership of its Board of Trustees. In Viciebsk, while the cooperation between the college and the local branch of EED was always strong, it has become even stronger as a result of the project and at least in the near horizon it looks like the college will serve as a demonstration basis for the EED to train the industry personnel and a basis for bringing together also other interested parties


Tag: Civil Societies and NGOs

32.

3.3.4.Sustainability and replication potential – a summary

Both EU and UNDP have more initiatives that directly or indirectly promote EE. At the time of evaluation EU had a grant competition open for non-state actors and, to the information available, the regional executive bodies which hosted the pilots were planning to apply.

The EU/UNDP project "Support to Local Development in the Republic of Belarus" is implemented by UNDP in partnership with the Ministry of Economy of the Republic of Belarus.  Another EU-UNDP project, “Supporting the Transition to a Green Economy in the Republic of Belarus” 2014-2017) aims at promoting “green growth” concepts and environmentally sustainable production and consumption patterns through support of local “green” initiatives and an information campaign as a support measure all over the Republic of Belarus.

UNDP/GEF project (2016-2021) “Supporting Green Urban Development in Small and Medium-Sized Cities in Belarus (Green Cities)” promotes the elaboration of green urban development plans and pilot green urban development initiatives related to EE and sustainable transport in small and medium cities in Belarus. The project is implemented in the following areas:

  • a development and adoption of green urban development plans;
  • development of pilots on sustainable urban transport in Novopolotsk and Polotsk;
  • development of pilots on EE in Novogrudok; and
  • replication mechanisms for green urban development in Belarus.

These projects will potentially help in promoting the replication of the best practices from the current project


Tag: Urbanization Sustainability Partnership

33.

3.4.1.Project planning, implementation and adaptive management

Overall the project execution was satisfactory, as the project has achieved almost all the planned deliverables (with the exception of the website- based virtual network and with the caveat that the delivery of the local EE strategies could be considered as “achieved” in a rather formal sense of the word): this is noteworthy given that the project duration was too short, especially compared to its ambitious objectives. In relation Component 2, the execution of the project was indeed exemplary.

In the case of Component 1, the project could have taken arguably a more active coordinating role vis a vis the Ministry of Education: as discussed this would have had positive impact on many aspects of the project, its effectiveness and sustainability potential. Another aspect where the project management could have been more efficient is in assessing the effectiveness of its multiple information products, even though this was not reflected in the formal results framework of the project.
 


Tag: Challenges Effectiveness Ownership Project and Programme management Coordination

34.

3.4.2 Financial management

The project was executed in the National execution modality (NIM). The financial management followed UNDP rules with the necessary due diligence in place.

Cost effectiveness

One of the project aims was to involve local companies and equipment producers from Belarus in the process of modernization of the pilot sites as much as possible. This target was reached: the majority of equipment and construction materials were purchased from local producers, for instance:

  • Special thermal insulation for roof and walls insulation for all pilot sites;
  • New energy efficient windows with thermal resistance not lower than 1 m2K/W for all pilot sites;
  • Heat recovery ventilation equipment for all pilot sites;
  • Heat substations heat exchange, control equipment and all appliances for all sites;
  • Heat meters for all sites; and
  • Energy efficient lighting.

Tag: Efficiency Human and Financial resources

35.

co-financing

The promised co-financing materialized in full and even exceeded expectations (see Table 14) The contribution from the local governments was US$1,117 million instead of US$0,92 million exceeding the required 40 percent mark.


Tag: Government Cost-sharing

36.

3.4.3.Reporting, Monitoring and RBM

Project followed the bare minimum approach following the requirements of the RF (which has limited and very basic indicators). This reflects also to the quality of the RF of ProDoc which was discussed in 3.1.6 whereby there are targets specified in the body of the text of the ProDoc but not reflected in the RF (this was discussed in Section 3.1.6). In many respects, more could have been done, in particular with respect to:

Capturing the effectiveness of various information products, e.g. ensuring that there are mechanisms for capturing the feedback of the users and using this feedback to adjust the next rounds of information campaign- in line with the best practices of results based management (RBM);

Analysing and reflecting upon the feedback received from the participants of various workshops (feedback was collected but not analysed)

Similarly, project reporting could have been better with reflections on lessons learnt as the project progressed. The lessons learned log was regularly monitored and updated by the project team, but the best practice is that lessons learnt are captured during the project implementation and published (see Recommendations).


Tag: Knowledge management Results-Based Management

37.

3.4.4 Contribution of implementing and executing agencies

EED provided strong support to the project especially in terms of ensuring the successful implementation of the pilots through its regional branches. EED could have assisted the project more in relation of attracting a closer involvement of the Ministry of Education. Perhaps EED could also have been more explicit in terms of ensuring the visibility of this project as a supporter of EED mandate and strategy. UNDP requirements outlined in the UNDP Programme and Operations Policies and Procedures, concerning the Project Assurance were pursued as expected.

The project also benefitted from the oversight of the EU office in Belarus: there were cases where the timely observations from the EU have helped the project to address some of the issues (e.g. related to ensuring EU visibility).


Tag: Oversight Project and Programme management

38.

3.4.5 Coordination and synergies

The fact that the 2 UNDP projects – the current one and the UNDP/GEF project “Improving energy efficiency in residential building in the Republic of Belarus” - are co-located had perhaps helped to establish closer links between the two. For example, a conference dedicated to EE in public and residential sectors was organized jointly (and cost shared) on the 15th of October 2015 within the framework of Energy and Environment Forum. 149 representatives of local authorities, teachers and other stakeholder groups from the pilot regions learned about modern technologies and approaches to increase EE of social and residential buildings

There were no formal linkages established with other international development agencies, and WB in particular. The EE expert engaged in the project has worked in the previous WB projects, and hence, informally the learning from these WB projects was transferred to the current project in the part concerning technical solutions. Closer links would have benefitted the project, e.g. in shaping the survey questions, in the light of the upcoming WB project related to EE in heating in residential buildings.  


Tag: Knowledge management Programme Synergy

Recommendations
1

Provided that the idea is supported by EED, work on the handover to the Association for the Education for Sustainable Development (AESD)  of all the soft copies of information products based on  a MoU which will specify that (a) AESD will work with the Ministry of Education to review of all the educational tools are cleared to be used as part of the extracurricular teaching process as well as identify the ones that could be recommended to be used country wide (b) and all those that are cleared will be made available to the public free of charge.

2

Produce a Lessons Learnt report. There could be several booklets, with one of them dedicated to the analysis of economic costs and benefits of the pilots. The other one could feature the learning from the financing side of the pilots with an analysis of the potential for replication. The third one could feature the findings from the survey on awareness levels (but the findings need to be repackaged and presented better). After the booklets are produced, work with the EED to ensure its wide dissemination.

3

The quality of the information products, including teaching aids is appreciated by the stakeholders, but – and this is a lesson learnt – in similar projects in the future it is important to secure more active participation of the Ministry of Education.

4

And finally, in the part concerning the information campaign, there should have been mechanisms to capture the feedback of the recipients of the products (also a lesson learnt): timely analysis and learning from this analysis and acting upon this learning would have increased the effectiveness of the information component.

5

Given that upcoming tariff reform there might be justified need to support the population with further projects targeting public awareness raising as well as support the government with developing a subsidy scheme that will target the poor with their energy costs.

1. Recommendation:

Provided that the idea is supported by EED, work on the handover to the Association for the Education for Sustainable Development (AESD)  of all the soft copies of information products based on  a MoU which will specify that (a) AESD will work with the Ministry of Education to review of all the educational tools are cleared to be used as part of the extracurricular teaching process as well as identify the ones that could be recommended to be used country wide (b) and all those that are cleared will be made available to the public free of charge.

Management Response: [Added: 2018/01/12] [Last Updated: 2021/02/19]

Discuss The project team to ensure that the proposed actions are taken. The CO to monitor the progress.

Key Actions:

2. Recommendation:

Produce a Lessons Learnt report. There could be several booklets, with one of them dedicated to the analysis of economic costs and benefits of the pilots. The other one could feature the learning from the financing side of the pilots with an analysis of the potential for replication. The third one could feature the findings from the survey on awareness levels (but the findings need to be repackaged and presented better). After the booklets are produced, work with the EED to ensure its wide dissemination.

Management Response: [Added: 2018/01/28] [Last Updated: 2021/02/19]

Ensure that the Lessons Learned log reflects all the important issues raised by the evaluation, along with lessons identified by the project team.

Key Actions:

3. Recommendation:

The quality of the information products, including teaching aids is appreciated by the stakeholders, but – and this is a lesson learnt – in similar projects in the future it is important to secure more active participation of the Ministry of Education.

Management Response: [Added: 2018/01/28] [Last Updated: 2021/02/19]

When preparing educational materials for children work closely with the Ministry of education and its agencies on the content of these materials, as well as on the use of these materials after projects end.

Key Actions:

4. Recommendation:

And finally, in the part concerning the information campaign, there should have been mechanisms to capture the feedback of the recipients of the products (also a lesson learnt): timely analysis and learning from this analysis and acting upon this learning would have increased the effectiveness of the information component.

Management Response: [Added: 2018/01/28] [Last Updated: 2021/02/19]

Consider such mechanisms to be developed and applied.

Key Actions:

5. Recommendation:

Given that upcoming tariff reform there might be justified need to support the population with further projects targeting public awareness raising as well as support the government with developing a subsidy scheme that will target the poor with their energy costs.

Management Response: [Added: 2018/01/28] [Last Updated: 2021/02/19]

Further develop projects targeting energy efficiency, including public awareness in this area, in the country reflecting the respective country’s priorities.

Key Actions:

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org