Energy Efficiency Project Final Evaluaiton

Report Cover Image
Evaluation Plan:
2017-2021, Saudi Arabia
Evaluation Type:
Final Project
Planned End Date:
12/2019
Completion Date:
12/2019
Status:
Completed
Management Response:
No
Evaluation Budget(US $):
18,100

Share

Document Type Language Size Status Downloads
Download document 190801 Final TOR for the Terminal Evaluation Mission - AS.doc tor English 149.50 KB Posted 404
Download document 191216 Evaluation Report NEEP 2 Saudi Arabia Final - MWT (2).pdf report English 908.44 KB Posted 1025
Title Energy Efficiency Project Final Evaluaiton
Atlas Project Number: 00063524
Evaluation Plan: 2017-2021, Saudi Arabia
Evaluation Type: Final Project
Status: Completed
Completion Date: 12/2019
Planned End Date: 12/2019
Management Response: Yes
UNDP Signature Solution:
  • 1. Resilience
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 1.5.1 Solutions adopted to achieve universal access to clean, affordable and sustainable energy
SDG Goal
  • Goal 12. Ensure sustainable consumption and production patterns
  • Goal 7. Ensure access to affordable, reliable, sustainable and modern energy for all
SDG Target
  • 12.2 By 2030, achieve the sustainable management and efficient use of natural resources
  • 7.3 By 2030, double the global rate of improvement in energy efficiency
Evaluation Budget(US $): 18,100
Source of Funding: Project budget
Evaluation Expenditure(US $): 18,100
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
Tiago Santos Independent Evaluator tqs@your-savings.pt
GEF Evaluation: No
Key Stakeholders: Saudi Energy Efficiency Center, Minsitry of Energy, King Abdulaziz City for Science and technology
Countries: SAUDI ARABIA
Lessons
1.

The fact that many other countries have developed their legislation of EE based on an overarching law, followed by secondary legislation and technical regulations, does not mean that the same or even better results cannot be achieved with a different approach. The KSA has used existing laws and regulations and has set EE standards for energy related products as a way to pragmatically regulate certain markets and ensure certain economy agents abide to minimum energy performance requirements.

SEEC’s way of developing the EE Standards and Labelling program, resulted in an overall framework that ensures integrity of the regulated market. It is not only based on specifying minimum energy performance requirements (e.g., for building elements, home-appliances or cars). It also consists on a well-balanced structure that ensures:

  • the verification of compliance, of the energy related products and market actors, with the rules;
  • the enforcement of those rules in a way that exemplifies the consequences of not complying, which helps deterring market actors to be informed and comply with the standards.

It was found that there is no standard for thermal comfort in buildings, which results in having the air conditioning systems of the majority of the services and commercial buildings, operating with temperature set-points way below a reasonable temperature. Since EE and energy conservation is giving the first steps in the KSA, a strong effort of changing mentalities and practices should be envisaged. Although it was not quantified there is certainly a high potential for energy and energy cost savings on this particular energy conservation measure, through-out the Kingdom.

The fact that District Cooling is not progressing as desired, may be attributed to a less good choice of placement of the topic within SEEP. Energy efficient district cooling should be seen as a supply of final energy, hence, it should be placed on the Utilities Technical Team instead of on the Urban Planning Technical Team. Furthermore, it was perceived that there is a limited vision of how district cooling can be implemented, i.e., the source of energy that can drive a district cooling plant. The integration of district cooling with the idea (present on Vision 2030) of expanding the utilization of natural gas, e.g., in urban grids, opens the opportunity to the utilization of trigeneration systems as an efficient way of supplying cooling effect, heat and power.


Findings
1.

Implementation

2012

The implementation of the NEEP 2 Project started with the approval of the respective project document in late 2011. The project was designed with 4 Outcomes and several outputs for each one. The annual project report of 2012 refers that additional outputs were defined in the course of the first year of the project. It was not brought to the knowledge of the evaluator, management alterations during 2012 to justify the alteration of the project course right on the first year of implementation. The annual project report of 2012 does not make clear to which Outcome do the new Outputs belong, or if it was foreseen the surging of a new outcome.

2013

In 2013, the first outcome of NEEP started to have two new outputs as replacement of the original ones. The first one, called support the implementation of National EE Strategy, presupposes that there is already an EE Strategy to be implemented. However, during the meetings and remote work an EE Strategy was not presented to the evaluator. Throughout the evaluation process, especially during the field missions, this topic was debated without any conclusive result; it appeared that there was some reservation in either, disclosing the existence of a documented EE Strategy or assuming that “navigation” was being done without any minimally elaborated guidance. The second output is called EE Standard and Labelling and is dedicated to deal with the certification and labelling of the EE of energy related products, for which SASO has already set standards.


Tag: Energy National Implementation Modality Awareness raising Capacity Building

2.

2014

In 2014, and on the following years, the structure of outcome 1 has been unchanged with two outputs. While the evaluation has seen significant progress on the second one (EE S&L), on the first one this was not the case. In fact, the annual project reports inclusively point out repetitions of the previous year not elucidating the reader about what exactly has been done regarding the elaboration of an energy conservation law, the definition of an EE Strategy for the Kingdom and about developing the respective action plans. Regarding outcome 2, in 2014, the capacity development needs assessment gave place to the beginning of the capacity building of SEEC’s staff. I.e., the former output was replaced by the latter one. SEEC continued to focus on the training of local trainers which happened since 2012 and is seen as very good. Likewise, it continued to focus on the capacity building of the relevant target groups. In fact, these two outputs have been ongoing throughout the years and until 2019. This reflects the internal (and correct) perception that EE is to be implemented by persons and that for it to happen, a continuous effort must be carried out so that people understand the benefits of EE (why they should be done) and the mechanics of EE improvements (how they should be done and verified).


Tag: Energy Rule of law Awareness raising Capacity Building Coordination

3.

2016

In 2016, all outputs indicated on the annual work plans were similar to those of 2015. Exception for this is the output related with energy auditing (under Outcome 5) that stopped existing. Therefore, the Outcome 6 (of 2015) passed being Outcome 5 (in 2016).

2017

2017, 2018 and 2019 were years in which there were no changes on the outputs of the project. What is translated on the above paragraphs, can be observed in the Table 1, below. It is believed that, in such a project, there are good reasons for adaptations to the context and circumstances. These are generally better perceived by the coordination/management of the project rather than by the evaluators (be it med-term or final evaluator). In this sense, there is not much criticism on the changes done to the project document.


Tag: Energy Human and Financial resources Monitoring and Evaluation Programme/Project Design Project and Programme management Coordination

4.

Relevance

From the meetings held during the evaluation field missions, the evaluator got the perception that the interviewees were not very knowledgeable about the NEEP project. In a way, SEEC staff with already some years of work experience (within SEEC) and SEEC staff with only a few years of working experience (within SEEC), have probably been conducted to think in terms of the SEEP and not in terms of the (original) NEEP. The NEEP 2 project has been losing its relevance due to the facts: • SEEC and other SEEP partners have been progressing well on the process of capacity development of its staff; • The Technical Teams (TTs) are apparently well equipped (intellectually and technically) and, most importantly, motivated to pursue the goal of SEEP. • The coordination and leadership of the TTs has been effective, with a systematic gathering of information and reporting (from the TT to the SEEP Coordinator, on a weekly basis and from the SEEP Coordinator to the SEEC Executive Committee, on a monthly basis). • SEEC and SEEP partners are producing results, although it is clear that EE in the KSA is still giving its first steps.


Tag: Energy Relevance Human and Financial resources Sustainability Technical Support

5.

Most Notable Project Results

• SEEC is the institution dealing with EE management at the national level. It was established in the 1 st of November 2010 and became independent from KACST in 2016, with a well-defined mandate. Not being an original output of the project, SEEC’s structure and status as public institution, has benefitted from the NEEP 2 project. However, it was observed (and it is recognized by SEEC’s top management), that, in terms of autonomy, there are still some steps to give.

• SEEC has created the Saudi EE Programme (SEEP), which could be seen as the concretization of a national strategy for EE. To make SEEP operative, there are 13 technical teams, all of them having persons from SEEC and from SEEP partnering institutions. The SEEP is based on a multiyear (3-years) action plan in which adaptive management is present. On the end of year 1 the plan is revised, and a new 3-years plan is drafted and approved. The SEEP action plans ends-up by being part of SEEC’s action plans.

• SEEP mobilized more than 150 professionals from more than 30 governmental entities and state-owned enterprises. This is very good.

• SEEC has used/implemented Information Systems to handle, simplify and ensure effectiveness of certain national EE actions. Examples of this are:


Tag: Energy Effectiveness Relevance National Monitoring and Evaluation Project and Programme management UN Reform

6.

There seems to exist a very robust and well-coordinated verification and enforcement process. In general, the current process of verifying and enforcing for thermal insulation consists of four steps as follows:

Step 1 (Applying for new building permit) 

When the owner/ Engineering office applies for new building permit to Ministry of Municipal and Rural Affairs, he should provide the following: - Saudi Electrical Company certificate (for new user). - Filling a thermal insulation pledge form which states the commitment of the owner/ Engineering office to install the thermal insulation based on the given values in Saudi Building Code (The attached file has thermal insulation values for Saudi regions).

Step 2 (Issuing new building permit) 

- Sending copies for both of thermal insulation pledge form & building permit information to Saudi Electrical Company 

 Step 3 (Notification for Visit) 

- Saudi Electrical Company will send a letter to owner/ Engineering office to schedule three visits to verify thermal insulation for both of all walls, ceiling and windows. 

Step 4 (Issuing Building Completion Certificate) This step has two scenarios: 

- The first case scenario: The owner did not install thermal insulation. As a result, the Saudi Electrical Company would notify him to visit Ministry of Municipal and Rural Affairs. Moreover, Ministry of Municipal and Rural Affairs will not issue building completion certificate for the owner - The second case scenario: Both of Electrical Company and Ministry of Municipal and Rural Affairs will issue thermal insulation & building completion certificates respectively. Besides, the owner would get the electrical service.


Tag: Energy Environment Policy Relevance Partnership Education Jobs and Livelihoods Capacity Building Technical Support

7.

Most Notable Concerns or Shortcomings

• The evaluation could not identify a documented and publicly accessible EE Strategy grounded on relevant elements of the Saudi economy or energy sector. This is a shortcoming because although SEEC may have defined a strategy, the NEEP partners (UNDP, for example) could not have access to it. It is believed that there was/is an EE strategy since the evaluation found that a consultancy company (BAIN) has been assisting SEEC on this aspect for the past 5 years.

The Energy Conservation law has been drafted (during NEEP1) but it has not been enacted because, on one hand, it was said to be conflicting with other existing laws. On the other hand, it was considered preferable to use existing laws to derive from them, the necessary regulatory frameworks. It was therefore, considered unnecessary. While this may in practice result well, there is the risk of not considering aspects that could result in a further optimized situation. For example, the energy conservation law could include aspects related with Public Procurement or could authorize SEEC to participate on national energy planning or on the decision-making process about stipulation of energy prices and tariff structures. These two examples are not to be seen as the only ones, and they can affect significantly the EE at the national level. • The two points above, which are missing, were main components of the first outcome of NEEP 2. This is a shortcoming. • The autonomy of SEEC is not yet assured. The organization has structural deficiencies that imped it to be operative without the support of UNDP


Tag: Energy Environment Policy Relevance Human and Financial resources Procurement Project and Programme management Sustainability Trade and Development

8.

Effectiveness of Project Activities by outcome

Legislation, Strategy and Action Plans

On this regard the project implementation has not demonstrated the achievement of all the outputs. The annual project reports repeatedly mention that the EE strategy is ongoing or will be developed in the next year. While it may be acceptable that Saudi Arabia can reduce the growth of energy intensity without an Energy Conservation Law, by making use of existing regulations and public entities’ functions and by ensuring an effective concerted action between relevant stakeholders, it is not comprehensible that there is no publicly available EE Strategy. Or if there is, it is on someone’s mind. Regardless of if there is or not an EE Strategy, it was not disclosed during the evaluation; hence it may be said that there was a lost opportunity to infer about the existence of improvement opportunities of said strategy. For example, it was not possible to know if the reasoning being followed ensure that economy sectors with an important role in terms of diversification of economic activities are having the appropriate attention or support. Chances are that there is not an EE strategy or if there is, it was not shown. In practical terms, the evaluation considers this as a shortcoming of implementation.


Tag: Energy Environment Policy Effectiveness Rule of law Project and Programme management Technical Support

9.

Capacity Building of SEEC and nation-wide Energy Managers and Leaders

The adoption of the international certificates of Certified Energy Manager (CEM), Certified Energy Auditor (CEA) and Certified Measurement and Verification Professional (CMVP) started in 2015. In that year, only the CEM was available and from the 36 candidates that have participated on the single session of that year, 17 have passed the examination. In 2016, the three certifications were already available. There were three courses of CEM, two courses of CEA and one course of CMVP. From the 91 applicants to become CEM, 48 (53%) have passed the examination. From the 41 applicants to become CEA, 26 (63%) have passed the examination. From the 11 applicants to become CMVP, 4 (36%) have passed the examination. The information provided by SEEC, includes a total of professionals certified. This total is the sum of all partial summations. Since an individual may have participated in 1, 2 or 3 of the available certifications, the evaluation does not consider this summation relevant for the effectiveness assessment of the project’s activities. This observation will be valid for all the subsequent years.


Tag: Energy National Monitoring and Evaluation Capacity Building

10.

Preparation and establishment of EEIS

As referred previously, the activities performed in this component are not the same as described on the project document. The evaluation does not consider this as a shortcoming; it rather sees it as a misalignment between project design and SEEC’s mandate. At the date of signature of project document (18-12-2011) the SEEC was already established, most likely, with a clear mandate. The Information System described in the project document extrapolates the scope of a national EE Center. It is something to be developed at the level of the Ministry responsible for energy aspects.

The work done so far to equip the technical teams of SEEP with the necessary ICT infrastructures appears to have been effective and into a certain extent, following a logical reasoning or even a strategy. The six EE Information Systems (already mentioned) respond to the current needs of the SEEP and SEEC has demonstrated being able to, in a concerted action with SEEP partners, implement them in a way that the intended users (from public and private sectors) can make use of them.


Tag: Energy Monitoring and Evaluation Project and Programme management

11.

Awareness raising for several sectors of the economy

The effectiveness of the awareness campaigns could not be fully assessed because the meetings held with the relevant persons from SEEC and SEEP, difficultly resulted on the delivery of evidences. For example, the evaluator did not have access to the results of surveys done before and after each campaign. He could not also have access to the survey itself. However, the description of the work done by the Awareness Technical Team, as well as the information contained on the annual financial reports, reveals that a strong effort has been done and there is a sort of communication strategy. The strategy indicated is simple – develop campaigns for the existing standards or for any innovation that results from SEEP (e.g., Letabqa).

Changes from the Baseline

The Changes from the Baseline is a tool that was developed to translate, in a simple manner, the changes that have occurred, vis-à-vis, the desired impact areas. This tool was filled in by the Project Manager and Project Coordinator and the results of it are in Annex 3.


Tag: Effectiveness Change Management Project and Programme management Awareness raising Technical Support

12.

Cost Effectiveness

The NEEP 2 project has been implemented under the modality of National Execution. Moreover, Saudi Arabia is not eligible to receive donations from the United Nations, given its economic status. This means that the funds used to implement the project’s activities should be entirely from the government of Saudi Arabia. The next table resumes the annual expenditure of the project, since 2012 until 2019. There are several discrepancies between what has been budgeted and what has actually been spent. The table contains on column 2, the total expenditure for each year. It is coloured in red or green to indicate if expenditure has been above or below the annual budget, respectively. The aim of this evaluation is not to make financial audit of the project and the project is not donor funded. Hence, this aspect will not be investigated further.

For the outcome 1, of the NEEP 2 project, around 8.2 million USD were utilized. Given that the results of this component are far from being what was expected, the cost-effectiveness of this component is questionable or difficulty justifiable. For the outcome 2, around 7.5 million USD were utilized. This is a component in which, from the evaluator’s perspective, the project has done more. The results outlined previously, in combination with the perception of the readiness of SEEC (and SEEP partners) to manage EE at the national level, indicate that there is probably a good or very good cost-effectiveness on this component.


Tag: Energy Environment Policy Effectiveness National Human and Financial resources Operational Efficiency Project and Programme management Awareness raising

13.

Impacts

 Impacts, vis-à-vis, Project Objective

The impact of the project, regarding the reduction of energy intensity, is a complex subject. Firstly, one must take into account the meaning of energy intensity. Energy Intensity of GDP (EIGDP) is a ratio between the total primary energy consumption of an economy and the Gross Domestic Product (GDP) of that economy. Hence, if we see a chart of the EIGDP like the one below, we could immediately think that the country is on the right track in terms of EE. However, to have a lower EIGDP, we can have several combinations on the tendencies of both, primary energy consumption and GDP. In rigor, it is the rate of growth of primary energy consumption and GDP that will determine what happens with EIGDP. This is to say that the indicators should not be assessed alone.

While the tendency of the EIGDP can be explained by the tendencies of both, primary energy consumption and GDP, the observed reduction of primary energy consumption may not mean that the Kingdom is using energy in a more efficient way. Since the structure of the economy of the KSA is changing (under diversification), the changes in structure may be: more low energy intensive activities with high value added to GDP. This is to say that the reduction of primary energy consumption from 2015 onwards, may or may not be a result of the SEEP. To have a clearer perception of this, a more indepth study would be necessary; a study focusing on the growth of the contribution to GDP of the several economic activities and a study focusing on the energy intensity of each economic activity branch.


Tag: Energy Environment Policy Impact

14.

Sustainability

The sustainability of the measures put in place though the NEEP 2 appears to not be in risk. The meetings held during the field missions allowed perceiving that there is political commitment to improve the Kingdom’s EE and that a meeting of the Board of SEEC was going to happen in the end of 2019. In this meeting important things would be discussed, such as the nomination of a DG for SEEC. Precautions must be taken, vis-à-vis, power interests. The national interests should be the motivating factor. There are, however, certain concerns (already mentioned) related with the autonomy of SEEC. The next phase of NEEP should put a strong emphasis on the need to ensure that SEEC has all the necessary means to continue performing the expected role.


Tag: Energy Sustainability National

15.

Gender, Environmental, and Social Aspects

There are few evidences about favouring or encouraging the access of women or any minority part of the community, to the trainings or any other project activity. However, the contrary was also not observed. The only references that were made in regard of these aspects were:

• SEEC has only men employees because, in the recent past (when SEEC was occupying another building) there were no physical conditions to have men and women working on the same place. The SEEC office was on two floors of a building and the disposition of the working places was mainly of the open-space type. In the future, SEEC could open job opportunities for women, either on the core technical teams or in the business support areas (HR, Finance, Procurement, IT).

• Thee are few women graduated in engineering or in areas relevant to EE development. Hence, the participation on trainings promoted by the project is mainly of men. There is however, no policy against participation of women on said trainings. The only issue is about creating conditions for the eventuality that women want to participate.

• There has been a communication campaign specifically targeting women on the context of housekeeping.

• With the possibility to have women driving cars in Saudi Arabia, an opportunity arises for the change on the choice of cars. Dedicated campaigns aiming to change mentality about the size of cars could target specifically women. Focusing on a well-chosen key message e.g., Small is Beautiful, the next generation of car buyers could make things different, in terms of specific energy consumption10 of road transportation means.

SEEC has achieved the status of ISO 50001 certified organization11 . This is very good because, denotes commitment from the top management to, on one hand, show that more than wanting that people outside SEEC do something to improve EE SEEC is also giving the example, and on the other hand, to care about the environmental impact of SEEC’s activities.


Tag: Energy Gender Equality National Human and Financial resources Jobs and Livelihoods

16.

Design, Intervention Logic and Indicators

The design of the NEEP 2 projects follows common practice in the sense that it is composed of several components, impact areas or outcomes, and, has several outputs or groups of activities that, in combination, allow reaching the desired outcomes. The critiques to the project design are grouped in two different areas: i) relevance and completeness of the proposed activities; ii) adequacy of the indicators.

With exception of the EEIS as described on the project document, all activities seemed to be relevant. Even if not explained in a very comprehensive way, the EEIS proposed in the project document extrapolates the area of knowledge of Energy Efficiency and therefore it is understandably out of the scope of SEEC’s mandate. The name itself does not match well with its description. The description of the Information System proposed on the project document is more of an energy observatory, that countries establish as a way to compile, store and publish energy related data that is useful for energy planning and for other aspects, such as, learning or make research. This does not mean that SEEC should not be called to participate on the development of such Information System. But the call and the initiative should depart from other entity (e.g., Ministry responsible for energy)


Tag: Energy Monitoring and Evaluation Programme/Project Design

Recommendations
1

The top management of SEEC should resolve the issues related with lack of autonomy, increased responsibilities and brain-drain, either through a revised and adequate salary scale or through providing working conditions of excellence (including continual capacity building and exposure to relevant knowledge-transfer programs).

2

Consolidation of the regulations of the several EE programs. In other words, complete the SBC 601 and 602, continue the work of the energy management system associated with industry, develop and set the minimum energy performance requirements for heavy trucks, establish a mechanism to trigger the revision of stringency of the standards for energy related products, etc ….

3

Given the importance of updated and accurate energy and economic related data for national-level energy planning, SEEC should proactively propose to the Ministry responsible for energy the creation of an Energy Observatory, in partnership with the General Authority for Statistics.

4

Initiate a process of utilizing the potential of energy and energy cost savings associated with highly cost-effective energy efficiency measures. Examples of these measures are: i) setting thermal comfort standards and have them integrated on the SBC 602; ii) promote, change mentality and award the choice of small and efficient cars.

5

Revise the placement of District Cooling on the SEEP teams and. District cooling (which is seen as a measure with high EE potential and very relevant to realize a Saudi Vision 2030 measure), should pass from the Urban Planning team to the Utilities team. In addition, under the Utilities Technical Team, carryout a national survey of thermal energy needs (heating or cooling) that could be supplied with efficient Cogeneration or Trigeneration systems, making use of the work done under the Urban Planning team (identification of zones where it is feasible to implement district cooling systems).

6

Initiate the development of a publicly accessible EE Strategy. The strategy should, desirably, be based on sector-specific baseline studies of energy consumption and on socio-economic analysis of the population and of the economy agents. In addition, the disaggregation of socio-economic agents would be done by using the International Standard Industrial Classification of All Economic Activities (ISIC). The strategy would encompass also a wide analysis of possibilities to overcome the commonly existing barrier: financing of energy efficiency.

7

Given the importance of water to the Saudi nation, make use of the existing links and synergies between relevant stakeholders (SEEC, MOMRA, SWECC) to explore more ways of ensuring that the nexus energy-water is considered in the future mandate of SEEC. The range of possibilities is vast: from end-use water efficiency (in all sectors of the economy) to high-efficiency water desalination plants, passing by the well managed water transportation and distribution.

8

Continually improve the qualifications and skills of professionals involved in activities having a direct impact on the energy consumption, energy efficiency and energy conservation

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org