Success message
error message
warn message
Terminal Evaluation of UNDP/GEF Project: Standards and Labels for Promoting Energy Efficiency in Russia
Commissioning Unit: Russian Federation
Evaluation Plan: 2011-2017
Evaluation Type: Project
Completion Date: 05/2018
Unit Responsible for providing Management Response: Russian Federation
Documents Related to overall Management Response:
1. Recommendation:

(to UNDP): Project preparations should be better resourced to allow for proper assessment of baseline conditions and design of appropriate actions for market transformation, in this case for energy efficient appliances and building equipment. The Russian S&L Project design had 2 basic flaws:

•              An overemphasis on assisting local manufacturers to upgrade their production lines to manufacture energy efficient equipment.  This Project activity should not have been approved at the design phase without first conducting the baseline surveys of local manufacturing capacities during Project preparations. With this baseline survey as part of the Project activity, there was a risk that the size of the local manufacturing sector for appliances would have been insignificant and not worthy of the project support. The PMU, however, did not conduct this baseline survey and dropped all activities within Outcome 3 to assist the local manufacturing sector in 2014 but not before the Project had wasted more than US$1.1 million on efforts to build local manufacturing capacity through RATEK; and

•              An under-ambitious target of “submitting legislative reform to initiate voluntary S&L regulations for government consideration”, instead of aiming to have mandatory standards for energy efficient household appliances and building equipment. While it is true and acknowledged in the ProDoc that adoption of these S&L schemes was going to take an unknown amount of time (possibly longer than the Project duration), the Project needed to have a target of some form of mandatory standards. Without mandatory standards, local manufacturers would never have an understanding of the quality of appliances and equipment that would meet compliance.  As such, local manufacturers would never agree to any “voluntary standards” since these could change and force local manufacturers to provide additional investments to their production lines. In hindsight, the Project should have raised its ambition level to having mandatory standards and force local manufacturers to adjust their strategic development plans to become producers of more energy efficient products.

Management Response: [Added: 2018/06/12] [Last Updated: 2018/06/13]

Overall comments: The Project “Standards and Labels for Promoting Energy Efficiency in Russia” received unsatisfactory rating of the Terminal Evaluation (TE).  According to the Evaluator, the Project has not achieved its intended outcomes and objectives for a number of reasons ranging from poor project design, the slow adaptive management, lack of a strategic approach to meeting the Project objectives, and finally, to the late commencement of meaningful activities that were initiated in 2015 by a qualified project manager and an international CTA with strong and relevant experience in S&L programmes in Turkey.


UNDP appreciates the key observations and recommendations of the report and by and large agrees with them.  UNDP is committed to using the evaluation findings to improve project design, implementation and oversight. At the same time, UNDP notes with concern issues with approach, methodology and insufficient evidence to support some of the conclusions and content of the Evaluation report.




The evaluation team was expected to follow a participatory and consultative approach to ensure engagement with government counterparts. Yet, the views of Rosstandard, the key national stakeholder and beneficiary of the project, are not reflected.  This is particularly important considering the full National Implementation Modality of the project.


Scope of evaluation


The Evaluator failed to evaluate properly the achievements of the project in the last two years of implementation and did not take into consideration that the results framework of the project has changed after ‘adaptive measures” cleared with GEF team, were undertaken.  The report focuses primarily on the period 2010-2014/pre-mid-term evaluation while Evaluation ToR clearly reflects the duration of the project, after its extension until June 2017. 


Those adaptive measures tried to address the original and fundamental problem of project design and the fact that the project was overly ambitious (which the Evaluator admits), especially as the UNDP Country Office in Moscow was being closed in 2010 and UNDP management had limited capacity to support policy dialogue and implementation of this initiative.  Adaptive measures included revising the project log frame and adjusting indicators against which the project progress should be measured. The changes were reflected in the official UNDP-GEF Project Implementation Review for 2016 which was provided to the evaluators and which evaluators do not seem to have taken fully into account.


Use of evidence


Another issue undermining credibility of the evaluation findings is how the Evaluator uses evidence. We are particularly concerned that certain evidence provided to the Evaluator was not reviewed; in another instance evidence reviewed leads to debatable conclusions and, finally, that the Evaluator uses irrelevant information as evidence.

Even when relevant documents were considered, questionable conclusions were drawn. For example, the evaluators claim that the project had “poor or non-existent engagement of relevant S&L stakeholders”, while minutes of the National Inter-Agency Coordination Council, listed among the reviewed documents, indicates the opposite.  The Evaluator was given opportunity to interact with national counterparts of the project, including through participation in ENES conference - an event organized by the project where a broad range of the project stakeholders were present. Another example could be found in paragraph 27 of the report, where the Evaluator claims that the 2016 PIR used the original outcome indicators, which is not incorrect.

Management response to Recommendation 1: UNDP agrees with the recommendation related to the need for sound baseline studies at the project design phase. Project preparations should be better resourced by UNDP-GEF to allow for proper assessment of baseline conditions and design of appropriate actions for market transformation. Quality of the ProDoc defines the overall results, their sustainability, as well as critically streamlines the evaluation process.


According to remarks of TE team the project management team can give some clarifications below:

The element related to the promotion of the voluntary labeling in Russia was a result of the influence of EU consultants on the project design and the attempt to transfer the EU market experience. Outside of the UNDP-GEF project, the other international development partners have also been working to promote the voluntary EE labeling in Russia as part of the EE S&L strategies (e.g. EBRD). The MTE recommended to define the work on voluntary labeling better but not withdraw from it completely. However, the PMU have been recognizing the need to focus on the mandatory S&L and agreed with the technical recommendations of the independent strategic review conducted in 2014.   

National trademarks have their own niches and their weight is negligible on the whole market. From the other hand, most international trademarks (Bosh, Siemens, Neff, Samsung, LG, Panasonic, Beko, etc.) are the local manufactures now. They have their own plants in the Russian Federation. All international trademarks mostly comply with EE standards and it was not reasonable for the S&L Project to «help» them. The comprehensive analysis of market shares of local trademarks was really needed before the beginning of the S&L project to save time and funds for other activities.

There were several unsuccessful attempts to assist the local manufacturing sector to increase EE of their products («Birusa» and «Posis» refrigerators’ manufacturers and «Arktos» ventilation equipment manufacture.

Private-public partnerships were not working in Russia as well as voluntary certification systems of separate companies or business associations. It became clear only after attempts to initiate some of them together with projects partners.

After getting recommendations from international consultants (mid-term evaluator and Dr. Yury Pashik) all activities in the area were closed (2015) and financing (1 145 616 USD) was redirected to the development of national EE testing infrastructure, testing standards and lab’s staff trainings.

We disagree that the project activities delivered by RATEK association were “waste” of the project funds. An independent strategic review undertaken after the MTE rated the relevance and quality of these activities very positively. Business association RATEK was not involved in the Outcome 3 activities, rather in the consumer awareness work and building capacities of the retail networks to promote EE appliances.  

Key Actions:

Key Action Responsible DueDate Status Comments Documents
For the S&L project manager with assistance from PSO and UNDP/GEF RTA, prepare Lessons Learned report to share with other UNDP/GEF projects.
[Added: 2018/06/13]
S&L project manager with assistance from the PSO and the UNDP/GEF RTA 2018/04 Completed
2. Recommendation:

(to UNDP): UNDP-GEF projects implemented by country offices need strong oversight by regional technical advisors who are qualified experts in the subject matters of the projects they are advising. In the case of the Russian S&L Project, there was a disconnect between the Russian PSO and the RTA of the UNDP Istanbul Regional Center and HQ on implementation and monitoring of the Project. Directives of the RTA are results-based (which was the case on this Project based on the RTA’s determination of the poor Project progress).  To overcome deliberate delays of a country office (as was deemed the case with the Russia PSO despite the poor progress of the Project between 2010 and 2015) to act on implementation directives from the RTA, and to create more urgency for a country office to act on these directives, approval of Project AWPs of a country office need to made conditional to meeting directives of an RTA that would force a country office to act on these directives.  This “leverage tactic” clearly worked in early 2014 and 2015 on the S&L Project when the PSO urgently sought a no-cost extension from the end of 2015 to 2017.

Management Response: [Added: 2018/06/12] [Last Updated: 2018/06/13]

The full-NIM implementation modality clearly defines the roles and responsibilities of UNDP and the Government over the project governance and management, with the NIM partner taking the lead in directing and implementing the project. Secondly, there is a clear division of responsibilities between the UNDP-GEF and the UNDP Country Office (PSO) over the project supervision, oversight and implementation support. The latter is outlined in the UNDP-GEF DOA and includes the description of the roles and responsibilities of the UNDP-GEF RTA.

RTA was member of the project Steering Committee and Selection Committee for the key staff and consultants. The RTA also had the authority to approve annual ASLs to the project, while the decisions on the project extension have been made by the UNDP-GEF Management.

The NIM implementation modality for UNDP projects has participatory approach to decision-making as one of its key elements towards transparent and inclusive project governance and the national ownership. In this context, any “directives” from the RTA were conveyed to the Steering Committee and the NIM Partners. Unfortunately, many directives and instructions were sent by the RTA to the PMU without informing the CO/PSO.

Key Actions:

3. Recommendation:

(to the Government of the Russian Federation): With an estimated 1 to 3 years before there is full agreement of harmonized EE technical regulations for the Customs Union, the Government of Russia should agree to develop its own national EE technical regulations as an interim measure for adoption of S&L schemes for household appliances and building equipment. This would permit the development of appropriate market surveillance plans, eco-design and energy labelling, a strengthened S&L regulatory framework, and product testing, all of which will need further support after closure of the S&L Project.

Management Response: [Added: 2018/06/12] [Last Updated: 2018/06/13]

The recommendation is addressed to the Government of the Russian Federation and therefore no action is required by the UNDP.

Key Actions:

4. Recommendation:

(to UNDP PSO):  The PSO should monitor adoption of the regulatory S&L framework (patterned after EU MEPS regulations) by the Ministry of Industry and Trade (MoIT) as a mandatory S&L programme.  This will provide some indications of the timetable for adoption of harmonized EE technical regulations for the Customs Union as applied to all manufacturers of electronic equipment and household appliances on the Russian market of the “performance threshold for products to enter the Russian market” especially pertaining to energy performance. The adoption of the S&L regulatory framework for “eco-design” and “energy labeling requirements” as well as enforcement of these requirements is the first major and indispensable step towards sustaining the testing lab investment, and possibly other Project results. For local manufacturers and suppliers of EE appliances and building equipment, they can comply through voluntary or mandatory S&L programmes as long as the definitions of voluntary and mandatory are clear. Mandatory programmes have requirements that are set out by mandatory laws and regulations that only apply to those particular electronic products (including public procurement that would define laws and regulations for specific electronic products). For voluntary programmes, a voluntary agreement would be entered into by and between relevant industry and the government agencies (MoIT and Rosstadnart) where the energy efficiency and other eco-design targets are set out. Manufacturers of products covered by these EE and eco-design targets can voluntarily participate in this programme; however, once the manufacturer participates, these targets become mandatory for them to achieve.  A good example of a “voluntary” actions by local manufacturers can be found on the Russian UNDP-GEF project “Transforming the Market for Efficient Lighting” (GEF ID: 3658); this is a recently completed project in Russia where local lighting manufacturers set their own MEPS for lighting devices (based on EU directives) and worked with test labs and the Government to achieve compliant products followed by investments to upgrade their manufacturing capacities.

Management Response: [Added: 2018/06/12] [Last Updated: 2018/06/13]

The project management team partly agrees with the recommendation.

The recommendation is wrongly addressed to the UNDP PSO, which will be closed in April 2018. However, the combined efforts of the PSO, governmental partners and project management team have provided a mechanisms for implementation of the recommendation, as the  Russia-UNDP Trust Fund project “Regulatory Framework to Promote Energy Efficiency in Countries of the Eurasian Economic Union” was recently approved by the Russian Government and UNDP.

Any mandatory national S&L programme related to all market participants can start only after adoption of mandatory MEPS as a part of regulatory S&L framework.  Such a regulatory framework is the area of Eurasian Economic Union (ECU Technical Regulation "On requirements to energy efficiency of electric power consuming devices"). It will be officially adopted not earlier the end of 2017. After the official adoption at the Custom Union level, the transitional period in 18 months will start. During this period, national market surveillance Bodies responsible for execution of the ECU Technical Regulation requirements will be established. Also, all supporting EE testing standards, national MVE regulations and testing infrastructure must be ready till the end of the transitional period to fulfill the regulatory S&L framework. The PSO will be closed at early 2018 and will not have any S&L experts onboard in 2018 and not be able to monitor adoption of any regulatory S&L framework as a mandatory S&L programme.

The example of a “voluntary” actions by local manufacturers which the TE team recommend looking at the case of Russian UNDP-GEF project “Transforming the Market for Efficient Lighting” (GEF ID: 3658) is not valid. It is the example of simple voluntary MEPS agreement system. There were no connections to the Governmental Authorities in this voluntary system. There was no certification. Indeed, the National Voluntary Certification System initiated by ROSSTANDARD is the first S&L Certification System initiated by the Governmental Agency directly.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
UNDP to launch a new regional project “Regulatory Framework to Promote Energy Efficiency in Countries of the Eurasian Economic Union” which could serve as a mechanism to secure a follow up to the EE regulatory reform.
[Added: 2018/06/13]
UNDP 2018/04 Completed The new project is launched, the first SC meeting conducted 20 March 2018. The project facilitates EE regulatory reform within the Custom Union.
5. Recommendation:

(to UNDP PSO):  Continue negotiations with Rosstandart to ensure implementation of the demo product testing programme (extended or limited scenario) during the remaining implementation period and even after official closure of the project to obtain the initial compliance profile of the marketplace, produce data for further market surveillance strategies and programmes, and to build testing experience among the testing centers where new laboratories are being developed. This recommendation is made since Rosstandart has agreed to take the ownership of the enforcement process.

Management Response: [Added: 2018/06/12] [Last Updated: 2018/06/13]

The project management team agrees with this recommendation, which needs to be directed to the PMU.

ROSSTANDARD’s Deputy Head Mr. Anton Shalaev officially claimed that ROSSTANDARD will support, develop and extend created EE testing labs and will integrate them in the demo product testing programmes under the National Voluntary Certification System and future conformity assessment programmes under the Technical Regulation of Custom Union. You can look at his official statement here (in Russian and English):

and at the official letter of support from Rosstandard here (in Russian and English):

Key Actions:

Key Action Responsible DueDate Status Comments Documents
For the S&L project manager, with assistance from PSO, continue negotiations with Rosstandart to ensure implementation of the demo product testing programme after official closure of the Russian S&L project
[Added: 2018/06/13]
S&L project manager, PSO 2017/12 Completed
6. Recommendation:

(to the Government of the Russian Federation):  In parallel to laboratory investment and testing staff training programme, immediately proceed with establishing a market surveillance organization (such as with ROSPOTREBNADZOR, Rosstandart or any other governmental authority) and developing market surveillance knowledge and skills within this designated organization.

Management Response: [Added: 2018/06/12]

The recommendation is addressed to the Government of the Russian Federation and therefore no action is required by the UNDP.

Key Actions:

7. Recommendation:

(to UNDP PSO): Assist Rosstandart in contacting other global manufacturers operating in Russia for possible cooperation in training testing staff of other testing centres outside of St. Petersburg. On March 1, 2017, the Project supported the training of Rostest testing staff for Test-St. Petersburg in cooperation with BSH.  Due to the success of this event, building the testing skills of the remaining 6 testing centers where new testing laboratories are being developed is a high priority.  Such training will ensure effective operation of these laboratories by well trained and skilled staff. Such training of all the laboratories can lead to ROSTEST evolving into a Centre of Excellence and participating in ongoing programs with other test laboratories, both nationally and internationally. Current model of cooperation with global manufacturers consists of:

•              Search of manufacturers with appropriate experience and ready to share their experience;

•              Organization of transfer of the experience within the Rosstandart system within the premises of this manufacturer;

•              Sharing of experience using the Rosstandart system with regional testing centers;

•              Further cooperation with the manufacturer.

Management Response: [Added: 2018/06/12] [Last Updated: 2018/06/13]

The project management team agrees with this recommendation. Representatives of all 6 labs created with the help of the S&L project participated in the staff training at BSH refrigerators and freezers testing laboratory in St.Peterburg. After that, experts of BSH company accomplished the technological audit of the Test-St.Peterburg lab and shared the results between all labs created in ROSSTANDARD structure. Next training will be held at Nizhegorodsky CSM lab in early 2018.

Project Manager shared the information about this successful training programme with the manufactures which have relevant testing facilities and expert staff directly and through the main business associations. Key experts of these companies are the participants of relevant Technical Committees (TKs) of ROSSTANDARD now. There were 4 meetings of ROSSTANDARD Deputy Head with the representatives of companies and business associations in 2017 organized by the S&L project. So, the ROSSTANDARD have the direct link to them now.

Key Actions:

8. Recommendation:

(to the Government of the Russian Federation): Assist ROSTEST and all CSMs in obtaining accreditation to EN-ISO/IEC17025 or national accreditation from national Accreditation Body (ROSACREDITATION) for each of the relevant test procedures in all its testing laboratories.

Management Response: [Added: 2018/06/12]

The recommendation is addressed to the Government of the Russian Federation and therefore no action is required by the UNDP

Key Actions:

Key Action Responsible DueDate Status Comments Documents
For the S&L project manager to send a letter to remind ROSSTANDARD that all questions related to the official accreditation, relevant usage and future financing are the area of ROSSTANDARD’s CSMs responsibility
[Added: 2018/06/13]
S&L project manager 2017/12 Completed
9. Recommendation:

(to UNDP PSO): With the remaining resources of the S&L Project, find external sources to ensure wider dissemination of the S&L awareness raising messages, possibly through the broadcasting of the videos produced under the Project’s campaign on national TV channels or national websites with high ratings with viewers.  This recommendation is made as a follow-up to the successful public awareness campaign that has already been launched by the S&L Project.

Management Response: [Added: 2018/06/12]

The project management team agrees with the recommendation and would like to point out the following:

The most effective and sustainable project outputs aimed to ensure wider dissemination of the S&L awareness raising messages were oriented on the promotion through Internet, social networks and solutions integrated in large PR projects and activities of project’s partners. All activities mentioned below had sustainable results and will be available after the end of Russian S&L project.

Public awareness raising campaign "A class – norm of living" and salesman’s training e-course for the largest network resellers were successfully implemented. The interactive e-learning course “EE Labeling of Refrigerators and Washing Machines» was previously developed and included in the corporative e-learning systems of the main resellers (Eldorado, Media-Markt and M-Video – at least 36% of overall market sales). More than 9000 salesmen were already trained. About 270 new salesmen participate in this course every month.

Series of on-line articles «How to choose household appliances» at Yandex Market ( and others with more than 120 000 views), virus movie at YouTube ( with more than 260.000 views, Fixies cartoon ( with more than 1.500.000 views and S&L project site ( are available.

Social networks’ groups related to the public awareness raising campaign "A class – norm of living" are working and more than 17.000 constant participants are there now (the coverage of target audience in 200.000 people including Internet blogs):


Mobile arcade game "Energy Keeper" (, is available for downloading and have already provided the coverage of target audience in 50.000 people.

Integration with the network Kidburg educational programme have already provided the coverage of target audience in 10.000 children ( S&L special trainings are available there every day.

An all-Russian contest of the best EE practices in the category "Home appliances and engineering equipment" was initiated and held.  S&L Project participated in major professional forums (ENES-2016 and «Moscow – EE City»), provided informational and organizational support during the interactions of the governmental authorities with representatives of the market and the expert community.

The Project Manager and UNDP PSO got an official request from the UNDP/GEF Project “Energy Efficient Standards, Certification, and Labelling for Appliances and Equipment in Kazakhstan” to share the results of the project. We are sure that the information package we shared will be of service.

Key Actions:

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213