Final evaluation: Project on HCFC Phase out

Report Cover Image
Evaluation Plan:
2016-2020, Uzbekistan
Evaluation Type:
Final Project
Planned End Date:
03/2018
Completion Date:
11/2018
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
20,000

Share

Document Type Language Size Status Downloads
Download document TOR of Terminal Evaluator HCFC phase out.pdf tor English 1600.05 KB Posted 8
Download document Terminal Evaluation_Report Regional_CEIT HCFC phase out.pdf report English 2791.82 KB Posted 9
Title Final evaluation: Project on HCFC Phase out
Atlas Project Number: 00063869
Evaluation Plan: 2016-2020, Uzbekistan
Evaluation Type: Final Project
Status: Completed
Completion Date: 11/2018
Planned End Date: 03/2018
Management Response: Yes
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 2.1.1 Low emission and climate resilient objectives addressed in national, sub-national and sectoral development plans and policies to promote economic diversification and green growth
  • 2. Output 2.5.1 Solutions developed, financed and applied at scale for energy efficiency and transformation to clean energy and zero-carbon development, for poverty eradication and structural transformation
Evaluation Budget(US $): 20,000
Source of Funding:
Evaluation Expenditure(US $): 5,184
Joint Programme: No
Mandatory Evaluation: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
GEF Evaluation: Yes
GEF Project Title: Initial Implementation of Accelerated HCFC Phase Out in the CEIT Region
Evaluation Type: Terminal Evaluation
Focal Area: Ozone Depleting Substances
Project Type: EA
GEF Phase: GEF-1
GEF Project ID: 4102
PIMS Number: 4309
Key Stakeholders: State committee for nature protection
Countries: UZBEKISTAN
Lessons
1.

The project design with the regional component to support the national components allowed for a consistent approach for implementation of the sub-component on ODS legislation through engagement of an international consultant. Moreover, it facilitated establishment of an equal basis of master trainers as a foundation for further cascading down of trainings enforcement officers and RAC service technicians in the participating countries. The centralized procurement of services through the regional component proved to be the efficient waxy of spending the limited project resources.

However, design of future similar projects should take into full account the decentralized project implementation at the multi-country level. Mandatory approval of the project by the implementing agency and all participating governments took about 6 months and in one country there was further delay in establishment of the national project implementation unit. All this resulted in uneven progress in implementation in the four countries which inevitably affected implementation of the regional component.

From this experience, the planned three-year duration proved to be too short for such a complex project and a two-year extension had to be requested. Such extension represented obvious challenges as to the allocation of the project management costs for the regional component and therefore it would be prudent to take into full account the predisposition for extension in the budgetary allocations of future similar regional component’s.

The original project document envisaged that the Project Implementation Unit (PIU) would consist of a 30% part-time project assistant who would work in coordination with the Regional Technical Advisor. This arrangement proved to be vastly insufficient for a project of this complexity and size (the regional component budget allocation US$ 1,080,000 and envisaged support to four countries in two sub-regions). The corrective measure taken at the 2015 Regional Project Board, namely recruitment of the regional Project Manager, albeit on 50%-time allocation basis, was a step in the right direction and dramatically improved effectiveness of the support provided to the substantive activities of the four national components as well as timeliness of the administrative/financial procedures required for the project extension and closure.

The experience gathered with the establishment of the PIU in Uzbekistan provides further evidence that the PIU composition requires a careful consideration at the project design. Instead of a single national project manager, the national Project Steering Committee decided to recruit several experts with expertise in differentiated areas such as investment/demonstration projects, monitoring & evaluation as well as in public relations & outreach. Such division of responsibilities allowed the national Project Manager to concentrate fully on the project management and coordination function while enabled use of a specific expertise for development and practical implementation of innovative activities and production of cutting-edge results such as the on-line monitoring of ODS consumption, development of interactive games and the photo contest on ozone layer protection.

The example of the photo contest, namely that it had expanded from the national component to reach world-wide proportions is an illustration of importance of assistance by the regional component that allocated additional funds and opened communication channels that were not available at the national level. This case also suggests that additional capacity on public outreach and external communication should be considered for any future regional project, at least on a part time basis. Such capacity will also help with translation of the technical language related to the Montreal Protocol into communications easily understandable by the general public and will thus make a notable contribution to the public awareness facet of the project.

The substantive revision of the national component for Ukraine included revision of the original investment technology conversion sub-project of a systems house Polyfoam. The revision included allocation of additional funds for technology conversion at the downstream clients of the systems house. This is considered as good practice for facilitation of technological conversion at SME companies that are not eligible for direct financial assistance due to their low consumption of HCFC. The experience from the completed Polyfoam investment sub-project will be vital for smooth implementation of the newly formulated investment sub-projects at two other systems houses that also include conversion of their downstream clients.

The cancellation of the national pilot ODS waste sub-projects and the subsequent commissioning of the ODS Waste Management Concept for the four countries by the regional component demonstrates beyond reasonable doubt that the area of ODS waste management and disposal requires a regional approach, in particular for smaller countries, given the low amounts of ODS in the current ODS banks in the project countries.

The support from the policy component of the project to initial capacity building on HFC phase-down was incorporated into the project as a response to the recent adoption of the Kigali Amendment of the Montreal Protocol. Although such activities were beyond the scope of the original project they represent an example of a prudent reaction to recent development in the area closely related to the theme of the project.  The exposure of the project beneficiaries to essential information on phase-down of HFC could be considered as an incentive for early ratification of the Kigali Amendment by the project countries and the first step towards future activities on HFC phase-down. The country specific roadmaps elaborated under the ODS Waste Management Concept cover apart from the existing ODS banks also unwanted HFC therefore the capacity building on HFC contributed to the discussion of potential solutions to the ODS waste disposal challenge through improved economy of the joint ODS/HFC management and disposal.

On the contrary, the aspiration of the project to support development of a regional network of RAC associations proved to be too ambitious for two reasons. Firstly, there is no experience with establishment of a similar regional mechanism for the Article 5 countries in the ECA region and secondly, the RAC associations were functional only in two of the four project beneficiary countries. Therefore, support for establishment of RAC associations should be provided primarily under national projects with additional support through bilateral exchanges with countries with already functional RAC associations.

The project has proven enormous value of the support to pilot demonstration sub-projects and confirmed the conducive and catalytic effect international development assistance can have for adoption of ozone-friendly technologies and non-ODS refrigerants by the private sector.  The uptake and replication of the initial project-funded demonstration cases by the private sector beneficiaries attests that such projects can leverage extensive investments into ozone-friendly technologies and refrigerants, in particular as cost performance (including energy efficiency) appears to be dominating factor for private sector investments rather than the environmental performance. Projects that can demonstrate substantive operational cost savings and related profit gains in addition to environmental benefits will have a huge impact on progress in HCFC phase-out and reduction of HCFC and HFC use in refrigeration and AC systems. Last but not least, such projects are a good example of cooperation of UN organization with private sector and in particular show a catalytic role UN can play in demonstration of environmental-friendly technologies and mobilizing of private sector funding.


Findings
Recommendations
1
  1. UNDP should ensure that standards of frequently used ODS are provided to the countries implementing ODS reduction projects to enable both qualitative and quantitative analysis of refrigerants and refrigerant mixtures. In case internationally certified standards ca­n’t be imported to the project countries, support should be provided for development and local certification of ODS proxy standards using imported virgin refrigerants of declared purity.
  2. UNDP should consider provision of sufficient number of refillable refrigerant containers to the already established as well as new refrigerant reclaiming centers.
  3. UNDP should ensure that international advice on good practices in refrigerant reclamation industry, including advice on elaboration of technical and business plans, is provided to the countries implementing ODS reduction projects in order to improve operations of their national reclamation schemes.
2

UNDP in cooperation with countries implementing ODS reduction projects should develop outreach activities aiming at the end-users of RAC equipment to explain risks and disadvantages of engagements with the informal servicing sub-sector. The end-user outreach programmes should in particular advocate that cheaper immediate options tend to lead to greater costs in the long term and as well as a worse environmental impact. 

3

UNDP together with the countries implementing ODS reduction projects should monitor developments under the UNIDO regional demonstration project on ODS disposal and ensure that national reporting systems are developed and functional for inventories of unwanted ODS and that information on the stock of ODS waste is readily available once a viable solution is proposed by the UNIDO project.

4

UNDP should consider conducting an analysis of economic benefits of good practices in refrigeration servicing and retirement of ODS-based equipment for inclusion in public outreach programmes directed on SMEs and residential segment of the end users.

5

5.1. UNDP should ensure enlarged participation of qualified national trainers in future ODS-related train-the-trainers programmes and to the extent possible organize T-o-T events with the established refrigerant training centres in the ECA region in order to improve cost-effectiveness and overcome the language barrier

5.2. UNDP should ensure that national counterparts from the countries implementing ODS reduction projects learn from the experience with the use of CO2 as refrigerant in the region. E-courses, study tours and train-the trainers programmes could be organized with the Training Centre on use of CO2 as refrigerant that was established at the NORD O.O.O. company in Moscow.

6

UNDP should ensure that indicators in the results framework are attached to a time frame and state when they will be measured. The timely dimension of the indicators will allow for prioritization of actions in the project implementation plans.

7

7.1. For procurement of portable refrigerant identifiers and RAC service tool kits, UNDP should consider either to conclude own LTAs or use LTAs already in place at sister organizations of the UN system that have acquired experience with procurement of equipment items for MP projects (e.g. UNIDO).

7.2. UNDP should ensure that Terms of Reference for procurement of major equipment items contain clear definition of related services to be guaranteed by the equipment suppliers, in particular that the supplier’s after sale service agents are operational in the recipient country or at least in the neighboring country. Provision of the after-sale services should be one of the criteria for commercial evaluation of bids submitted under the procurement event.

8

UNDP should ensure that national project implementation teams establish on-going operational monitoring on actually provided co-financing for the projects

1. Recommendation:
  1. UNDP should ensure that standards of frequently used ODS are provided to the countries implementing ODS reduction projects to enable both qualitative and quantitative analysis of refrigerants and refrigerant mixtures. In case internationally certified standards ca­n’t be imported to the project countries, support should be provided for development and local certification of ODS proxy standards using imported virgin refrigerants of declared purity.
  2. UNDP should consider provision of sufficient number of refillable refrigerant containers to the already established as well as new refrigerant reclaiming centers.
  3. UNDP should ensure that international advice on good practices in refrigerant reclamation industry, including advice on elaboration of technical and business plans, is provided to the countries implementing ODS reduction projects in order to improve operations of their national reclamation schemes.
Management Response: [Added: 2018/10/26]

Fully agree

Key Actions:

Key Action Responsible DueDate Status Comments Documents
1.1. Organization discussions on the possibility to import virgin ODS standards to enable both qualitative and quantitative analysis of refrigerants and refrigerant mixtures with the national partners during the development of new GEF/UNDP project, 1.2. Consideration provision the established as well as new refrigerant reclaiming centers with additional refillable refrigerant containers during new GEF/UNDP project 1.3. Consideration development of technical and business plans for RRR Centers during implementation of new UNDP/GEF project
[Added: 2018/10/26] [Last Updated: 2018/11/29]
UNDP Sustainable Development Cluster 2018/12 Initiated 1.1. Activities on close cooperation with the State Committee for Ecology and Environmental Protection and the State Customs Committee on improvement import/export control of ODS, including provision customs laboratories with virgin ODS standards included in the proposed project document of new UNDP/GEF MSP 1.2. Within new MSP new and established RRR centers will be provided with additional refillable refrigerant containers 1.3. Development of technical and business plans for RRR Centers planned during implementation of new UNDP/GEF MSP on Complete HCFC Phase-out
2. Recommendation:

UNDP in cooperation with countries implementing ODS reduction projects should develop outreach activities aiming at the end-users of RAC equipment to explain risks and disadvantages of engagements with the informal servicing sub-sector. The end-user outreach programmes should in particular advocate that cheaper immediate options tend to lead to greater costs in the long term and as well as a worse environmental impact. 

Management Response: [Added: 2018/10/26]

Fully agree

Key Actions:

Key Action Responsible DueDate Status Comments Documents
2.1. Development and inclusion to the Project Document of the new UNDP/GEF Project outreach activities with explanation risks and disadvantages of engagements with the informal servicing sub-sector
[Added: 2018/10/26] [Last Updated: 2018/11/29]
UNDP Sustainable Development Cluster 2018/12 Initiated The proposed Project Document of new UNDP/GEF Project has separate component on Outreach and activities aiming at awareness raising of the end-users of RAC equipment to explain risks and disadvantages of engagements with the informal servicing sub-sector
3. Recommendation:

UNDP together with the countries implementing ODS reduction projects should monitor developments under the UNIDO regional demonstration project on ODS disposal and ensure that national reporting systems are developed and functional for inventories of unwanted ODS and that information on the stock of ODS waste is readily available once a viable solution is proposed by the UNIDO project.

Management Response: [Added: 2018/10/26]

Fully agree

Key Actions:

Key Action Responsible DueDate Status Comments Documents
3.1. Inclusion into the work plan of the new UNDP/GEF project activities for studying developments of UNIDO regional demonstration projects on ODS disposal and preparation proposal(s)
[Added: 2018/10/26] [Last Updated: 2018/11/29]
UNDP Sustainable Development Cluster 2018/12 Initiated Activities on networking improvement with international organizations and study advanced experience of foreign countries and development institutions included into the workplan of new
4. Recommendation:

UNDP should consider conducting an analysis of economic benefits of good practices in refrigeration servicing and retirement of ODS-based equipment for inclusion in public outreach programmes directed on SMEs and residential segment of the end users.

Management Response: [Added: 2018/10/26]

Fully agree

Key Actions:

Key Action Responsible DueDate Status Comments Documents
4.1. Consideration, within new projects, conducting analysis of economic benefits of good practices in refrigeration servicing and retirement of ODS-based equipment and inclusion into work plan of new projects a public outreach programmes directed on SMEs and residential segment of the end users.
[Added: 2018/10/26] [Last Updated: 2018/11/29]
UNDP Sustainable Development Cluster 2018/12 Not Initiated Activities on conducting analysis of economic benefits of good practices in refrigeration servicing and retirement of ODS-based equipment included into the workplan of new UNDP/GEF MSP
5. Recommendation:

5.1. UNDP should ensure enlarged participation of qualified national trainers in future ODS-related train-the-trainers programmes and to the extent possible organize T-o-T events with the established refrigerant training centres in the ECA region in order to improve cost-effectiveness and overcome the language barrier

5.2. UNDP should ensure that national counterparts from the countries implementing ODS reduction projects learn from the experience with the use of CO2 as refrigerant in the region. E-courses, study tours and train-the trainers programmes could be organized with the Training Centre on use of CO2 as refrigerant that was established at the NORD O.O.O. company in Moscow.

Management Response: [Added: 2018/10/26] [Last Updated: 2018/10/26]

Fully agree

Key Actions:

Key Action Responsible DueDate Status Comments Documents
5.1. Planning involvement of qualified national trainers in future ODS-related train-the-trainers programmes in the training activities of new UNDP/GEF projects 5.2. Planning organization of study tours and train-the trainers programmes on application of CO2 within new UNDP/GEF projects
[Added: 2018/10/26] [Last Updated: 2018/11/29]
UNDP Sustainable Development Cluster 2018/12 Initiated 5.1. Organization and conducting training programme (ToTs, site and online trainings) for refrigeration technicians with involvement of local trainers planned within the proposed ProDoc of new UNDP/GEF MSP 5.2. Within new UNDP/GEF MSP, UNDP and related national partners/ beneficiaries will organize study the experience on application of natural refrigerants in cooling/air condition, including CO2
6. Recommendation:

UNDP should ensure that indicators in the results framework are attached to a time frame and state when they will be measured. The timely dimension of the indicators will allow for prioritization of actions in the project implementation plans.

Management Response: [Added: 2018/10/26]

Fully agree

Key Actions:

Key Action Responsible DueDate Status Comments Documents
6.1. Development of Project Results Framework of new GEF MSP on Complete HCFC Phase-out with measurable indicators tailored to the time-frame for prioritization of actions in the project implementation plans
[Added: 2018/10/26] [Last Updated: 2018/11/29]
UNDP Sustainable Development Cluster 2018/12 Initiated Project Results Framework of new GEF MSP on Complete HCFC Phase-out contains measurable indicators tailored to the time-frame for prioritization of actions in the project implementation plans
7. Recommendation:

7.1. For procurement of portable refrigerant identifiers and RAC service tool kits, UNDP should consider either to conclude own LTAs or use LTAs already in place at sister organizations of the UN system that have acquired experience with procurement of equipment items for MP projects (e.g. UNIDO).

7.2. UNDP should ensure that Terms of Reference for procurement of major equipment items contain clear definition of related services to be guaranteed by the equipment suppliers, in particular that the supplier’s after sale service agents are operational in the recipient country or at least in the neighboring country. Provision of the after-sale services should be one of the criteria for commercial evaluation of bids submitted under the procurement event.

Management Response: [Added: 2018/10/26] [Last Updated: 2018/10/26]

Fully agree

Key Actions:

Key Action Responsible DueDate Status Comments Documents
7.1. Consideration using LTAs (own or other UN Family organizations) during procurement of RAC service equipment and toolkit during implementation of new UNDP/GEF project 7.2. Review UNDP Uzbekistan procurement rules and procedures on existence of recommended criteria for provision after-sale services
[Added: 2018/10/26] [Last Updated: 2018/11/29]
UNDP Sustainable Development Cluster 2018/12 Initiated 7.1. During implementation of new MSP, UNDP will study existing LTAs and use them when possible for procurement of RAC service equipment and toolkit 7.2. UNDP has reviewed procurement rules and procedures and identified that the provision of after-sale services is one of the vendors selection criteria
8. Recommendation:

UNDP should ensure that national project implementation teams establish on-going operational monitoring on actually provided co-financing for the projects

Management Response: [Added: 2018/10/26]

Fully agree

Key Actions:

Key Action Responsible DueDate Status Comments Documents
8.1. Inclusion into M&E plans of future projects operational monitoring on factually provided co-financing for the projects
[Added: 2018/10/26]
UNDP Sustainable Development Cluster 2018/10 Completed Ongoing operational monitoring on factually provided co-financing for the projects by project beneficiaries included into the M&E plans of new UNDP/GEF MSP on Complete HCFC Phase-out

Latest Evaluations

Contact us

220 East 42nd Street
20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org