Evaluation Detail

Mid-term Evaluation of the "HCFC Phase-out Management Plan" project

Evaluation Plan:: 2017-2021, Jamaica
Evaluation Type:: Mid Term Project
Planned End Date:: 06/2017
Completion Date:: 09/2017
Status:: Completed
Management Response:: Yes
Evaluation Budget(US $):: 19,000

Document	Type	Language	Size	Status	Downloads
Evaluation of HPMP.pdf	tor	English	579.86 KB	Posted	1229
HCFC PHASE-OUT MANAGEMENT PLAN_MIDTERM_EVALUATION.pdf	report	English	1847.98 KB	Posted	1250

Title

Mid-term Evaluation of the "HCFC Phase-out Management Plan" project

Atlas Project Number:

00067121

Evaluation Plan:

2017-2021, Jamaica

Evaluation Type:

Mid Term Project

Status:

Completed

Completion Date:

09/2017

Planned End Date:

06/2017

Management Response:

Yes

Focus Area:

1. Environment & Sustainable Development
2. Others

Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)

1. Output 1.4. Scaled up action on climate change adaptation and mitigation across sectors which is funded and implemented

SDG Goal

Goal 12. Ensure sustainable consumption and production patterns

SDG Target

12.4 By 2020, achieve the environmentally sound management of chemicals and all wastes throughout their life cycle, in accordance with agreed international frameworks, and significantly reduce their release to air, water and soil in order to minimize their adverse impacts on human health and the environment

Evaluation Budget(US $):

19,000

Source of Funding:

Project

Evaluation Expenditure(US $):

20,000

Joint Programme:

Joint Evaluation:

Evaluation Team members:

Name	Title	Email	Nationality
Elsie Laurence Chounoune	Deputy Representative, UNDP	elsie.chounoune@undp.org	JAMAICA
Vivienne Williams Thompson	Director, NEPA	Vivienne.WThompson@nepa.gov.jm	JAMAICA

GEF Evaluation:

Key Stakeholders:

UNDP, National Environment and Planning Agency (NEPA)

Countries:

JAMAICA

Lessons
Findings
1.	Project Alignment to Country and Project Sponsor Policy Agendas Assessment was made using the following indicators: Aligned to a significant extent with CP and GoJ Policy; Some alignment with CP and GoJ Policy; There is limited alignment with CP and GoJ Policy and, There is no alignment with CP and GoJ Policy Finding(s): The Project is aligned to a significant extent with both the UNDP and the GoJ Agenda a) The Project is aligned to the UNDP Country Plan: Specifically UNDAF Outcome #3 Ensuring Environmental Security, Reduction of Poverty and Increased Social Inclusion of the Poor. b) The Project is aligned to the GoJ 2030 Vision; Specifically Goal #4 – Jamaica has a Healthy Natural Environment and Output #14, Hazard Risk Reduction & Adaptation to Climate Change c) The Project is aligned to GoJ commitment to International Agreements: Jamaica has ratified the Vienna Convention for the Protection of the Ozone Layer and The Montreal Protocol on Substances that Deplete the Ozone Layer Tag: Relevance National
2.	Project Relevance to Problem Assessment was made using the following scale: Project Outputs aligned with issues raised in HPMP background research; The majority of the issues raised in the HPMP background research are addressed; Some of the issues raised in the HPMP background research are addressed and, There is no relationship of the Project Outputs to the issues r0aised in the HPMP background research Finding(s): Some of the issues raised in the HPMP background research are addressed Tag: Relevance Implementation Modality Monitoring and Evaluation Programme/Project Design Technical Support
3.	Project Relevance to Industry Stakeholders Assessment was made using the following scale: Project is fully aligned with the concerns and state of the industry; ? There is significant alignment with the industry concerns and current reality; ? The majority of the concerns and issues are addressed to some degree; ? There is limited alignment between the concerns of the industry and the Project and, ? There is no alignment of the Project with the concerns and issues of the industry. Finding(s): The majority of the concerns and issues of Industry Stakeholders are addressed to some degree. Specifically: Tag: Relevance Partnership Programme/Project Design Shared Services Technical Support
4.	Progress Achieving Outputs and Impact on Outcomes Assessment of project progress was made using the traffic light system: Green Light: Outputs at time of MTE are consistent with project targets ? Yellow Light: Some of the targets set have been met as a result of the project outputs; ? Red Light: There has been negligible or no progress towards targets. Finding(s): With the exception of Component I, Project outputs receive a ‘red light’ in the traffic light scoring system. Table 2, and Table 3 present Progress towards Project Results and Project towards Outcome Analysis. Tag: Effectiveness Impact Human and Financial resources Service delivery Technical Support
5.	Project Management Project Governance and Management (PGM) is the responsibility of the IP, the PSC and the Project Sponsors (UNDP). Their roles in the process vary but each is critical to ensure that a Project remains on track. Effectiveness of project oversight was assessed using the following scale: The PGM Team provided effective and responsive management of the Project ? The PGM Team provided adequate management and oversight of the Project ? The PGM Team provided minimal management oversight and of the Project ? The PGM Team provided inadequate management oversight of the Project Finding(s): There was inadequate management and oversight of the Project. Tag: Effectiveness Human and Financial resources Implementation Modality Operational Efficiency Project and Programme management Results-Based Management Technical Support
6.	Effective Monitoring & Evaluation of Project The following scale was used to assess the extent to which there was effective monitoring and evaluation of the Project: M&E indicators were established, appropriate and clear to support effective project monitoring; Some M&E Indicators were established, they were somewhat appropriate, and allowed for some level of monitoring; The M&E indicators were inadequate and did not allow for effective monitoring and evaluation of the project. Finding(s): There were some M&E indicators and if tracked would allow for some level of monitoring and evaluation of the Project. The IP established no system of tracking any of the Project outputs and determining if they were aligned to Project targets. Tag: Effectiveness Human and Financial resources Monitoring and Evaluation Project and Programme management
7.	PROJECT EFFICIENCY Use of Project Funds The use of project funds is assessed using the following scale: Project funds spent to date are consistent with the budget and have achieved expected targets; ? Project funds spent to date are consistent with the budget but have not achieved the expected project targets, Project funds spent to date have been somewhat consistent with the budget and has resulted in achievement of project targets; Project funds have not been spent in accordance with the budget and projects may or may not have been achieved. Finding(s): With the exception of Component I Project funds have not been spent in accordance with the budget and Project targets have not been achieved. Tag: Efficiency Human and Financial resources Procurement
8.	Use of Project Resources to Increase Efficiency There have been consistent recommendations presented with a view to increasing both efficiency and project impact; There have been some (intermittent) recommendations presented aimed at increasing efficiency and impact; There have been some changes however they have had an adverse effect on efficiency and, There is no evidence that efforts were made to present changes which were likely to improve efficiencies and or impact of the Project. Finding(s): There have been changes in the use of the financial resources of the Project however these changes have not increased the efficiency nor have they resulted in supporting the desired output of the number of RAC practitioners trained. Tag: Efficiency Business Model Human and Financial resources Operational Efficiency Project and Programme management
9.	Project Risk Matrix The following scale was used to assess the extent to which the IP/PSC used the Project risk log to support the implementation of the Project. There is a robust risk monitoring process and the risk log is utilized to support the implementation of the Project; There is indication that occasionally the risk matrix is referenced; There is minimal use of the risk matrix to manage the implementation of the Project; The risk matrix is not used as a tool to manage the project. Finding(s): There has been minimal use of the risk matrix to manage the implementation of the Project Tag: Efficiency Human and Financial resources Implementation Modality Monitoring and Evaluation Risk Management
10.	PROJECT SUSTAINABILITY Reducing Importation of ODS SSS has confirmed that with the conversion of their foam manufacturing process away from HCFC 141b there would be no economical reason to revert to using this product. Tag: Sustainability
11.	Local Trainers A team of Local Trainers would allow for ongoing delivery of the three modules both within the project duration and after the end of the Project. The following scale was used to assess if this has been achieved: All participants to the ToT were clear on their role post the training; The majority of the participants were clear on their role post the training event; A few of the participants were clear on their role post the training event and, The participants had no indication they were required to conduct any training post the workshop. Finding(s): Only a 25% ToT trainees understood that they were to conduct training following the ToT Workshops. Further none were engaged as Trainers. One was engaged as a Technical Assistant to support the NC delivery of training sessions. Tag: Sustainability Human and Financial resources Project and Programme management
12.	Tracking Local Trainers & Technicians There is a robust ongoing tracking of Trainees of the ToT and TT; There is an update on the majority of the ToT trainees and their activities; There is a monitoring of some of the ToT trainees and, There is no monitoring of any trainees. Finding(s): There is no monitoring of trainees from the ToT or TT Workshops The IP indicates that there is no mechanism for tracking participants to either the ToT or the Technician Training workshops. This despite the recommendation at an early PSC meeting that this should be undertaken. Tag: Sustainability Human and Financial resources Monitoring and Evaluation
13.	PROJECT IMPACT The Project is at its mid-point with an additional four years left. The activities for Component I achieve the desired outcome, a 97% reduction in the importation of HCFC 14b. This will clearly have the desired impact on the environment. Tag: Impact Human and Financial resources Technical Support
14.	GENDER RESPONSIVENESS There is no indication that gender was a consideration in the design and/or execution of this project; There is some indication that gender was a consideration in the design and/or the execution of this project and, There is strong evidence that there was a focus on ensuring during design and implementation that the project was gender responsive. Finding(s): There is no indication that gender was a consideration in the design and/or execution of the Project. Tag: Gender Equality Gender Parity Implementation Modality Programme/Project Design

Recommendations
1	There is clearly a demand for ongoing training and upgrading of RAC practitioners which makes the key component of this Project relevant, however the delivery of short term workshops should be guided by a review of the landscape. The IP could with an appropriate facilitator: Conduct a series of focus group sessions to determine the critical training needs of the target population, best times of the week and day to conduct training and if there is any scope for shared cost of training and/or training material. Have a working session with major employers, many who would have already shifted to alternative technology which resulted in a reduction in the importation levels for HCFCs, determine what they see as training needs for technicians and their willingness to be partners in the process. Hold a conversation with major technical training facilities who offer short and/or long term training for the RAC industry. Confirm their understanding of training demands. This will leave the Project with a better understanding of both demand and supply, the IP along with the Technical Committee can review the outputs from these consultations and make recommendations for any adjustments to the current content, delivery mode and scheduling.
2	Develop an M&E Framework, inclusive of all reports which must be reviewed by the PSC at its quarterly meetings, establish the criteria for consideration for change to any element of the redesigned programme. In the event that the content developed and currently being used for workshops will be utilized for follow on training contract a resource with appropriate skills to ensure that the material takes into consideration the learning style of adult learners in the Jamaica context, this is primarily tactile and/or visual. The end result is likely to be two sets of Workshop Kits.
3	Assess the model implemented by Canada and UNEP for training of technicians, there was an initial training of 25 local technicians, they were the corps of trainers and were affiliated to technical training facilities. In the second stage they went on to train 127 technicians by conducting 8 3-Day workshops with certificates being issued through HEART Trust/NTA VTDI. This is likely the model which guided the design of Component II of the HPMP. In the event that the content developed and currently being used for workshops will be utilized for follow on training contract, a resource with appropriate skills to ensure that the material takes into consideration the learning style of adult learners in the Jamaica context, this is primarily tactile and/or visual. Identify 15 master trainers and provide a ‘refresher’ including techniques on how to deliver training to adult learners. Provide a stipend to meet out of pocket cost, which is consistent with the current design of the Project. Develop a toolkit for the trainers to allow for similarity in data collected at each training site, to include - Pre-screening applications which are submitted at the time of application and guided a decision on the Workshop the applicant should attend; Pre and Post Test Tools; Standard attendance register and a template for preparing post training reports for submission Utilize island wide training facilities of HEART Trust/ National Tool Engineering Institute (NTEI), National Caribbean University, Caribbean Maritime Institute and similar Organisations as the hub for scheduled training over a period of time which ensures maximum attendance.
4	To increase the participation of female practitioners in the RAC industry participating in workshops, there must be a specific and targeted campaign.

1. Recommendation:

There is clearly a demand for ongoing training and upgrading of RAC practitioners which makes the key component of this Project relevant, however the delivery of short term workshops should be guided by a review of the landscape.

The IP could with an appropriate facilitator:

Conduct a series of focus group sessions to determine the critical training needs of the target population, best times of the week and day to conduct training and if there is any scope for shared cost of training and/or training material.
Have a working session with major employers, many who would have already shifted to alternative technology which resulted in a reduction in the importation levels for HCFCs, determine what they see as training needs for technicians and their willingness to be partners in the process.
Hold a conversation with major technical training facilities who offer short and/or long term training for the RAC industry. Confirm their understanding of training demands.

This will leave the Project with a better understanding of both demand and supply, the IP along with the Technical Committee can review the outputs from these consultations and make recommendations for any adjustments to the current content, delivery mode and scheduling.

Management Response: [Added: 2017/12/12] [Last Updated: 2021/01/13]

Need assessments were completed by July 2018.

Key Actions:

Key Action	Responsible	DueDate	Status	Comments	Documents
Conduct of a Training Needs Assessment mong RAC practitioners [Added: 2017/12/12] [Last Updated: 2018/12/17]	Project Manager and Project Focal Management	2018/03	Completed	Training Needs Assessment was completed by July 2018

2. Recommendation:

Develop an M&E Framework, inclusive of all reports which must be reviewed by the PSC at its quarterly meetings, establish the criteria for consideration for change to any element of the redesigned programme.

In the event that the content developed and currently being used for workshops will be utilized for follow on training contract a resource with appropriate skills to ensure that the material takes into consideration the learning style of adult learners in the Jamaica context, this is primarily tactile and/or visual. The end
result is likely to be two sets of Workshop Kits.

Management Response: [Added: 2017/12/12] [Last Updated: 2021/01/13]

M&E Analyst and project manager developed M&E framework

Key Actions:

Key Action	Responsible	DueDate	Status	Comments	Documents
Support the development of the M&E Framework [Added: 2017/12/12] [Last Updated: 2018/12/20]	Project Manager and Programme Analyst, M&E	2018/02	Completed	Programme Analyst, M&E provided support for the development of the M&E Framework

3. Recommendation:

Assess the model implemented by Canada and UNEP for training of technicians, there was an initial training of 25 local technicians, they were the corps of trainers and were affiliated to technical training facilities. In the second stage they went on to train 127 technicians by conducting 8 3-Day workshops with certificates being issued through HEART Trust/NTA VTDI. This is likely the model which guided the design of Component II of the HPMP. In the event that the content developed and currently being used for workshops will be utilized for follow on training contract, a resource with appropriate skills to ensure that the material takes into consideration the learning style of adult learners in the Jamaica context, this is primarily tactile and/or visual.

Identify 15 master trainers and provide a ‘refresher’ including techniques on how to deliver training to adult learners. Provide a stipend to meet out of pocket cost, which is consistent with the current design of the Project. Develop a toolkit for the trainers to allow for similarity in data collected at each training site, to include - Pre-screening applications which are submitted at the time of application and guided a decision on the Workshop the applicant should attend; Pre and Post Test Tools; Standard attendance register and a template for preparing post training reports for submission
Utilize island wide training facilities of HEART Trust/ National Tool Engineering Institute (NTEI), National Caribbean University, Caribbean Maritime Institute and similar Organisations as the hub for scheduled training over a period of time which ensures maximum attendance.

Management Response: [Added: 2017/12/12] [Last Updated: 2021/01/13]

Based on the training Needs Assessment we will attempt to negotiate/advocate with learning instiutions for support of the findings into thier curriculum. The ToT approach should be used to also train the technicians.

Key Actions:

Key Action	Responsible	DueDate	Status	Comments	Documents
Once the findings from the needs assessment are received, an implementation plan will be developed to not only disseminate the findings to the training institutions but also use as a base for any additional training activities the project will conduct in 2018 [Added: 2017/12/12] [Last Updated: 2022/01/31]	Project Manager	2022/03	Overdue-Initiated

4. Recommendation:

To increase the participation of female practitioners in the RAC industry participating in workshops, there must be a specific and targeted campaign.

Management Response: [Added: 2017/12/12] [Last Updated: 2021/01/13]

Support the strengthening of the local association, JARVA, which can suppport a campaign to increase female practitioners

Key Actions:

Key Action	Responsible	DueDate	Status	Comments	Documents
Support the strengthening of JARVA, the local industry organization [Added: 2017/12/12] [Last Updated: 2018/12/17]	Project Manager	2018/05	No Longer Applicable	[Justification: JARVA is no longer operational and as such this mechanism will not be utilized. Additionally, this aspect of the consultancy was removed because the consultant was over budget. ]

Mid-term Evaluation of the "HCFC Phase-out Management Plan" project

Share

Management Response: [Added: 2017/12/12] [Last Updated: 2021/01/13]

Key Actions:

Management Response: [Added: 2017/12/12] [Last Updated: 2021/01/13]

Key Actions:

Management Response: [Added: 2017/12/12] [Last Updated: 2021/01/13]

Key Actions:

Management Response: [Added: 2017/12/12] [Last Updated: 2021/01/13]

Key Actions:

Evaluations by Unit

Latest Evaluations

Evaluation Snap Shot

Contact us