Terminal Evaluation - Alternatives to DDT usage in Production of Antifouling Paint

Report Cover Image
Evaluation Plan:
2011-2015, China
Evaluation Type:
Project
Planned End Date:
04/2014
Completion Date:
04/2014
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
36,000

Share

Document Type Language Size Status Downloads
Download document TE Report (final) 18 November 2014.pdf report English 1651.77 KB Posted 635
Title Terminal Evaluation - Alternatives to DDT usage in Production of Antifouling Paint
Atlas Project Number: 00053562
Evaluation Plan: 2011-2015, China
Evaluation Type: Project
Status: Completed
Completion Date: 04/2014
Planned End Date: 04/2014
Management Response: Yes
Focus Area:
  • 1. Others
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 1.3. Solutions developed at national and sub-national levels for sustainable management of natural resources, ecosystem services, chemicals and waste
Evaluation Budget(US $): 36,000
Source of Funding: GEF
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Nationality
Hilda van der Veen International Consultant
Zhu Jianxin Associate Professor
GEF Evaluation: Yes
Evaluation Type:
Focal Area: Persistent Organic Pollutants
Project Type: EA
GEF Phase: GEF-1
PIMS Number: 3664
Key Stakeholders:
Countries: CHINA, PEOPLE'S REPUBLIC
Lessons
Findings
Recommendations
1 Recommendation: Organize the TE close to operational project completion (TE was conducted at the time approximately 1.8 Million US$ was still unspent. Although most of the remaining project funds had already been allocated to project activities that had almost come to an end, some project activities that would benefit from remaining project funds had not yet started. As such the TE team was unable to evaluate these activities. It would be recommended that for future TEs of GEF projects, that the TE would take place closer to operational closure of the project)
2 Recommendation: Extension of Project Duration (At the time of the Terminal Evaluation the project activities had not yet beenentirely completed (approximately ~20% of the project?s budget - although most of it committed - had not yet been spent). There were Project activities that were outstanding. The evaluators feel that if the project will be operationally closed in a rush, sustainability of project results will be seriously jeopardized. Instead it is recommended that the project will aim to operationally close by December 2014.... It is recommended that in this year?s Project Implementation Review (PIR) such an extension is requested after agreement has been reached on the proposed extension with the UNDP Bangkok Regional Service Center (RSC)).
3 Recommendation: UNDP involvement (UNDP Country Office participates more frequently, at least twice a year, in project site/field visits. Obtaining a better understanding of challenges faced by national stakeholders and beneficiaries in implementing project activities, would allowthe China UNDP Country Office to better anticipate the support FECO and national counterparts may require to speed up project implementation. Tow particular aspects can be mentioned: i) UNDP should ensure frequent training as well as regular procurement support to project subcontractors and Project Management Offices; ii) UNDP could play a more active role in supporting FECO and GEF projects in headhunting for high quality experts in niche areas that might not be easily accessible for national partners/stakeholders)
4 Recommendation: Project Design (For a country like China, where needs change rapidly over time due to the rapid pace of the country?s development, the project?s implementation team should keep in mind that a project would require to be redirected more frequently than projects in countries with a slower pace of development. ?Future GEF Chemicals and Waste projects in China would benefit from a detailed review of project and countryneeds at the time of the project?s Inception Workshop (and redirect project activities at that time if necessary) and plan for a critical Mid-Term Technical Review (in lieu of a more general MTE) to help the project team align the project?s activities and scope with the needs of the country and sector at that point in time.)
5 Recommendation: Earlier redirection of project activities (It took until 2012 ? 2013 before the project started supporting cleaner production activities at shipyards and risk assessment capacity building at 2 laboratories. This late redirecting of project activities resulted in ~46% of the budget being spent in 2013 (project year 6). This could have been avoided by earlier redirecting the project?s strategy/approach and deciding on additional activities with project beneficiaries. Alternatively such a decision could have taken place immediately after the MTE).
6 Recommendation: FECO involvement (: i) Project staff turnover (FECO project Coordinator) was high, which sometimes jeopardized the speed of project implementation and created delays. In the future FECO would, rather than appointing a single Project Coordinator, appoint a project team to oversee project management instead. Although one single person can take the lead on project implementation it would be recommended that an additional colleague is involved in the project on a part-time basis, so as to ensure that when the project coordinator might leave, the unit still contains one person who is familiar with the project)
7 Recommendation: Subcontracting as a means to support project beneficiaries (In the case of the Weihai Donghai Shipyard Co. Ltd., the manner in which the procurement was undertaken (and in particular the demonstration, testing and training of staff on the technology?s use) would have benefitted from more PMO, FECO and or UNDP involvement. Possibly, the sub- contracting modality for project beneficiaries should be applied exclusively when beneficiaries have a minimal amount of in-house capacity to undertake procurement)
8 Recommendation: Large number of project stakeholders and their understanding of how they contribute to project objectives (To further improve the involvement of stakeholders, encourage experiences exchanges and lessons-learned as well as understanding the role of a particular stakeholder, in the larger scheme of the project, it would be recommended that all project stakeholders and sub-contractors, meet at least once a year to exchange information on the status of project implementation (similar to the FECO organized meeting in May 2009), which would also allow for the exchange of lessons-learned between regions).
9 Recommendation: Capturing lessons- learned and project results
(when the project comes to an end chances are high that valuable information and guidelines (e.g. Risk Assessment (RA) results, RA templates and Standard Operating Procedures (SOPs), Cleaner Production Guidelines, Ship Dismantling guidelines, photos, etc.) could potentially be lost if they are not captured, documented and disseminated before the project comes to an end. Currently the project?s website is only available in Chinese, which doesn?t allow for the dissemination of project results beyond China. It would be recommended that the most useful documents prepared under the project would be translated in English and posted on the project website. It would also be recommended that the RA results are published at national level and the ship dismantling guidelines when finalized are shared with the IMO Convention)
10 Recommendation: Project Implementation Reviews (PIRs) (Quality of PIR was Marginally Satisfactory ? yearly PIR exercise was not used as a monitoring tool. It is strongly recommended that for future GEF funded Chemical and Waste projects, both UNDP China and FECO spend adequate time on preparing and completing a good quality PIR each year)
1. Recommendation: Recommendation: Organize the TE close to operational project completion (TE was conducted at the time approximately 1.8 Million US$ was still unspent. Although most of the remaining project funds had already been allocated to project activities that had almost come to an end, some project activities that would benefit from remaining project funds had not yet started. As such the TE team was unable to evaluate these activities. It would be recommended that for future TEs of GEF projects, that the TE would take place closer to operational closure of the project)
Management Response: [Added: 2014/12/17]

Outstanding project activities while TE was conducted was due to the late re-direction of remaining project funds, a decision that was quite complicated to accomplish. In the future, UNDP and FECO will make better evaluation of project progress to schedule the TE, so that all project activities are available to be evaluated during the TE process. It is however noted that the evaluators considered that missing the evaluation of those unfinished project activities ?is unlikely the project?s rating will change until the project is operationally closed.?

Key Actions:

2. Recommendation: Recommendation: Extension of Project Duration (At the time of the Terminal Evaluation the project activities had not yet beenentirely completed (approximately ~20% of the project?s budget - although most of it committed - had not yet been spent). There were Project activities that were outstanding. The evaluators feel that if the project will be operationally closed in a rush, sustainability of project results will be seriously jeopardized. Instead it is recommended that the project will aim to operationally close by December 2014.... It is recommended that in this year?s Project Implementation Review (PIR) such an extension is requested after agreement has been reached on the proposed extension with the UNDP Bangkok Regional Service Center (RSC)).
Management Response: [Added: 2014/12/17]

The recommendation to extend project completion to 31 December 2014 is welcome. This will enable the completion of outstanding activities to generate expected outcomes and impacts. The extension also allow for the completion of compilation and dissemination of experience and lessons learned, at a training and completion workshop that took place 16 and 17 October 2014, with extensive participation of project participants and generated excellent outcomes in exchange and sharing knowledge and experience.

Key Actions:

3. Recommendation: Recommendation: UNDP involvement (UNDP Country Office participates more frequently, at least twice a year, in project site/field visits. Obtaining a better understanding of challenges faced by national stakeholders and beneficiaries in implementing project activities, would allowthe China UNDP Country Office to better anticipate the support FECO and national counterparts may require to speed up project implementation. Tow particular aspects can be mentioned: i) UNDP should ensure frequent training as well as regular procurement support to project subcontractors and Project Management Offices; ii) UNDP could play a more active role in supporting FECO and GEF projects in headhunting for high quality experts in niche areas that might not be easily accessible for national partners/stakeholders)
Management Response: [Added: 2014/12/17]

At the project?s Inception Workshop and subsequent to the IW, UNDP and FECO had organized several times training workshop and exchanges with project participants on the rules and procedures of UNDP, FECO and GEF rules and procedures to ensure competent implementation of project activities. In the future, UNDP and FECO will be more proactive to anticipate the needs of the different levels of capacities of the project participants, and exercise close communication with them, to deliver assistance to address their potential needs. UNDP will also be more proactive to anticipate the project?s need on the provision of competent and qualified international expertise.

Key Actions:

4. Recommendation: Recommendation: Project Design (For a country like China, where needs change rapidly over time due to the rapid pace of the country?s development, the project?s implementation team should keep in mind that a project would require to be redirected more frequently than projects in countries with a slower pace of development. ?Future GEF Chemicals and Waste projects in China would benefit from a detailed review of project and countryneeds at the time of the project?s Inception Workshop (and redirect project activities at that time if necessary) and plan for a critical Mid-Term Technical Review (in lieu of a more general MTE) to help the project team align the project?s activities and scope with the needs of the country and sector at that point in time.)
Management Response: [Added: 2014/12/17]

It is a complicate process to gauge/anticipate social and economic changes, in particular of a fast pace development like China, any such foresight/forecast will not be of significant meaningful. Both UNDP and FECO agree with the evaluators that frequent review and appropriate early redirection of project resources will definitely benefit project outcomes and impacts, and achieve effective use of GEF resources

Key Actions:

5. Recommendation: Recommendation: Earlier redirection of project activities (It took until 2012 ? 2013 before the project started supporting cleaner production activities at shipyards and risk assessment capacity building at 2 laboratories. This late redirecting of project activities resulted in ~46% of the budget being spent in 2013 (project year 6). This could have been avoided by earlier redirecting the project?s strategy/approach and deciding on additional activities with project beneficiaries. Alternatively such a decision could have taken place immediately after the MTE).
Management Response: [Added: 2014/12/17]

UNDP and FECO agree with the evaluators that frequent review and appropriate early redirection of project resources will definitely benefit project outcomes and impacts, and achieve effective use of GEF resources

Key Actions:

6. Recommendation: Recommendation: FECO involvement (: i) Project staff turnover (FECO project Coordinator) was high, which sometimes jeopardized the speed of project implementation and created delays. In the future FECO would, rather than appointing a single Project Coordinator, appoint a project team to oversee project management instead. Although one single person can take the lead on project implementation it would be recommended that an additional colleague is involved in the project on a part-time basis, so as to ensure that when the project coordinator might leave, the unit still contains one person who is familiar with the project)
Management Response: [Added: 2014/12/17]

FECO has established and operated an effective project management system for over 20 years, including rules for procurement, stakeholder involvement and coordination, and monitoring and evaluation. Any new staff brought to manage a project will be trained and updated on rules and procedures, and will be properly briefed on project progress. Through such mechanism, FECO has successfully kept the continuity of project management despite of turnover of project management staff, a situation that is not uncommon in a large organization structure like FECO.

Key Actions:

7. Recommendation: Recommendation: Subcontracting as a means to support project beneficiaries (In the case of the Weihai Donghai Shipyard Co. Ltd., the manner in which the procurement was undertaken (and in particular the demonstration, testing and training of staff on the technology?s use) would have benefitted from more PMO, FECO and or UNDP involvement. Possibly, the sub- contracting modality for project beneficiaries should be applied exclusively when beneficiaries have a minimal amount of in-house capacity to undertake procurement)
Management Response: [Added: 2014/12/17]

The China Ministry of Finance encourages national capacity in the implementation of project activities. Another consideration in subcontracting is to encourage co-financing from the project participants / beneficiaries and allow them to employ their technical expertise to gain maximum technology and economic benefits. The equipment in question was fed with larger steel shot, thus caused jam in the tube that day. Additional trial operation and training have been provided to the operators after the TE mission. In general, project participants were assessed of their financial and technical capability before qualifying their participation. Provision of trial operation and operator training were usually a pre0condition of qualifying a subcontractor. In general, the sub- contracting modality has proven to be efficient and effective.

Key Actions:

8. Recommendation: Recommendation: Large number of project stakeholders and their understanding of how they contribute to project objectives (To further improve the involvement of stakeholders, encourage experiences exchanges and lessons-learned as well as understanding the role of a particular stakeholder, in the larger scheme of the project, it would be recommended that all project stakeholders and sub-contractors, meet at least once a year to exchange information on the status of project implementation (similar to the FECO organized meeting in May 2009), which would also allow for the exchange of lessons-learned between regions).
Management Response: [Added: 2014/12/17]

In response to the recommendation of the MTE, FECO and UNDP had already implemented more frequent meetings, workshops and exchanges with and amongst all key stakeholders, especially with the three implementing partners at the 3 sea areas. Key stakeholders were also invited to participate in Annual Review Meetings to be briefed on project progress, provide inputs into the Annual Work Plan, share experience and lessons learned in their implementation activities. This has become a standard feature in all GEF-funded project.

Key Actions:

9. Recommendation: Recommendation: Capturing lessons- learned and project results
(when the project comes to an end chances are high that valuable information and guidelines (e.g. Risk Assessment (RA) results, RA templates and Standard Operating Procedures (SOPs), Cleaner Production Guidelines, Ship Dismantling guidelines, photos, etc.) could potentially be lost if they are not captured, documented and disseminated before the project comes to an end. Currently the project?s website is only available in Chinese, which doesn?t allow for the dissemination of project results beyond China. It would be recommended that the most useful documents prepared under the project would be translated in English and posted on the project website. It would also be recommended that the RA results are published at national level and the ship dismantling guidelines when finalized are shared with the IMO Convention)
Management Response: [Added: 2014/12/17]

At time of TE, project information, data, reports and other materials were still being assembled and organized. Compilation of project results, implementation experience and lesson learned were summarized and disseminated at the training and completion workshops that took place on 16 and 17 October 2014, with participation of all project participants who made extensive exchanges and shared their experience and insights of their involvement in the project activities. FECO has now also set up the English version of the project website and began populating the website with project information, experience gained and lessons learned in the English version. Furthermore, a book on the risk assessment experiences, a special issue of professional magazine consisting of all major stakeholders? experiences were prepared. A video has also been prepared highlighting the project outcomes, impacts and experience, and was shown at the October training and completion workshop. Therefore, project results and impacts will definitely be properly documented and shared, within and outside China.

Key Actions:

10. Recommendation: Recommendation: Project Implementation Reviews (PIRs) (Quality of PIR was Marginally Satisfactory ? yearly PIR exercise was not used as a monitoring tool. It is strongly recommended that for future GEF funded Chemical and Waste projects, both UNDP China and FECO spend adequate time on preparing and completing a good quality PIR each year)
Management Response: [Added: 2014/12/17]

PIRs have been prepared annually in strict compliance with the GEF requirements, with joint review by UNDP Country Office and FECO. Major issues identified in the PIR have been addressed and reflected in subsequent year?s annual work plans, quarterly operational reviews, and annual project reviews. UNDP Country Office and FECO will exercise due diligence in the future to improve the inputs into the PIR, to improve its quality.

Key Actions:

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org